2024-07-22

Title: Offline Digital Euro: a Minimum Viable CBDC using Groth-Sahai proofs

• Authors: Leon Kempen, Johan Pouwelse
• Subjects: cs.CR, q-fin.TR
• Abstract URL: https://arxiv.org/abs/2407.13776
• Pdf URL: https://arxiv.org/pdf/2407.13776
• Copy Paste: [[2407.13776]] Offline Digital Euro: a Minimum Viable CBDC using Groth-Sahai proofs(https://arxiv.org/abs/2407.13776)
• Keywords: privacy, protect
• Abstract: Current digital payment solutions are fragile and offer less privacy than traditional cash. Their critical dependency on an online service used to perform and validate transactions makes them void if this service is unreachable. Moreover, no transaction can be executed during server malfunctions or power outages. Due to climate change, the likelihood of extreme weather increases. As extreme weather is a major cause of power outages, the frequency of power outages is expected to increase. The lack of privacy is an inherent result of their account-based design or the use of a public ledger. The critical dependency and lack of privacy can be resolved with a Central Bank Digital Currency that can be used offline. This thesis proposes a design and a first implementation for an offline-first digital euro. The protocol offers complete privacy during transactions using zero-knowledge proofs. Furthermore, transactions can be executed offline without third parties and retroactive double-spending detection is facilitated. To protect the users' privacy, but also guard against money laundering, we have added the following privacy-guarding mechanism. The bank and trusted third parties for law enforcement must collaborate to decrypt transactions, revealing the digital pseudonym used in the transaction. Importantly, the transaction can be decrypted without decrypting prior transactions attached to the digital euro. The protocol has a working initial implementation showcasing its usability and demonstrating functionality.

Title: RDBE: Reasoning Distillation-Based Evaluation Enhances Automatic Essay Scoring

• Authors: Ali Ghiasvand Mohammadkhani
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13781
• Pdf URL: https://arxiv.org/pdf/2407.13781
• Copy Paste: [[2407.13781]] RDBE: Reasoning Distillation-Based Evaluation Enhances Automatic Essay Scoring(https://arxiv.org/abs/2407.13781)
• Keywords: interpretability, large language model
• Abstract: Recently, various encoder-only and encoder-decoder pre-trained models like BERT and T5 have been applied to automatic essay scoring (AES) as small language models. However, existing studies have primarily treated this task akin to a classification problem, focusing solely on outputting scores in the target text without offering interpretations for the generated scores. Departing from the approaches, we introduce Reasoning Distillation-Based Evaluation (RDBE), which integrates interpretability to elucidate the rationale behind model scores while enhancing performance through initial reasoning. This interpretive capability is acquired during training by leveraging generated reasoning from a large language model (LLM) to distill a small language model (SLM). Our experimental results demonstrate the efficacy of RDBE across all scoring rubrics considered in the dataset. RDBE outperforms both zero-shot LLM generation and generation from a baseline fine-tuned model, establishing itself as state-of-the-art in the corresponding dataset. This highlights its practical interpretative output and enhanced performance.

Title: Enhancing Software Supply Chain Resilience: Strategy For Mitigating Software Supply Chain Security Risks And Ensuring Security Continuity In Development Lifecycle

• Authors: Ahmed Akinsola, Abdullah Akinde
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13785
• Pdf URL: https://arxiv.org/pdf/2407.13785
• Copy Paste: [[2407.13785]] Enhancing Software Supply Chain Resilience: Strategy For Mitigating Software Supply Chain Security Risks And Ensuring Security Continuity In Development Lifecycle(https://arxiv.org/abs/2407.13785)
• Keywords: secure, security
• Abstract: This article delves into the strategic approaches and preventive measures necessary to safeguard the software supply chain against evolving threats. It aims to foster an understanding of the challenges and vulnerabilities inherent in software supply chain resilience and to promote transparency and trust in the digital infrastructure that underpins contemporary society. By examining the concept of software supply chain resilience and assessing the current state of supply chain security, the article provides a foundation for discussing strategies and practices that can mitigate security risks and ensure security continuity throughout the development lifecycle. Through this comprehensive analysis, the article contributes to the ongoing effort to strengthen the security posture of software supply chains, thereby ensuring the reliable and secure operation of digital systems in a connected world

Title: Continuous Embedding Attacks via Clipped Inputs in Jailbreaking Large Language Models

• Authors: Zihao Xu, Yi Liu, Gelei Deng, Kailong Wang, Yuekang Li, Ling Shi, Stjepan Picek
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13796
• Pdf URL: https://arxiv.org/pdf/2407.13796
• Copy Paste: [[2407.13796]] Continuous Embedding Attacks via Clipped Inputs in Jailbreaking Large Language Models(https://arxiv.org/abs/2407.13796)
• Keywords: security, attack, large language model
• Abstract: Security concerns for large language models (LLMs) have recently escalated, focusing on thwarting jailbreaking attempts in discrete prompts. However, the exploration of jailbreak vulnerabilities arising from continuous embeddings has been limited, as prior approaches primarily involved appending discrete or continuous suffixes to inputs. Our study presents a novel channel for conducting direct attacks on LLM inputs, eliminating the need for suffix addition or specific questions provided that the desired output is predefined. We additionally observe that extensive iterations often lead to overfitting, characterized by repetition in the output. To counteract this, we propose a simple yet effective strategy named CLIP. Our experiments show that for an input length of 40 at iteration 1000, applying CLIP improves the ASR from 62% to 83%

Title: Less is More: Sparse Watermarking in LLMs with Enhanced Text Quality

• Authors: Duy C. Hoang, Hung T. Q. Le, Rui Chu, Ping Li, Weijie Zhao, Yingjie Lao, Khoa D. Doan
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13803
• Pdf URL: https://arxiv.org/pdf/2407.13803
• Copy Paste: [[2407.13803]] Less is More: Sparse Watermarking in LLMs with Enhanced Text Quality(https://arxiv.org/abs/2407.13803)
• Keywords: watermark, large language model
• Abstract: With the widespread adoption of Large Language Models (LLMs), concerns about potential misuse have emerged. To this end, watermarking has been adapted to LLM, enabling a simple and effective way to detect and monitor generated text. However, while the existing methods can differentiate between watermarked and unwatermarked text with high accuracy, they often face a trade-off between the quality of the generated text and the effectiveness of the watermarking process. In this work, we present a novel type of LLM watermark, Sparse Watermark, which aims to mitigate this trade-off by applying watermarks to a small subset of generated tokens distributed across the text. The key strategy involves anchoring watermarked tokens to words that have specific Part-of-Speech (POS) tags. Our experimental results demonstrate that the proposed watermarking scheme achieves high detectability while generating text that outperforms previous LLM watermarking methods in quality across various tasks

Title: Revisiting Attention for Multivariate Time Series Forecasting

• Authors: Haixiang Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13806
• Pdf URL: https://arxiv.org/pdf/2407.13806
• Copy Paste: [[2407.13806]] Revisiting Attention for Multivariate Time Series Forecasting(https://arxiv.org/abs/2407.13806)
• Keywords: transformer
• Abstract: Current Transformer methods for Multivariate Time-Series Forecasting (MTSF) are all based on the conventional attention mechanism. They involve sequence embedding and performing a linear projection of Q, K, and V, and then computing attention within this latent space. We have never delved into the attention mechanism to explore whether such a mapping space is optimal for MTSF. To investigate this issue, this study first proposes Frequency Spectrum attention (FSatten), a novel attention mechanism based on the frequency domain space. It employs the Fourier transform for embedding and introduces Multi-head Spectrum Scaling (MSS) to replace the conventional linear mapping of Q and K. FSatten can accurately capture the periodic dependencies between sequences and outperform the conventional attention without changing mainstream architectures. We further design a more general method dubbed Scaled Orthogonal attention (SOatten). We propose an orthogonal embedding and a Head-Coupling Convolution (HCC) based on the neighboring similarity bias to guide the model in learning comprehensive dependency patterns. Experiments show that FSatten and SOatten surpass the SOTA which uses conventional attention, making it a good alternative as a basic attention mechanism for MTSF. The codes and log files will be released at: this https URL.

Title: Which objects help me to act effectively? Reasoning about physically-grounded affordances

• Authors: Anne Kemmeren, Gertjan Burghouts, Michael van Bekkum, Wouter Meijer, Jelle van Mil
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.13811
• Pdf URL: https://arxiv.org/pdf/2407.13811
• Copy Paste: [[2407.13811]] Which objects help me to act effectively? Reasoning about physically-grounded affordances(https://arxiv.org/abs/2407.13811)
• Keywords: large language model
• Abstract: For effective interactions with the open world, robots should understand how interactions with known and novel objects help them towards their goal. A key aspect of this understanding lies in detecting an object's affordances, which represent the potential effects that can be achieved by manipulating the object in various ways. Our approach leverages a dialogue of large language models (LLMs) and vision-language models (VLMs) to achieve open-world affordance detection. Given open-vocabulary descriptions of intended actions and effects, the useful objects in the environment are found. By grounding our system in the physical world, we account for the robot's embodiment and the intrinsic properties of the objects it encounters. In our experiments, we have shown that our method produces tailored outputs based on different embodiments or intended effects. The method was able to select a useful object from a set of distractors. Finetuning the VLM for physical properties improved overall performance. These results underline the importance of grounding the affordance search in the physical world, by taking into account robot embodiment and the physical properties of objects.

Title: Many Perception Tasks are Highly Redundant Functions of their Input Data

• Authors: Rahul Ramesh, Anthony Bisulco, Ronald W. DiTullio, Linran Wei, Vijay Balasubramanian, Kostas Daniilidis, Pratik Chaudhari
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13841
• Pdf URL: https://arxiv.org/pdf/2407.13841
• Copy Paste: [[2407.13841]] Many Perception Tasks are Highly Redundant Functions of their Input Data(https://arxiv.org/abs/2407.13841)
• Keywords: segmentation
• Abstract: We show that many perception tasks, from visual recognition, semantic segmentation, optical flow, depth estimation to vocalization discrimination, are highly redundant functions of their input data. Images or spectrograms, projected into different subspaces, formed by orthogonal bases in pixel, Fourier or wavelet domains, can be used to solve these tasks remarkably well regardless of whether it is the top subspace where data varies the most, some intermediate subspace with moderate variability--or the bottom subspace where data varies the least. This phenomenon occurs because different subspaces have a large degree of redundant information relevant to the task.

Title: CoxSE: Exploring the Potential of Self-Explaining Neural Networks with Cox Proportional Hazards Model for Survival Analysis

• Authors: Abdallah Alabdallah, Omar Hamed, Mattias Ohlsson, Thorsteinn Rögnvaldsson, Sepideh Pashami
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.13849
• Pdf URL: https://arxiv.org/pdf/2407.13849
• Copy Paste: [[2407.13849]] CoxSE: Exploring the Potential of Self-Explaining Neural Networks with Cox Proportional Hazards Model for Survival Analysis(https://arxiv.org/abs/2407.13849)
• Keywords: robust, explainability
• Abstract: The Cox Proportional Hazards (CPH) model has long been the preferred survival model for its explainability. However, to increase its predictive power beyond its linear log-risk, it was extended to utilize deep neural networks sacrificing its explainability. In this work, we explore the potential of self-explaining neural networks (SENN) for survival analysis. we propose a new locally explainable Cox proportional hazards model, named CoxSE, by estimating a locally-linear log-hazard function using the SENN. We also propose a modification to the Neural additive (NAM) models hybrid with SENN, named CoxSENAM, which enables the control of the stability and consistency of the generated explanations. Several experiments using synthetic and real datasets have been performed comparing with a NAM-based model, DeepSurv model explained with SHAP, and a linear CPH model. The results show that, unlike the NAM-based model, the SENN-based model can provide more stable and consistent explanations while maintaining the same expressiveness power of the black-box model. The results also show that, due to their structural design, NAM-based models demonstrated better robustness to non-informative features. Among these models, the hybrid model exhibited the best robustness.

Title: X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs

• Authors: Sirnam Swetha, Jinyu Yang, Tal Neiman, Mamshad Nayeem Rizve, Son Tran, Benjamin Yao, Trishul Chilimbi, Mubarak Shah
• Subjects: cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.13851
• Pdf URL: https://arxiv.org/pdf/2407.13851
• Copy Paste: [[2407.13851]] X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs(https://arxiv.org/abs/2407.13851)
• Keywords: transformer, generative, large language model
• Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have revolutionized the field of vision-language understanding by integrating visual perception capabilities into Large Language Models (LLMs). The prevailing trend in this field involves the utilization of a vision encoder derived from vision-language contrastive learning (CL), showing expertise in capturing overall representations while facing difficulties in capturing detailed local patterns. In this work, we focus on enhancing the visual representations for MLLMs by combining high-frequency and detailed visual representations, obtained through masked image modeling (MIM), with semantically-enriched low-frequency representations captured by CL. To achieve this goal, we introduce X-Former which is a lightweight transformer module designed to exploit the complementary strengths of CL and MIM through an innovative interaction mechanism. Specifically, X-Former first bootstraps vision-language representation learning and multimodal-to-multimodal generative learning from two frozen vision encoders, i.e., CLIP-ViT (CL-based) and MAE-ViT (MIM-based). It further bootstraps vision-to-language generative learning from a frozen LLM to ensure visual features from X-Former can be interpreted by the LLM. To demonstrate the effectiveness of our approach, we assess its performance on tasks demanding detailed visual understanding. Extensive evaluations indicate that X-Former excels in visual reasoning tasks involving both structural and semantic categories in the GQA dataset. Assessment on fine-grained visual perception benchmark further confirms its superior capabilities in visual understanding.

Title: SecureVAX: A Blockchain-Enabled Secure Vaccine Passport System

• Authors: Debendranath Das, Sushmita Ruj, Subhamoy Maitra
• Subjects: cs.CR, cs.DC, cs.ET
• Abstract URL: https://arxiv.org/abs/2407.13852
• Pdf URL: https://arxiv.org/pdf/2407.13852
• Copy Paste: [[2407.13852]] SecureVAX: A Blockchain-Enabled Secure Vaccine Passport System(https://arxiv.org/abs/2407.13852)
• Keywords: secure, privacy
• Abstract: A vaccine passport serves as documentary proof, providing passport holders with greater freedom while roaming around during pandemics. It confirms vaccination against certain infectious diseases like COVID-19, Ebola, and flu. The key challenges faced by the digital vaccine passport system include passport forgery, unauthorized data access, and inaccurate information input by vaccination centers. Privacy concerns also need to be addressed to ensure that the user's personal identification information (PII) is not compromised. Additionally, it is necessary to track vaccine vials or doses to verify their authenticity, prevent misuse and illegal sales, as well as to restrict the illicit distribution of vaccines. To address these challenges, we propose a Blockchain-Enabled Secure Vaccine Passport System, leveraging the power of smart contracts. Our solution integrates off-chain and on-chain cryptographic computations, facilitating secure communication among various entities. We have utilized the InterPlanetary File System (IPFS) to store encrypted vaccine passports of citizens securely. Our prototype is built on the Ethereum platform, with smart contracts deployed on the Sepolia Test network, allowing for performance evaluation and validation of the system's effectiveness. By combining IPFS as a distributed data storage platform and Ethereum as a blockchain platform, our solution paves the way for secure, efficient, and globally interoperable vaccine passport management, supporting comprehensive vaccination initiatives worldwide.

Title: Network Traffic Analysis of Medical Devices

• Authors: Nowfel Mashnoor, Batyr Charyyev
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13857
• Pdf URL: https://arxiv.org/pdf/2407.13857
• Copy Paste: [[2407.13857]] Network Traffic Analysis of Medical Devices(https://arxiv.org/abs/2407.13857)
• Keywords: security, attack
• Abstract: The availability of medical devices such as glucose levels and blood pressure measuring devices is continuously increasing. These devices have gained user interest as they are easy to use. However, medical devices introduce extra complexity to the network by being numerous, heterogeneous, and more vulnerable to cyber-attacks. For better network management and overall network security, it is important to understand the network traffic characteristics of the devices. Thus, in this paper, we analyze in detail the traffic characteristics of 8 medical devices both at the device level and at the level of individual functionality of each device. We collect and share both network and Bluetooth traffic from a total of 51 functionalities of the devices. Our analysis includes different metrics such as protocols, amount of incoming/outgoing traffic, DNS queries, and analysis of traffic destinations. We observed that devices have unique network and Bluetooth traffic characteristics, that might be useful in developing networking tools for medical devices.

Title: A Closer Look at GAN Priors: Exploiting Intermediate Features for Enhanced Model Inversion Attacks

• Authors: Yixiang Qiu, Hao Fang, Hongyao Yu, Bin Chen, MeiKang Qiu, Shu-Tao Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13863
• Pdf URL: https://arxiv.org/pdf/2407.13863
• Copy Paste: [[2407.13863]] A Closer Look at GAN Priors: Exploiting Intermediate Features for Enhanced Model Inversion Attacks(https://arxiv.org/abs/2407.13863)
• Keywords: security, privacy, attack, extraction, generative
• Abstract: Model Inversion (MI) attacks aim to reconstruct privacy-sensitive training data from released models by utilizing output information, raising extensive concerns about the security of Deep Neural Networks (DNNs). Recent advances in generative adversarial networks (GANs) have contributed significantly to the improved performance of MI attacks due to their powerful ability to generate realistic images with high fidelity and appropriate semantics. However, previous MI attacks have solely disclosed private information in the latent space of GAN priors, limiting their semantic extraction and transferability across multiple target models and datasets. To address this challenge, we propose a novel method, Intermediate Features enhanced Generative Model Inversion (IF-GMI), which disassembles the GAN structure and exploits features between intermediate blocks. This allows us to extend the optimization space from latent code to intermediate features with enhanced expressive capabilities. To prevent GAN priors from generating unrealistic images, we apply a L1 ball constraint to the optimization process. Experiments on multiple benchmarks demonstrate that our method significantly outperforms previous approaches and achieves state-of-the-art results under various settings, especially in the out-of-distribution (OOD) scenario. Our code is available at: this https URL

Title: Keypoint Aware Masked Image Modelling

• Authors: Madhava Krishna, A V Subramanyam
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13873
• Pdf URL: https://arxiv.org/pdf/2407.13873
• Copy Paste: [[2407.13873]] Keypoint Aware Masked Image Modelling(https://arxiv.org/abs/2407.13873)
• Keywords: transformer
• Abstract: SimMIM is a widely used method for pretraining vision transformers using masked image modeling. However, despite its success in fine-tuning performance, it has been shown to perform sub-optimally when used for linear probing. We propose an efficient patch-wise weighting derived from keypoint features which captures the local information and provides better context during SimMIM's reconstruction phase. Our method, KAMIM, improves the top-1 linear probing accuracy from 16.12% to 33.97%, and finetuning accuracy from 76.78% to 77.3% when tested on the ImageNet-1K dataset with a ViT-B when trained for the same number of epochs. We conduct extensive testing on different datasets, keypoint extractors, and model architectures and observe that patch-wise weighting augments linear probing performance for larger pretraining datasets. We also analyze the learned representations of a ViT-B trained using KAMIM and observe that they behave similar to contrastive learning with regard to its behavior, with longer attention distances and homogenous self-attention across layers. Our code is publicly available at this https URL.

Title: Privacy-preserving gradient-based fair federated learning

• Authors: Janis Adamek, Moritz Schulze Darup
• Subjects: cs.LG, cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2407.13881
• Pdf URL: https://arxiv.org/pdf/2407.13881
• Copy Paste: [[2407.13881]] Privacy-preserving gradient-based fair federated learning(https://arxiv.org/abs/2407.13881)
• Keywords: privacy, federate, fair
• Abstract: Federated learning (FL) schemes allow multiple participants to collaboratively train neural networks without the need to directly share the underlying data.However, in early schemes, all participants eventually obtain the same model. Moreover, the aggregation is typically carried out by a third party, who obtains combined gradients or weights, which may reveal the model. These downsides underscore the demand for fair and privacy-preserving FL schemes. Here, collaborative fairness asks for individual model quality depending on the individual data contribution. Privacy is demanded with respect to any kind of data outsourced to the third party. Now, there already exist some approaches aiming for either fair or privacy-preserving FL and a few works even address both features. In our paper, we build upon these seminal works and present a novel, fair and privacy-preserving FL scheme. Our approach, which mainly relies on homomorphic encryption, stands out for exclusively using local gradients. This increases the usability in comparison to state-of-the-art approaches and thereby opens the door to applications in control.

Title: Attention in SRAM on Tenstorrent Grayskull

• Authors: Moritz Thüning
• Subjects: cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2407.13885
• Pdf URL: https://arxiv.org/pdf/2407.13885
• Copy Paste: [[2407.13885]] Attention in SRAM on Tenstorrent Grayskull(https://arxiv.org/abs/2407.13885)
• Keywords: transformer
• Abstract: When implementations of the Transformer's self-attention layer utilize SRAM instead of DRAM, they can achieve significant speedups. The Tenstorrent Grayskull architecture provides a large SRAM, distributed across a grid of cores. This work presents a fused kernel for Grayskull, that exclusively utilizes its large SRAM by combining matrix multiplication, attention score scaling and Softmax operations. Additionally, a dedicated Softmax kernel utilizing the SRAM and a CPU implementation serving as a baseline are presented. The Softmax operation consumes most of the runtime in the computation of attention weights from queries and keys on Grayskull. The speedup of the dedicated Softmax kernel compared to the CPU implementation is up to $10 \times$, and the Softmax implementation inside the fused kernel is approximately $1.8 \times$ faster than the dedicated Softmax kernel. The time and memory complexity of all implementations is quadratic in sequence length. Currently, the Grayskull e150 is approximately $30 \times$ cheaper for the general public than an Nvidia H100 PCIe (a state-of-the-art GPU) and offers approximately $1.5 \times$ more SRAM.

Title: Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Dataset

• Authors: Yi Sheng, Junhuan Yang, Jinyang Li, James Alaina, Xiaowei Xu, Yiyu Shi, Jingtong Hu, Weiwen Jiang, Lei Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13896
• Pdf URL: https://arxiv.org/pdf/2407.13896
• Copy Paste: [[2407.13896]] Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Dataset(https://arxiv.org/abs/2407.13896)
• Keywords: fair
• Abstract: As Artificial Intelligence (AI) increasingly integrates into our daily lives, fairness has emerged as a critical concern, particularly in medical AI, where datasets often reflect inherent biases due to social factors like the underrepresentation of marginalized communities and socioeconomic barriers to data collection. Traditional approaches to mitigating these biases have focused on data augmentation and the development of fairness-aware training algorithms. However, this paper argues that the architecture of neural networks, a core component of Machine Learning (ML), plays a crucial role in ensuring fairness. We demonstrate that addressing fairness effectively requires a holistic approach that simultaneously considers data, algorithms, and architecture. Utilizing Automated ML (AutoML) technology, specifically Neural Architecture Search (NAS), we introduce a novel framework, BiaslessNAS, designed to achieve fair outcomes in analyzing skin lesion datasets. BiaslessNAS incorporates fairness considerations at every stage of the NAS process, leading to the identification of neural networks that are not only more accurate but also significantly fairer. Our experiments show that BiaslessNAS achieves a 2.55% increase in accuracy and a 65.50% improvement in fairness compared to traditional NAS methods, underscoring the importance of integrating fairness into neural network architecture for better outcomes in medical AI applications.

Title: Crafting Efficient Fine-Tuning Strategies for Large Language Models

• Authors: Michael Oliver, Guan Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13906
• Pdf URL: https://arxiv.org/pdf/2407.13906
• Copy Paste: [[2407.13906]] Crafting Efficient Fine-Tuning Strategies for Large Language Models(https://arxiv.org/abs/2407.13906)
• Keywords: extraction, large language model
• Abstract: This paper addresses the challenges of efficiently fine-tuning large language models (LLMs) by exploring data efficiency and hyperparameter optimization. We investigate the minimum data required for effective fine-tuning and propose a novel hyperparameter optimization method that leverages early-stage model performance. Our experiments demonstrate that fine-tuning with as few as 200 samples can improve model accuracy from 70\% to 88\% in a product attribute extraction task. We identify a saturation point of approximately 6,500 samples, beyond which additional data yields diminishing returns. Our proposed bayesian hyperparameter optimization method, which evaluates models at 20\% of total training time, correlates strongly with final model performance, with 4 out of 5 top early-stage models remaining in the top 5 at completion. This approach led to a 2\% improvement in accuracy over baseline models when evaluated on an independent test set. These findings offer actionable insights for practitioners, potentially reducing computational load and dependency on extensive datasets while enhancing overall performance of fine-tuned LLMs.

Title: Improving Malware Detection with Adversarial Domain Adaptation and Control Flow Graphs

• Authors: Adrian Shuai Li, Arun Iyengar, Ashish Kundu, Elisa Bertino
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.13918
• Pdf URL: https://arxiv.org/pdf/2407.13918
• Copy Paste: [[2407.13918]] Improving Malware Detection with Adversarial Domain Adaptation and Control Flow Graphs(https://arxiv.org/abs/2407.13918)
• Keywords: robust
• Abstract: In the application of deep learning for malware classification, it is crucial to account for the prevalence of malware evolution, which can cause trained classifiers to fail on drifted malware. Existing solutions to combat concept drift use active learning: they select new samples for analysts to label, and then retrain the classifier with the new labels. Our key finding is, the current retraining techniques do not achieve optimal results. These models overlook that updating the model with scarce drifted samples requires learning features that remain consistent across pre-drift and post-drift data. Furthermore, the model should be capable of disregarding specific features that, while beneficial for classification of pre-drift data, are absent in post-drift data, thereby preventing prediction degradation. In this paper, we propose a method that learns retained information in malware control flow graphs post-drift by leveraging graph neural network with adversarial domain adaptation. Our approach considers drift-invariant features within assembly instructions and flow of code execution. We further propose building blocks for more robust evaluation of drift adaptation techniques that computes statistically distant malware clusters. Our approach is compared with the previously published training methods in active learning systems, and the other domain adaptation technique. Our approach demonstrates a significant enhancement in predicting unseen malware family in a binary classification task and predicting drifted malware families in a multi-class setting. In addition, we assess alternative malware representations. The best results are obtained when our adaptation method is applied to our graph representations.

Title: DuoFormer: Leveraging Hierarchical Visual Representations by Local and Global Attention

• Authors: Xiaoya Tang, Bodong Zhang, Beatrice S. Knudsen, Tolga Tasdizen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13920
• Pdf URL: https://arxiv.org/pdf/2407.13920
• Copy Paste: [[2407.13920]] DuoFormer: Leveraging Hierarchical Visual Representations by Local and Global Attention(https://arxiv.org/abs/2407.13920)
• Keywords: extraction, transformer
• Abstract: We here propose a novel hierarchical transformer model that adeptly integrates the feature extraction capabilities of Convolutional Neural Networks (CNNs) with the advanced representational potential of Vision Transformers (ViTs). Addressing the lack of inductive biases and dependence on extensive training datasets in ViTs, our model employs a CNN backbone to generate hierarchical visual representations. These representations are then adapted for transformer input through an innovative patch tokenization. We also introduce a 'scale attention' mechanism that captures cross-scale dependencies, complementing patch attention to enhance spatial understanding and preserve global perception. Our approach significantly outperforms baseline models on small and medium-sized medical datasets, demonstrating its efficiency and generalizability. The components are designed as plug-and-play for different CNN architectures and can be adapted for multiple applications. The code is available at this https URL.

Title: Synthetic Counterfactual Faces

• Authors: Guruprasad V Ramesh, Harrison Rosenberg, Ashish Hooda, Kassem Fawaz
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13922
• Pdf URL: https://arxiv.org/pdf/2407.13922
• Copy Paste: [[2407.13922]] Synthetic Counterfactual Faces(https://arxiv.org/abs/2407.13922)
• Keywords: robust, biometric, fair, explainability, generative
• Abstract: Computer vision systems have been deployed in various applications involving biometrics like human faces. These systems can identify social media users, search for missing persons, and verify identity of individuals. While computer vision models are often evaluated for accuracy on available benchmarks, more annotated data is necessary to learn about their robustness and fairness against semantic distributional shifts in input data, especially in face data. Among annotated data, counterfactual examples grant strong explainability characteristics. Because collecting natural face data is prohibitively expensive, we put forth a generative AI-based framework to construct targeted, counterfactual, high-quality synthetic face data. Our synthetic data pipeline has many use cases, including face recognition systems sensitivity evaluations and image understanding system probes. The pipeline is validated with multiple user studies. We showcase the efficacy of our face generation pipeline on a leading commercial vision model. We identify facial attributes that cause vision systems to fail.

Title: BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization

• Authors: Ahmed Allam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13928
• Pdf URL: https://arxiv.org/pdf/2407.13928
• Copy Paste: [[2407.13928]] BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization(https://arxiv.org/abs/2407.13928)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have become pivotal in advancing natural language processing, yet their potential to perpetuate biases poses significant concerns. This paper introduces a new framework employing Direct Preference Optimization (DPO) to mitigate gender, racial, and religious biases in LLM-generated English text. By developing a loss function that favors less biased over biased completions, our approach cultivates a preference for respectful and non-discriminatory language in LLMs. We also contribute a manually designed dataset for training LLMs to recognize and correct biases. This dataset encompasses a diverse range of prompts paired with both biased and unbiased completions. Implementing this approach on the Microsoft Phi-2 model, we demonstrate substantial reductions in biased outputs as our model outperforms the baseline model on almost all bias benchmarks. Our model also achieves better performance compared to other open-source models on most benchmarks. By reducing biases in the language generated by the model, our study marks a significant step towards developing more ethical and socially responsible LLMs. We publicly release BiasDPO dataset on HuggingFace.

Title: RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark

• Authors: Yuan-Hao Ho, Jen-Hao Cheng, Sheng Yao Kuan, Zhongyu Jiang, Wenhao Chai, Hsiang-Wei Huang, Chih-Lung Lin, Jenq-Neng Hwang
• Subjects: cs.CV, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.13930
• Pdf URL: https://arxiv.org/pdf/2407.13930
• Copy Paste: [[2407.13930]] RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark(https://arxiv.org/abs/2407.13930)
• Keywords: privacy
• Abstract: Traditional methods for human localization and pose estimation (HPE), which mainly rely on RGB images as an input modality, confront substantial limitations in real-world applications due to privacy concerns. In contrast, radar-based HPE methods emerge as a promising alternative, characterized by distinctive attributes such as through-wall recognition and privacy-preserving, rendering the method more conducive to practical deployments. This paper presents a Radar Tensor-based human pose (RT-Pose) dataset and an open-source benchmarking framework. The RT-Pose dataset comprises 4D radar tensors, LiDAR point clouds, and RGB images, and is collected for a total of 72k frames across 240 sequences with six different complexity-level actions. The 4D radar tensor provides raw spatio-temporal information, differentiating it from other radar point cloud-based datasets. We develop an annotation process using RGB images and LiDAR point clouds to accurately label 3D human skeletons. In addition, we propose HRRadarPose, the first single-stage architecture that extracts the high-resolution representation of 4D radar tensors in 3D space to aid human keypoint estimation. HRRadarPose outperforms previous radar-based HPE work on the RT-Pose benchmark. The overall HRRadarPose performance on the RT-Pose dataset, as reflected in a mean per joint position error (MPJPE) of 9.91cm, indicates the persistent challenges in achieving accurate HPE in complex real-world scenarios. RT-Pose is available at this https URL.

Title: Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction

• Authors: Suma Bailis, Jane Friedhoff, Feiyang Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13943
• Pdf URL: https://arxiv.org/pdf/2407.13943
• Copy Paste: [[2407.13943]] Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction(https://arxiv.org/abs/2407.13943)
• Keywords: large language model
• Abstract: This paper introduces Werewolf Arena, a novel framework for evaluating large language models (LLMs) through the lens of the classic social deduction game, Werewolf. In Werewolf Arena, LLMs compete against each other, navigating the game's complex dynamics of deception, deduction, and persuasion. The framework introduces a dynamic turn-taking system based on bidding, mirroring real-world discussions where individuals strategically choose when to speak. We demonstrate the framework's utility through an arena-style tournament featuring Gemini and GPT models. Our results reveal distinct strengths and weaknesses in the models' strategic reasoning and communication. These findings highlight Werewolf Arena's potential as a challenging and scalable LLM benchmark.

Title: FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking

• Authors: Zhuoer Wang, Leonardo F. R. Ribeiro, Alexandros Papangelis, Rohan Mukherjee, Tzu-Yen Wang, Xinyan Zhao, Arijit Biswas, James Caverlee, Angeliki Metallinou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13945
• Pdf URL: https://arxiv.org/pdf/2407.13945
• Copy Paste: [[2407.13945]] FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking(https://arxiv.org/abs/2407.13945)
• Keywords: large language model
• Abstract: API call generation is the cornerstone of large language models' tool-using ability that provides access to the larger world. However, existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user's request. To address these limitations, we propose an output-side optimization approach called FANTASE. Two of the unique contributions of FANTASE are its State-Tracked Constrained Decoding (SCD) and Reranking components. SCD dynamically incorporates appropriate API constraints in the form of Token Search Trie for efficient and guaranteed generation faithfulness with respect to the API documentation. The Reranking component efficiently brings in the supervised signal by leveraging a lightweight model as the discriminator to rerank the beam-searched candidate generations of the large language model. We demonstrate the superior performance of FANTASE in API call generation accuracy, inference efficiency, and context efficiency with DSTC8 and API Bank datasets.

Title: BRSR-OpGAN: Blind Radar Signal Restoration using Operational Generative Adversarial Network

• Authors: Muhammad Uzair Zahid, Serkan Kiranyaz, Alper Yildirim, Moncef Gabbouj
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.13949
• Pdf URL: https://arxiv.org/pdf/2407.13949
• Copy Paste: [[2407.13949]] BRSR-OpGAN: Blind Radar Signal Restoration using Operational Generative Adversarial Network(https://arxiv.org/abs/2407.13949)
• Keywords: generative
• Abstract: Objective: Many studies on radar signal restoration in the literature focus on isolated restoration problems, such as denoising over a certain type of noise, while ignoring other types of artifacts. Additionally, these approaches usually assume a noisy environment with a limited set of fixed signal-to-noise ratio (SNR) levels. However, real-world radar signals are often corrupted by a blend of artifacts, including but not limited to unwanted echo, sensor noise, intentional jamming, and interference, each of which can vary in type, severity, and duration. This study introduces Blind Radar Signal Restoration using an Operational Generative Adversarial Network (BRSR-OpGAN), which uses a dual domain loss in the temporal and spectral domains. This approach is designed to improve the quality of radar signals, regardless of the diversity and intensity of the corruption. Methods: The BRSR-OpGAN utilizes 1D Operational GANs, which use a generative neuron model specifically optimized for blind restoration of corrupted radar signals. This approach leverages GANs' flexibility to adapt dynamically to a wide range of artifact characteristics. Results: The proposed approach has been extensively evaluated using a well-established baseline and a newly curated extended dataset called the Blind Radar Signal Restoration (BRSR) dataset. This dataset was designed to simulate real-world conditions and includes a variety of artifacts, each varying in severity. The evaluation shows an average SNR improvement over 15.1 dB and 14.3 dB for the baseline and BRSR datasets, respectively. Finally, even on resource-constrained platforms, the proposed approach can be applied in real-time.

Title: The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations

• Authors: Tyler LaBonte, John C. Hill, Xinchen Zhang, Vidya Muthukumar, Abhishek Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13957
• Pdf URL: https://arxiv.org/pdf/2407.13957
• Copy Paste: [[2407.13957]] The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations(https://arxiv.org/abs/2407.13957)
• Keywords: robust
• Abstract: Modern machine learning models are prone to over-reliance on spurious correlations, which can often lead to poor performance on minority groups. In this paper, we identify surprising and nuanced behavior of finetuned models on worst-group accuracy via comprehensive experiments on four well-established benchmarks across vision and language tasks. We first show that the commonly used class-balancing techniques of mini-batch upsampling and loss upweighting can induce a decrease in worst-group accuracy (WGA) with training epochs, leading to performance no better than without class-balancing. While in some scenarios, removing data to create a class-balanced subset is more effective, we show this depends on group structure and propose a mixture method which can outperform both techniques. Next, we show that scaling pretrained models is generally beneficial for worst-group accuracy, but only in conjuction with appropriate class-balancing. Finally, we identify spectral imbalance in finetuning features as a potential source of group disparities -- minority group covariance matrices incur a larger spectral norm than majority groups once conditioned on the classes. Our results show more nuanced interactions of modern finetuned models with group robustness than was previously known. Our code is available at this https URL.

Title: Continual Learning for Remote Physiological Measurement: Minimize Forgetting and Simplify Inference

• Authors: Qian Liang, Yan Chen, Yang Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13974
• Pdf URL: https://arxiv.org/pdf/2407.13974
• Copy Paste: [[2407.13974]] Continual Learning for Remote Physiological Measurement: Minimize Forgetting and Simplify Inference(https://arxiv.org/abs/2407.13974)
• Keywords: fair
• Abstract: Remote photoplethysmography (rPPG) has gained significant attention in recent years for its ability to extract physiological signals from facial videos. While existing rPPG measurement methods have shown satisfactory performance in intra-dataset and cross-dataset scenarios, they often overlook the incremental learning scenario, where training data is presented sequentially, resulting in the issue of catastrophic forgetting. Meanwhile, most existing class incremental learning approaches are unsuitable for rPPG measurement. In this paper, we present a novel method named ADDP to tackle continual learning for rPPG measurement. We first employ adapter to efficiently finetune the model on new tasks. Then we design domain prototypes that are more applicable to rPPG signal regression than commonly used class prototypes. Based on these prototypes, we propose a feature augmentation strategy to consolidate the past knowledge and an inference simplification strategy to convert potentially forgotten tasks into familiar ones for the model. To evaluate ADDP and enable fair comparisons, we create the first continual learning protocol for rPPG measurement. Comprehensive experiments demonstrate the effectiveness of our method for rPPG continual learning. Source code is available at \url{this https URL}

Title: Personalized Privacy Protection Mask Against Unauthorized Facial Recognition

• Authors: Ka-Ho Chow, Sihao Hu, Tiansheng Huang, Ling Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13975
• Pdf URL: https://arxiv.org/pdf/2407.13975
• Copy Paste: [[2407.13975]] Personalized Privacy Protection Mask Against Unauthorized Facial Recognition(https://arxiv.org/abs/2407.13975)
• Keywords: privacy, protect, attack, robust
• Abstract: Face recognition (FR) can be abused for privacy intrusion. Governments, private companies, or even individual attackers can collect facial images by web scraping to build an FR system identifying human faces without their consent. This paper introduces Chameleon, which learns to generate a user-centric personalized privacy protection mask, coined as P3-Mask, to protect facial images against unauthorized FR with three salient features. First, we use a cross-image optimization to generate one P3-Mask for each user instead of tailoring facial perturbation for each facial image of a user. It enables efficient and instant protection even for users with limited computing resources. Second, we incorporate a perceptibility optimization to preserve the visual quality of the protected facial images. Third, we strengthen the robustness of P3-Mask against unknown FR models by integrating focal diversity-optimized ensemble learning into the mask generation process. Extensive experiments on two benchmark datasets show that Chameleon outperforms three state-of-the-art methods with instant protection and minimal degradation of image quality. Furthermore, Chameleon enables cost-effective FR authorization using the P3-Mask as a personalized de-obfuscation key, and it demonstrates high resilience against adaptive adversaries.

Title: PlacidDreamer: Advancing Harmony in Text-to-3D Generation

• Authors: Shuo Huang, Shikun Sun, Zixuan Wang, Xiaoyu Qin, Yanmin Xiong, Yuan Zhang, Pengfei Wan, Di Zhang, Jia Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.13976
• Pdf URL: https://arxiv.org/pdf/2407.13976
• Copy Paste: [[2407.13976]] PlacidDreamer: Advancing Harmony in Text-to-3D Generation(https://arxiv.org/abs/2407.13976)
• Keywords: diffusion
• Abstract: Recently, text-to-3D generation has attracted significant attention, resulting in notable performance enhancements. Previous methods utilize end-to-end 3D generation models to initialize 3D Gaussians, multi-view diffusion models to enforce multi-view consistency, and text-to-image diffusion models to refine details with score distillation algorithms. However, these methods exhibit two limitations. Firstly, they encounter conflicts in generation directions since different models aim to produce diverse 3D assets. Secondly, the issue of over-saturation in score distillation has not been thoroughly investigated and solved. To address these limitations, we propose PlacidDreamer, a text-to-3D framework that harmonizes initialization, multi-view generation, and text-conditioned generation with a single multi-view diffusion model, while simultaneously employing a novel score distillation algorithm to achieve balanced saturation. To unify the generation direction, we introduce the Latent-Plane module, a training-friendly plug-in extension that enables multi-view diffusion models to provide fast geometry reconstruction for initialization and enhanced multi-view images to personalize the text-to-image diffusion model. To address the over-saturation problem, we propose to view score distillation as a multi-objective optimization problem and introduce the Balanced Score Distillation algorithm, which offers a Pareto Optimal solution that achieves both rich details and balanced saturation. Extensive experiments validate the outstanding capabilities of our PlacidDreamer. The code is available at \url{this https URL}.

Title: Double Gradient Reversal Network for Single-Source Domain Generalization in Multi-mode Fault Diagnosis

• Authors: Guangqiang Li, M. Amine Atoui, Xiangshun Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.13978
• Pdf URL: https://arxiv.org/pdf/2407.13978
• Copy Paste: [[2407.13978]] Double Gradient Reversal Network for Single-Source Domain Generalization in Multi-mode Fault Diagnosis(https://arxiv.org/abs/2407.13978)
• Keywords: extraction
• Abstract: Domain generalization achieves fault diagnosis on unseen modes. In process industrial systems, fault samples are limited, and only single-mode fault data can be obtained. Extracting domain-invariant fault features from single-mode data for unseen mode fault diagnosis poses challenges. Existing methods utilize a generator module to simulate samples of unseen modes. However, multi-mode samples contain complex spatiotemporal information, which brings significant difficulties to accurate sample generation. Therefore, double gradient reversal network (DGRN) is proposed. First, the model is pre-trained to acquire fault knowledge from the single seen mode. Then, pseudo-fault feature generation strategy is designed by Adaptive instance normalization, to simulate fault features of unseen mode. The dual adversarial training strategy is created to enhance the diversity of pseudo-fault features, which models unseen modes with significant distribution differences. Subsequently, domain-invariant feature extraction strategy is constructed by contrastive learning and adversarial learning. This strategy extracts common features of faults and helps multi-mode fault diagnosis. Finally, the experiments were conducted on Tennessee Eastman process and continuous stirred-tank reactor. The experiments demonstrate that DGRN achieves high classification accuracy on unseen modes while maintaining a small model size.

Title: Reexamining Racial Disparities in Automatic Speech Recognition Performance: The Role of Confounding by Provenance

• Authors: Changye Li, Trevor Cohen, Serguei Pakhomov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.13982
• Pdf URL: https://arxiv.org/pdf/2407.13982
• Copy Paste: [[2407.13982]] Reexamining Racial Disparities in Automatic Speech Recognition Performance: The Role of Confounding by Provenance(https://arxiv.org/abs/2407.13982)
• Keywords: fair
• Abstract: Automatic speech recognition (ASR) models trained on large amounts of audio data are now widely used to convert speech to written text in a variety of applications from video captioning to automated assistants used in healthcare and other domains. As such, it is important that ASR models and their use is fair and equitable. Prior work examining the performance of commercial ASR systems on the Corpus of Regional African American Language (CORAAL) demonstrated significantly worse ASR performance on African American English (AAE). The current study seeks to understand the factors underlying this disparity by examining the performance of the current state-of-the-art neural network based ASR system (Whisper, OpenAI) on the CORAAL dataset. Two key findings have been identified as a result of the current study. The first confirms prior findings of significant dialectal variation even across neighboring communities, and worse ASR performance on AAE that can be improved to some extent with fine-tuning of ASR models. The second is a novel finding not discussed in prior work on CORAAL: differences in audio recording practices within the dataset have a significant impact on ASR accuracy resulting in a ``confounding by provenance'' effect in which both language use and recording quality differ by study location. These findings highlight the need for further systematic investigation to disentangle the effects of recording quality and inherent linguistic diversity when examining the fairness and bias present in neural ASR models, as any bias in ASR accuracy may have negative downstream effects on disparities in various domains of life in which ASR technology is used.

Title: Enhancing Data-Limited Graph Neural Networks by Actively Distilling Knowledge from Large Language Models

• Authors: Quan Li, Tianxiang Zhao, Lingwei Chen, Junjie Xu, Suhang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13989
• Pdf URL: https://arxiv.org/pdf/2407.13989
• Copy Paste: [[2407.13989]] Enhancing Data-Limited Graph Neural Networks by Actively Distilling Knowledge from Large Language Models(https://arxiv.org/abs/2407.13989)
• Keywords: large language model
• Abstract: Graphs have emerged as critical data structures for content analysis in various domains, such as social network analysis, bioinformatics, and recommendation systems. Node classification, a fundamental task in this context, is typically tackled using graph neural networks (GNNs). Unfortunately, conventional GNNs still face challenges in scenarios with few labeled nodes, despite the prevalence of few-shot node classification tasks in real-world applications. To address this challenge, various approaches have been proposed, including graph meta-learning, transfer learning, and methods based on Large Language Models (LLMs). However, traditional meta-learning and transfer learning methods often require prior knowledge from base classes or fail to exploit the potential advantages of unlabeled nodes. Meanwhile, LLM-based methods may overlook the zero-shot capabilities of LLMs and rely heavily on the quality of generated contexts. In this paper, we propose a novel approach that integrates LLMs and GNNs, leveraging the zero-shot inference and reasoning capabilities of LLMs and employing a Graph-LLM-based active learning paradigm to enhance GNNs' performance. Extensive experiments demonstrate the effectiveness of our model in improving node classification accuracy with considerably limited labeled data, surpassing state-of-the-art baselines by significant margins.

Title: RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering

• Authors: Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, Vittorio Castelli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.13998
• Pdf URL: https://arxiv.org/pdf/2407.13998
• Copy Paste: [[2407.13998]] RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering(https://arxiv.org/abs/2407.13998)
• Keywords: robust, large language model
• Abstract: Question answering based on retrieval augmented generation (RAG-QA) is an important research topic in NLP and has a wide range of real-world applications. However, most existing datasets for this task are either constructed using a single source corpus or consist of short extractive answers, which fall short of evaluating large language model (LLM) based RAG-QA systems on cross-domain generalization. To address these limitations, we create Long-form RobustQA (LFRQA), a new dataset comprising human-written long-form answers that integrate short extractive answers from multiple documents into a single, coherent narrative, covering 26K queries and large corpora across seven different domains. We further propose RAG-QA Arena by directly comparing model-generated answers against LFRQA's answers using LLMs as evaluators. We show via extensive experiments that RAG-QA Arena and human judgments on answer quality are highly correlated. Moreover, only 41.3% of the most competitive LLM's answers are preferred to LFRQA's answers, demonstrating RAG-QA Arena as a challenging evaluation platform for future research.

Title: Component Selection for Craft Assembly Tasks

• Authors: Vitor Hideyo Isume (1), Takuya Kiyokawa (1), Natsuki Yamanobe (2), Yukiyasu Domae (2), Weiwei Wan (1), Kensuke Harada (1 and 2) ((1) Osaka University, (2) AIST)
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.14001
• Pdf URL: https://arxiv.org/pdf/2407.14001
• Copy Paste: [[2407.14001]] Component Selection for Craft Assembly Tasks(https://arxiv.org/abs/2407.14001)
• Keywords: segmentation
• Abstract: Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labelled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.

Title: Investigating the Indirect Object Identification circuit in Mamb

• Authors: Danielle Ensign, Adrià Garriga-Alonso
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14008
• Pdf URL: https://arxiv.org/pdf/2407.14008
• Copy Paste: [[2407.14008]] Investigating the Indirect Object Identification circuit in Mamb(https://arxiv.org/abs/2407.14008)
• Keywords: interpretability, transformer
• Abstract: How well will current interpretability techniques generalize to future models? A relevant case study is Mamba, a recent recurrent architecture with scaling comparable to Transformers. We adapt pre-Mamba techniques to Mamba and partially reverse-engineer the circuit responsible for the Indirect Object Identification (IOI) task. Our techniques provide evidence that 1) Layer 39 is a key bottleneck, 2) Convolutions in layer 39 shift names one position forward, and 3) The name entities are stored linearly in Layer 39's SSM. Finally, we adapt an automatic circuit discovery tool, positional Edge Attribution Patching, to identify a Mamba IOI circuit. Our contributions provide initial evidence that circuit-based mechanistic interpretability tools work well for the Mamba architecture.

Title: Scale Disparity of Instances in Interactive Point Cloud Segmentation

• Authors: Chenrui Han, Xuan Yu, Yuxuan Xie, Yili Liu, Sitong Mao, Shunbo Zhou, Rong Xiong, Yue Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14009
• Pdf URL: https://arxiv.org/pdf/2407.14009
• Copy Paste: [[2407.14009]] Scale Disparity of Instances in Interactive Point Cloud Segmentation(https://arxiv.org/abs/2407.14009)
• Keywords: transformer, segmentation
• Abstract: Interactive point cloud segmentation has become a pivotal task for understanding 3D scenes, enabling users to guide segmentation models with simple interactions such as clicks, therefore significantly reducing the effort required to tailor models to diverse scenarios and new categories. However, in the realm of interactive segmentation, the meaning of instance diverges from that in instance segmentation, because users might desire to segment instances of both thing and stuff categories that vary greatly in scale. Existing methods have focused on thing categories, neglecting the segmentation of stuff categories and the difficulties arising from scale disparity. To bridge this gap, we propose ClickFormer, an innovative interactive point cloud segmentation model that accurately segments instances of both thing and stuff categories. We propose a query augmentation module to augment click queries by a global query sampling strategy, thus maintaining consistent performance across different instance scales. Additionally, we employ global attention in the query-voxel transformer to mitigate the risk of generating false positives, along with several other network structure improvements to further enhance the model's segmentation performance. Experiments demonstrate that ClickFormer outperforms existing interactive point cloud segmentation methods across both indoor and outdoor datasets, providing more accurate segmentation results with fewer user clicks in an open-world setting.

Title: Semi-supervised reference-based sketch extraction using a contrastive learning framework

• Authors: Chang Wook Seo, Amirsaman Ashtari, Junyong Noh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14026
• Pdf URL: https://arxiv.org/pdf/2407.14026
• Copy Paste: [[2407.14026]] Semi-supervised reference-based sketch extraction using a contrastive learning framework(https://arxiv.org/abs/2407.14026)
• Keywords: extraction
• Abstract: Sketches reflect the drawing style of individual artists; therefore, it is important to consider their unique styles when extracting sketches from color images for various applications. Unfortunately, most existing sketch extraction methods are designed to extract sketches of a single style. Although there have been some attempts to generate various style sketches, the methods generally suffer from two limitations: low quality results and difficulty in training the model due to the requirement of a paired dataset. In this paper, we propose a novel multi-modal sketch extraction method that can imitate the style of a given reference sketch with unpaired data training in a semi-supervised manner. Our method outperforms state-of-the-art sketch extraction methods and unpaired image translation methods in both quantitative and qualitative evaluations.

Title: HeCiX: Integrating Knowledge Graphs and Large Language Models for Biomedical Research

• Authors: Prerana Sanjay Kulkarni, Muskaan Jain, Disha Sheshanarayana, Srinivasan Parthiban
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14030
• Pdf URL: https://arxiv.org/pdf/2407.14030
• Copy Paste: [[2407.14030]] HeCiX: Integrating Knowledge Graphs and Large Language Models for Biomedical Research(https://arxiv.org/abs/2407.14030)
• Keywords: large language model
• Abstract: Despite advancements in drug development strategies, 90% of clinical trials fail. This suggests overlooked aspects in target validation and drug optimization. In order to address this, we introduce HeCiX-KG, Hetionet-Clinicaltrials neXus Knowledge Graph, a novel fusion of data from this http URL and Hetionet in a single knowledge graph. HeCiX-KG combines data on previously conducted clinical trials from this http URL, and domain expertise on diseases and genes from Hetionet. This offers a thorough resource for clinical researchers. Further, we introduce HeCiX, a system that uses LangChain to integrate HeCiX-KG with GPT-4, and increase its usability. HeCiX shows high performance during evaluation against a range of clinically relevant issues, proving this model to be promising for enhancing the effectiveness of clinical research. Thus, this approach provides a more holistic view of clinical trials and existing biological data.

Title: Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance

• Authors: Yongshuo Zhu, Lu Li, Keyan Chen, Chenyang Liu, Fugen Zhou, Zhenwei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14032
• Pdf URL: https://arxiv.org/pdf/2407.14032
• Copy Paste: [[2407.14032]] Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance(https://arxiv.org/abs/2407.14032)
• Keywords: robust, extraction, large language model
• Abstract: Remote sensing image change captioning (RSICC) aims to articulate the changes in objects of interest within bi-temporal remote sensing images using natural language. Given the limitations of current RSICC methods in expressing general features across multi-temporal and spatial scenarios, and their deficiency in providing granular, robust, and precise change descriptions, we introduce a novel change captioning (CC) method based on the foundational knowledge and semantic guidance, which we term Semantic-CC. Semantic-CC alleviates the dependency of high-generalization algorithms on extensive annotations by harnessing the latent knowledge of foundation models, and it generates more comprehensive and accurate change descriptions guided by pixel-level semantics from change detection (CD). Specifically, we propose a bi-temporal SAM-based encoder for dual-image feature extraction; a multi-task semantic aggregation neck for facilitating information interaction between heterogeneous tasks; a straightforward multi-scale change detection decoder to provide pixel-level semantic guidance; and a change caption decoder based on the large language model (LLM) to generate change description sentences. Moreover, to ensure the stability of the joint training of CD and CC, we propose a three-stage training strategy that supervises different tasks at various stages. We validate the proposed method on the LEVIR-CC and LEVIR-CD datasets. The experimental results corroborate the complementarity of CD and CC, demonstrating that Semantic-CC can generate more accurate change descriptions and achieve optimal performance across both tasks.

Title: Generative Language Model for Catalyst Discovery

• Authors: Dong Hyeon Mok, Seoin Back
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14040
• Pdf URL: https://arxiv.org/pdf/2407.14040
• Copy Paste: [[2407.14040]] Generative Language Model for Catalyst Discovery(https://arxiv.org/abs/2407.14040)
• Keywords: transformer, generative
• Abstract: Discovery of novel and promising materials is a critical challenge in the field of chemistry and material science, traditionally approached through methodologies ranging from trial-and-error to machine learning-driven inverse design. Recent studies suggest that transformer-based language models can be utilized as material generative models to expand chemical space and explore materials with desired properties. In this work, we introduce the Catalyst Generative Pretrained Transformer (CatGPT), trained to generate string representations of inorganic catalyst structures from a vast chemical space. CatGPT not only demonstrates high performance in generating valid and accurate catalyst structures but also serves as a foundation model for generating desired types of catalysts by fine-tuning with sparse and specified datasets. As an example, we fine-tuned the pretrained CatGPT using a binary alloy catalyst dataset designed for screening two-electron oxygen reduction reaction (2e-ORR) catalyst and generate catalyst structures specialized for 2e-ORR. Our work demonstrates the potential of language models as generative tools for catalyst discovery.

Title: Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization

• Authors: Zipeng Qi, Lichen Bai, Haoyi Xiong, and Zeke Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14041
• Pdf URL: https://arxiv.org/pdf/2407.14041
• Copy Paste: [[2407.14041]] Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization(https://arxiv.org/abs/2407.14041)
• Keywords: diffusion, generative
• Abstract: Diffusion models that can generate high-quality data from randomly sampled Gaussian noises have become the mainstream generative method in both academia and industry. Are randomly sampled Gaussian noises equally good for diffusion models? While a large body of works tried to understand and improve diffusion models, previous works overlooked the possibility to select or optimize the sampled noise the possibility of selecting or optimizing sampled noises for improving diffusion models. In this paper, we mainly made three contributions. First, we report that not all noises are created equally for diffusion models. We are the first to hypothesize and empirically observe that the generation quality of diffusion models significantly depend on the noise inversion stability. This naturally provides us a noise selection method according to the inversion stability. Second, we further propose a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. Our method is the first one that works on noise space to generally improve generated results without fine-tuning diffusion models. Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo, in terms of human preference and other objective evaluation metrics. For example, the human preference winning rates of noise selection and noise optimization over the baselines can be up to 57% and 72.5%, respectively, on DrawBench.

Title: Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

• Authors: Yuhang Chen, Chenxing Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14043
• Pdf URL: https://arxiv.org/pdf/2407.14043
• Copy Paste: [[2407.14043]] Kinematics-based 3D Human-Object Interaction Reconstruction from Single View(https://arxiv.org/abs/2407.14043)
• Keywords: robust
• Abstract: Reconstructing 3D human-object interaction (HOI) from single-view RGB images is challenging due to the absence of depth information and potential occlusions. Existing methods simply predict the body poses merely rely on network training on some indoor datasets, which cannot guarantee the rationality of the results if some body parts are invisible due to occlusions that appear easily. Inspired by the end-effector localization task in robotics, we propose a kinematics-based method that can drive the joints of human body to the human-object contact regions accurately. After an improved forward kinematics algorithm is proposed, the Multi-Layer Perceptron is introduced into the solution of inverse kinematics process to determine the poses of joints, which achieves precise results than the commonly-used numerical methods in robotics. Besides, a Contact Region Recognition Network (CRRNet) is also proposed to robustly determine the contact regions using a single-view video. Experimental results demonstrate that our method outperforms the state-of-the-art on benchmark BEHAVE. Additionally, our approach shows good portability and can be seamlessly integrated into other methods for optimizations.

Title: ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?

• Authors: Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, Daniel Fried
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14044
• Pdf URL: https://arxiv.org/pdf/2407.14044
• Copy Paste: [[2407.14044]] ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?(https://arxiv.org/abs/2407.14044)
• Keywords: large language model
• Abstract: Although large language models (LLMs) have been largely successful in generating functionally correct programs, conditioning models to produce efficient solutions while ensuring correctness remains a challenge. Further, unreliability in benchmarking code efficiency is a hurdle across varying hardware specifications for popular interpreted languages such as Python. In this paper, we present ECCO, a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. On ECCO, we adapt and thoroughly investigate the three most promising existing LLM-based approaches: in-context learning, iterative refinement with execution or NL feedback, and fine-tuning conditioned on execution and editing history. While most methods degrade functional correctness and moderately increase program efficiency, we find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency. We release our benchmark to support future work on LLM-based generation of efficient code.

Title: OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking

• Authors: Zekun Qian, Ruize Han, Wei Feng, Junhui Hou, Linqi Song, Song Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14047
• Pdf URL: https://arxiv.org/pdf/2407.14047
• Copy Paste: [[2407.14047]] OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking(https://arxiv.org/abs/2407.14047)
• Keywords: generative
• Abstract: We study a novel yet practical problem of open-corpus multi-object tracking (OCMOT), which extends the MOT into localizing, associating, and recognizing generic-category objects of both seen (base) and unseen (novel) classes, but without the category text list as prompt. To study this problem, the top priority is to build a benchmark. In this work, we build OCTrackB, a large-scale and comprehensive benchmark, to provide a standard evaluation platform for the OCMOT problem. Compared to previous datasets, OCTrackB has more abundant and balanced base/novel classes and the corresponding samples for evaluation with less bias. We also propose a new multi-granularity recognition metric to better evaluate the generative object recognition in OCMOT. By conducting the extensive benchmark evaluation, we report and analyze the results of various state-of-the-art methods, which demonstrate the rationale of OCMOT, as well as the usefulness and advantages of OCTrackB.

Title: Prompted Aspect Key Point Analysis for Quantitative Review Summarization

• Authors: An Quang Tang, Xiuzhen Zhang, Minh Ngoc Dinh, Erik Cambria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14049
• Pdf URL: https://arxiv.org/pdf/2407.14049
• Copy Paste: [[2407.14049]] Prompted Aspect Key Point Analysis for Quantitative Review Summarization(https://arxiv.org/abs/2407.14049)
• Keywords: large language model
• Abstract: Key Point Analysis (KPA) aims for quantitative summarization that provides key points (KPs) as succinct textual summaries and quantities measuring their prevalence. KPA studies for arguments and reviews have been reported in the literature. A majority of KPA studies for reviews adopt supervised learning to extract short sentences as KPs before matching KPs to review comments for quantification of KP prevalence. Recent abstractive approaches still generate KPs based on sentences, often leading to KPs with overlapping and hallucinated opinions, and inaccurate quantification. In this paper, we propose Prompted Aspect Key Point Analysis (PAKPA) for quantitative review summarization. PAKPA employs aspect sentiment analysis and prompted in-context learning with Large Language Models (LLMs) to generate and quantify KPs grounded in aspects for business entities, which achieves faithful KPs with accurate quantification, and removes the need for large amounts of annotated data for supervised training. Experiments on the popular review dataset Yelp and the aspect-oriented review summarization dataset SPACE show that our framework achieves state-of-the-art performance. Source code and data are available at: this https URL

Title: PointRegGPT: Boosting 3D Point Cloud Registration using Generative Point-Cloud Pairs for Training

• Authors: Suyi Chen, Hao Xu, Haipeng Li, Kunming Luo, Guanghui Liu, Chi-Wing Fu, Ping Tan, Shuaicheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14054
• Pdf URL: https://arxiv.org/pdf/2407.14054
• Copy Paste: [[2407.14054]] PointRegGPT: Boosting 3D Point Cloud Registration using Generative Point-Cloud Pairs for Training(https://arxiv.org/abs/2407.14054)
• Keywords: diffusion, generative
• Abstract: Data plays a crucial role in training learning-based methods for 3D point cloud registration. However, the real-world dataset is expensive to build, while rendering-based synthetic data suffers from domain gaps. In this work, we present PointRegGPT, boosting 3D point cloud registration using generative point-cloud pairs for training. Given a single depth map, we first apply a random camera motion to re-project it into a target depth map. Converting them to point clouds gives a training pair. To enhance the data realism, we formulate a generative model as a depth inpainting diffusion to process the target depth map with the re-projected source depth map as the condition. Also, we design a depth correction module to alleviate artifacts caused by point penetration during the re-projection. To our knowledge, this is the first generative approach that explores realistic data generation for indoor point cloud registration. When equipped with our approach, several recent algorithms can improve their performance significantly and achieve SOTA consistently on two common benchmarks. The code and dataset will be released on this https URL.

Title: Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings

• Authors: Praveen Srinivasa Varadhan, Ashwin Sankar, Giri Raju, Mitesh M. Khapra
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.14056
• Pdf URL: https://arxiv.org/pdf/2407.14056
• Copy Paste: [[2407.14056]] Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings(https://arxiv.org/abs/2407.14056)
• Keywords: fair
• Abstract: We release Rasa, the first multilingual expressive TTS dataset for any Indian language, which contains 10 hours of neutral speech and 1-3 hours of expressive speech for each of the 6 Ekman emotions covering 3 languages: Assamese, Bengali, & Tamil. Our ablation studies reveal that just 1 hour of neutral and 30 minutes of expressive data can yield a Fair system as indicated by MUSHRA scores. Increasing neutral data to 10 hours, with minimal expressive data, significantly enhances expressiveness. This offers a practical recipe for resource-constrained languages, prioritizing easily obtainable neutral data alongside smaller amounts of expressive data. We show the importance of syllabically balanced data and pooling emotions to enhance expressiveness. We also highlight challenges in generating specific emotions, e.g., fear and surprise.

Title: LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference

• Authors: Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, Mahyar Najibi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14057
• Pdf URL: https://arxiv.org/pdf/2407.14057
• Copy Paste: [[2407.14057]] LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference(https://arxiv.org/abs/2407.14057)
• Keywords: transformer, large language model
• Abstract: The inference of transformer-based large language models consists of two sequential stages: 1) a prefilling stage to compute the KV cache of prompts and generate the first token, and 2) a decoding stage to generate subsequent tokens. For long prompts, the KV cache must be computed for all tokens during the prefilling stage, which can significantly increase the time needed to generate the first token. Consequently, the prefilling stage may become a bottleneck in the generation process. An open question remains whether all prompt tokens are essential for generating the first token. To answer this, we introduce a novel method, LazyLLM, that selectively computes the KV for tokens important for the next token prediction in both the prefilling and decoding stages. Contrary to static pruning approaches that prune the prompt at once, LazyLLM allows language models to dynamically select different subsets of tokens from the context in different generation steps, even though they might be pruned in previous steps. Extensive experiments on standard datasets across various tasks demonstrate that LazyLLM is a generic method that can be seamlessly integrated with existing language models to significantly accelerate the generation without fine-tuning. For instance, in the multi-document question-answering task, LazyLLM accelerates the prefilling stage of the LLama 2 7B model by 2.34x while maintaining accuracy.

Title: Refining Tuberculosis Detection in CXR Imaging: Addressing Bias in Deep Neural Networks via Interpretability

• Authors: Özgür Acar Güler, Manuel Günther, André Anjos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14064
• Pdf URL: https://arxiv.org/pdf/2407.14064
• Copy Paste: [[2407.14064]] Refining Tuberculosis Detection in CXR Imaging: Addressing Bias in Deep Neural Networks via Interpretability(https://arxiv.org/abs/2407.14064)
• Keywords: interpretability
• Abstract: Automatic classification of active tuberculosis from chest X-ray images has the potential to save lives, especially in low- and mid-income countries where skilled human experts can be scarce. Given the lack of available labeled data to train such systems and the unbalanced nature of publicly available datasets, we argue that the reliability of deep learning models is limited, even if they can be shown to obtain perfect classification accuracy on the test data. One way of evaluating the reliability of such systems is to ensure that models use the same regions of input images for predictions as medical experts would. In this paper, we show that pre-training a deep neural network on a large-scale proxy task, as well as using mixed objective optimization network (MOON), a technique to balance different classes during pre-training and fine-tuning, can improve the alignment of decision foundations between models and experts, as compared to a model directly trained on the target dataset. At the same time, these approaches keep perfect classification accuracy according to the area under the receiver operating characteristic curve (AUROC) on the test set, and improve generalization on an independent, unseen dataset. For the purpose of reproducibility, our source code is made available online.

Title: MSCT: Addressing Time-Varying Confounding with Marginal Structural Causal Transformer for Counterfactual Post-Crash Traffic Prediction

• Authors: Shuang Li, Ziyuan Pu, Nan Zhang, Duxin Chen, Lu Dong, Daniel J. Graham, Yinhai Wang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.14065
• Pdf URL: https://arxiv.org/pdf/2407.14065
• Copy Paste: [[2407.14065]] MSCT: Addressing Time-Varying Confounding with Marginal Structural Causal Transformer for Counterfactual Post-Crash Traffic Prediction(https://arxiv.org/abs/2407.14065)
• Keywords: transformer
• Abstract: Traffic crashes profoundly impede traffic efficiency and pose economic challenges. Accurate prediction of post-crash traffic status provides essential information for evaluating traffic perturbations and developing effective solutions. Previous studies have established a series of deep learning models to predict post-crash traffic conditions, however, these correlation-based methods cannot accommodate the biases caused by time-varying confounders and the heterogeneous effects of crashes. The post-crash traffic prediction model needs to estimate the counterfactual traffic speed response to hypothetical crashes under various conditions, which demonstrates the necessity of understanding the causal relationship between traffic factors. Therefore, this paper presents the Marginal Structural Causal Transformer (MSCT), a novel deep learning model designed for counterfactual post-crash traffic prediction. To address the issue of time-varying confounding bias, MSCT incorporates a structure inspired by Marginal Structural Models and introduces a balanced loss function to facilitate learning of invariant causal features. The proposed model is treatment-aware, with a specific focus on comprehending and predicting traffic speed under hypothetical crash intervention strategies. In the absence of ground-truth data, a synthetic data generation procedure is proposed to emulate the causal mechanism between traffic speed, crashes, and covariates. The model is validated using both synthetic and real-world data, demonstrating that MSCT outperforms state-of-the-art models in multi-step-ahead prediction performance. This study also systematically analyzes the impact of time-varying confounding bias and dataset distribution on model performance, contributing valuable insights into counterfactual prediction for intelligent transportation systems.

Title: Stable-Hair: Real-World Hair Transfer via Diffusion Model

• Authors: Yuxuan Zhang, Qing Zhang, Yiren Song, Jiaming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14078
• Pdf URL: https://arxiv.org/pdf/2407.14078
• Copy Paste: [[2407.14078]] Stable-Hair: Real-World Hair Transfer via Diffusion Model(https://arxiv.org/abs/2407.14078)
• Keywords: robust, diffusion
• Abstract: Current hair transfer methods struggle to handle diverse and intricate hairstyles, thus limiting their applicability in real-world scenarios. In this paper, we propose a novel diffusion-based hair transfer framework, named \textit{Stable-Hair}, which robustly transfers a wide range of real-world hairstyles onto user-provided faces for virtual hair try-on. To achieve this goal, our Stable-Hair framework is designed as a two-stage pipeline. In the first stage, we train a Bald Converter alongside stable diffusion to remove hair from the user-provided face images, resulting in bald images. In the second stage, we specifically designed three modules: a Hair Extractor, a Latent IdentityNet, and Hair Cross-Attention Layers to transfer the target hairstyle with highly detailed and high-fidelity to the bald image. Specifically, the Hair Extractor is trained to encode reference images with the desired hairstyles. To preserve the consistency of identity content and background between the source images and the transfer results, we employ a Latent IdentityNet to encode the source images. With the assistance of our Hair Cross-Attention Layers in the U-Net, we can accurately and precisely transfer the highly detailed and high-fidelity hairstyle to the bald image. Extensive experiments have demonstrated that our approach delivers state-of-the-art (SOTA) results among existing hair transfer methods. Project page: \textcolor{red}{\url{this https URL}}

Title: An Improved Method for Class-specific Keyword Extraction: A Case Study in the German Business Registry

• Authors: Stephen Meisenbacher, Tim Schopf, Weixin Yan, Patrick Holl, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14085
• Pdf URL: https://arxiv.org/pdf/2407.14085
• Copy Paste: [[2407.14085]] An Improved Method for Class-specific Keyword Extraction: A Case Study in the German Business Registry(https://arxiv.org/abs/2407.14085)
• Keywords: extraction
• Abstract: The task of $\textit{keyword extraction}$ is often an important initial step in unsupervised information extraction, forming the basis for tasks such as topic modeling or document classification. While recent methods have proven to be quite effective in the extraction of keywords, the identification of $\textit{class-specific}$ keywords, or only those pertaining to a predefined class, remains challenging. In this work, we propose an improved method for class-specific keyword extraction, which builds upon the popular $\textbf{KeyBERT}$ library to identify only keywords related to a class described by $\textit{seed keywords}$. We test this method using a dataset of German business registry entries, where the goal is to classify each business according to an economic sector. Our results reveal that our method greatly improves upon previous approaches, setting a new standard for $\textit{class-specific}$ keyword extraction.

Title: Temporal Correlation Meets Embedding: Towards a 2nd Generation of JDE-based Real-Time Multi-Object Tracking

• Authors: Yunfei Zhang, Chao Liang, Jin Gao, Zhipeng Zhang, Weiming Hu, Stephen Maybank, Xue Zhou, Liang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14086
• Pdf URL: https://arxiv.org/pdf/2407.14086
• Copy Paste: [[2407.14086]] Temporal Correlation Meets Embedding: Towards a 2nd Generation of JDE-based Real-Time Multi-Object Tracking(https://arxiv.org/abs/2407.14086)
• Keywords: robust, extraction
• Abstract: Joint Detection and Embedding(JDE) trackers have demonstrated excellent performance in Multi-Object Tracking(MOT) tasks by incorporating the extraction of appearance features as auxiliary tasks through embedding Re-Identification task(ReID) into the detector, achieving a balance between inference speed and tracking performance. However, solving the competition between the detector and the feature extractor has always been a challenge. Also, the issue of directly embedding the ReID task into MOT has remained unresolved. The lack of high discriminability in appearance features results in their limited utility. In this paper, we propose a new learning approach using cross-correlation to capture temporal information of objects. The feature extraction network is no longer trained solely on appearance features from each frame but learns richer motion features by utilizing feature heatmaps from consecutive frames, addressing the challenge of inter-class feature similarity. Furthermore, we apply our learning approach to a more lightweight feature extraction network, and treat the feature matching scores as strong cues rather than auxiliary cues, employing a appropriate weight calculation to reflect the compatibility between our obtained features and the MOT task. Our tracker, named TCBTrack, achieves state-of-the-art performance on multiple public benchmarks, i.e., MOT17, MOT20, and DanceTrack datasets. Specifically, on the DanceTrack test set, we achieve 56.8 HOTA, 58.1 IDF1 and 92.5 MOTA, making it the best online tracker that can achieve real-time performance. Comparative evaluations with other trackers prove that our tracker achieves the best balance between speed, robustness and accuracy.

Title: Score Normalization for Demographic Fairness in Face Recognition

• Authors: Yu Linghu, Tiago des Freitas Pereira, Christophe Ecabert, Sébastien Marcel, Manuel Günther
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14087
• Pdf URL: https://arxiv.org/pdf/2407.14087
• Copy Paste: [[2407.14087]] Score Normalization for Demographic Fairness in Face Recognition(https://arxiv.org/abs/2407.14087)
• Keywords: security, biometric, fair
• Abstract: Fair biometric algorithms have similar verification performance across different demographic groups given a single decision threshold. Unfortunately, for state-of-the-art face recognition networks, score distributions differ between demographics. Contrary to work that tries to align those distributions by extra training or fine-tuning, we solely focus on score post-processing methods. As proved, well-known sample-centered score normalization techniques, Z-norm and T-norm, do not improve fairness for high-security operating points. Thus, we extend the standard Z/T-norm to integrate demographic information in normalization. Additionally, we investigate several possibilities to incorporate cohort similarities for both genuine and impostor pairs per demographic to improve fairness across different operating points. We run experiments on two datasets with different demographics (gender and ethnicity) and show that our techniques generally improve the overall fairness of five state-of-the-art pre-trained face recognition networks, without downgrading verification performance. We also indicate that an equal contribution of False Match Rate (FMR) and False Non-Match Rate (FNMR) in fairness evaluation is required for the highest gains. Code and protocols are available.

Title: On the Robustness of Fully-Spiking Neural Networks in Open-World Scenarios using Forward-Only Learning Algorithms

• Authors: Erik B. Terres-Escudero, Javier Del Ser, Aitor Martínez-Seras, Pablo Garcia-Bringas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14097
• Pdf URL: https://arxiv.org/pdf/2407.14097
• Copy Paste: [[2407.14097]] On the Robustness of Fully-Spiking Neural Networks in Open-World Scenarios using Forward-Only Learning Algorithms(https://arxiv.org/abs/2407.14097)
• Keywords: robust, explainability
• Abstract: In the last decade, Artificial Intelligence (AI) models have rapidly integrated into production pipelines propelled by their excellent modeling performance. However, the development of these models has not been matched by advancements in algorithms ensuring their safety, failing to guarantee robust behavior against Out-of-Distribution (OoD) inputs outside their learning domain. Furthermore, there is a growing concern with the sustainability of AI models and their required energy consumption in both training and inference phases. To mitigate these issues, this work explores the use of the Forward-Forward Algorithm (FFA), a biologically plausible alternative to Backpropagation, adapted to the spiking domain to enhance the overall energy efficiency of the model. By capitalizing on the highly expressive topology emerging from the latent space of models trained with FFA, we develop a novel FF-SCP algorithm for OoD Detection. Our approach measures the likelihood of a sample belonging to the in-distribution (ID) data by using the distance from the latent representation of samples to class-representative manifolds. Additionally, to provide deeper insights into our OoD pipeline, we propose a gradient-free attribution technique that highlights the features of a sample pushing it away from the distribution of any class. Multiple experiments using our spiking FFA adaptation demonstrate that the achieved accuracy levels are comparable to those seen in analog networks trained via back-propagation. Furthermore, OoD detection experiments on multiple datasets prove that FF-SCP outperforms avant-garde OoD detectors within the spiking domain in terms of several metrics used in this area. We also present a qualitative analysis of our explainability technique, exposing the precision by which the method detects OoD features, such as embedded artifacts or missing regions.

Title: Zero-Shot Underwater Gesture Recognition

• Authors: Sandipan Sarma, Gundameedi Sai Ram Mohan, Hariansh Sehgal, Arijit Sur
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14103
• Pdf URL: https://arxiv.org/pdf/2407.14103
• Copy Paste: [[2407.14103]] Zero-Shot Underwater Gesture Recognition(https://arxiv.org/abs/2407.14103)
• Keywords: transformer, generative
• Abstract: Hand gesture recognition allows humans to interact with machines non-verbally, which has a huge application in underwater exploration using autonomous underwater vehicles. Recently, a new gesture-based language called CADDIAN has been devised for divers, and supervised learning methods have been applied to recognize the gestures with high accuracy. However, such methods fail when they encounter unseen gestures in real time. In this work, we advocate the need for zero-shot underwater gesture recognition (ZSUGR), where the objective is to train a model with visual samples of gestures from a few ``seen'' classes only and transfer the gained knowledge at test time to recognize semantically-similar unseen gesture classes as well. After discussing the problem and dataset-specific challenges, we propose new seen-unseen splits for gesture classes in CADDY dataset. Then, we present a two-stage framework, where a novel transformer learns strong visual gesture cues and feeds them to a conditional generative adversarial network that learns to mimic feature distribution. We use the trained generator as a feature synthesizer for unseen classes, enabling zero-shot learning. Extensive experiments demonstrate that our method outperforms the existing zero-shot techniques. We conclude by providing useful insights into our framework and suggesting directions for future research.

Title: GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentation

• Authors: Florian Chabot, Nicolas Granger, Guillaume Lapouge
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14108
• Pdf URL: https://arxiv.org/pdf/2407.14108
• Copy Paste: [[2407.14108]] GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentation(https://arxiv.org/abs/2407.14108)
• Keywords: transformer, segmentation
• Abstract: The Bird's-eye View (BeV) representation is widely used for 3D perception from multi-view camera images. It allows to merge features from different cameras into a common space, providing a unified representation of the 3D scene. The key component is the view transformer, which transforms image views into the BeV. However, actual view transformer methods based on geometry or cross-attention do not provide a sufficiently detailed representation of the scene, as they use a sub-sampling of the 3D space that is non-optimal for modeling the fine structures of the environment. In this paper, we propose GaussianBeV, a novel method for transforming image features to BeV by finely representing the scene using a set of 3D gaussians located and oriented in 3D space. This representation is then splattered to produce the BeV feature map by adapting recent advances in 3D representation rendering based on gaussian splatting. GaussianBeV is the first approach to use this 3D gaussian modeling and 3D scene rendering process online, i.e. without optimizing it on a specific scene and directly integrated into a single stage model for BeV scene understanding. Experiments show that the proposed representation is highly effective and place GaussianBeV as the new state-of-the-art on the BeV semantic segmentation task on the nuScenes dataset.

Title: MC-PanDA: Mask Confidence for Panoptic Domain Adaptation

• Authors: Ivan Martinović, Josip Šarić, Siniša Šegvić
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14110
• Pdf URL: https://arxiv.org/pdf/2407.14110
• Copy Paste: [[2407.14110]] MC-PanDA: Mask Confidence for Panoptic Domain Adaptation(https://arxiv.org/abs/2407.14110)
• Keywords: transformer, segmentation
• Abstract: Domain adaptive panoptic segmentation promises to resolve the long tail of corner cases in natural scene understanding. Previous state of the art addresses this problem with cross-task consistency, careful system-level optimization and heuristic improvement of teacher predictions. In contrast, we propose to build upon remarkable capability of mask transformers to estimate their own prediction uncertainty. Our method avoids noise amplification by leveraging fine-grained confidence of panoptic teacher predictions. In particular, we modulate the loss with mask-wide confidence and discourage back-propagation in pixels with uncertain teacher or confident student. Experimental evaluation on standard benchmarks reveals a substantial contribution of the proposed selection techniques. We report 47.4 PQ on Synthia to Cityscapes, which corresponds to an improvement of 6.2 percentage points over the state of the art. The source code is available at this https URL.

Title: AuditNet: A Conversational AI-based Security Assistant [DEMO]

• Authors: Shohreh Deldari, Mohammad Goudarzi, Aditya Joshi, Arash Shaghaghi, Simon Finn, Flora D. Salim, Sanjay Jha
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14116
• Pdf URL: https://arxiv.org/pdf/2407.14116
• Copy Paste: [[2407.14116]] AuditNet: A Conversational AI-based Security Assistant [DEMO](https://arxiv.org/abs/2407.14116)
• Keywords: security, large language model
• Abstract: In the age of information overload, professionals across various fields face the challenge of navigating vast amounts of documentation and ever-evolving standards. Ensuring compliance with standards, regulations, and contractual obligations is a critical yet complex task across various professional fields. We propose a versatile conversational AI assistant framework designed to facilitate compliance checking on the go, in diverse domains, including but not limited to network infrastructure, legal contracts, educational standards, environmental regulations, and government policies. By leveraging retrieval-augmented generation using large language models, our framework automates the review, indexing, and retrieval of relevant, context-aware information, streamlining the process of verifying adherence to established guidelines and requirements. This AI assistant not only reduces the manual effort involved in compliance checks but also enhances accuracy and efficiency, supporting professionals in maintaining high standards of practice and ensuring regulatory compliance in their respective fields. We propose and demonstrate AuditNet, the first conversational AI security assistant designed to assist IoT network security experts by providing instant access to security standards, policies, and regulations.

Title: Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation

• Authors: Jinda Lu, Shuo Wang, Yanbin Hao, Haifeng Liu, Xiang Wang, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14117
• Pdf URL: https://arxiv.org/pdf/2407.14117
• Copy Paste: [[2407.14117]] Rethinking Visual Content Refinement in Low-Shot CLIP Adaptation(https://arxiv.org/abs/2407.14117)
• Keywords: robust
• Abstract: Recent adaptations can boost the low-shot capability of Contrastive Vision-Language Pre-training (CLIP) by effectively facilitating knowledge transfer. However, these adaptation methods are usually operated on the global view of an input image, and thus biased perception of partial local details of the image. To solve this problem, we propose a Visual Content Refinement (VCR) before the adaptation calculation during the test stage. Specifically, we first decompose the test image into different scales to shift the feature extractor's attention to the details of the image. Then, we select the image view with the max prediction margin in each scale to filter out the noisy image views, where the prediction margins are calculated from the pre-trained CLIP model. Finally, we merge the content of the aforementioned selected image views based on their scales to construct a new robust representation. Thus, the merged content can be directly used to help the adapter focus on both global and local parts without any extra training parameters. We apply our method to 3 popular low-shot benchmark tasks with 13 datasets and achieve a significant improvement over state-of-the-art methods. For example, compared to the baseline (Tip-Adapter) on the few-shot classification task, our method achieves about 2\% average improvement for both training-free and training-need settings.

Title: Shape and Style GAN-based Multispectral Data Augmentation for Crop/Weed Segmentation in Precision Farming

• Authors: Mulham Fawakherji, Vincenzo Suriani, Daniele Nardi, Domenico Daniele Bloisi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14119
• Pdf URL: https://arxiv.org/pdf/2407.14119
• Copy Paste: [[2407.14119]] Shape and Style GAN-based Multispectral Data Augmentation for Crop/Weed Segmentation in Precision Farming(https://arxiv.org/abs/2407.14119)
• Keywords: segmentation
• Abstract: The use of deep learning methods for precision farming is gaining increasing interest. However, collecting training data in this application field is particularly challenging and costly due to the need of acquiring information during the different growing stages of the cultivation of interest. In this paper, we present a method for data augmentation that uses two GANs to create artificial images to augment the training data. To obtain a higher image quality, instead of re-creating the entire scene, we take original images and replace only the patches containing objects of interest with artificial ones containing new objects with different shapes and styles. In doing this, we take into account both the foreground (i.e., crop samples) and the background (i.e., the soil) of the patches. Quantitative experiments, conducted on publicly available datasets, demonstrate the effectiveness of the proposed approach. The source code and data discussed in this work are available as open source.

Title: Comparing and Contrasting Deep Learning Weather Prediction Backbones on Navier-Stokes and Atmospheric Dynamics

• Authors: Matthias Karlbauer, Danielle C. Maddix, Abdul Fatir Ansari, Boran Han, Gaurav Gupta, Yuyang Wang, Andrew Stuart, Michael W. Mahoney
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14129
• Pdf URL: https://arxiv.org/pdf/2407.14129
• Copy Paste: [[2407.14129]] Comparing and Contrasting Deep Learning Weather Prediction Backbones on Navier-Stokes and Atmospheric Dynamics(https://arxiv.org/abs/2407.14129)
• Keywords: transformer
• Abstract: Remarkable progress in the development of Deep Learning Weather Prediction (DLWP) models positions them to become competitive with traditional numerical weather prediction (NWP) models. Indeed, a wide number of DLWP architectures -- based on various backbones, including U-Net, Transformer, Graph Neural Network (GNN), and Fourier Neural Operator (FNO) -- have demonstrated their potential at forecasting atmospheric states. However, due to differences in training protocols, forecast horizons, and data choices, it remains unclear which (if any) of these methods and architectures are most suitable for weather forecasting and for future model development. Here, we step back and provide a detailed empirical analysis, under controlled conditions, comparing and contrasting the most prominent DLWP models, along with their backbones. We accomplish this by predicting synthetic two-dimensional incompressible Navier-Stokes and real-world global weather dynamics. In terms of accuracy, memory consumption, and runtime, our results illustrate various tradeoffs. For example, on synthetic data, we observe favorable performance of FNO; and on the real-world WeatherBench dataset, our results demonstrate the suitability of ConvLSTM and SwinTransformer for short-to-mid-ranged forecasts. For long-ranged weather rollouts of up to 365 days, we observe superior stability and physical soundness in architectures that formulate a spherical data representation, i.e., GraphCast and Spherical FNO. In addition, we observe that all of these model backbones ``saturate,'' i.e., none of them exhibit so-called neural scaling, which highlights an important direction for future work on these and related models.

Title: I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction

• Authors: Zaiqiao Meng, Hao Zhou, Yifang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14133
• Pdf URL: https://arxiv.org/pdf/2407.14133
• Copy Paste: [[2407.14133]] I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction(https://arxiv.org/abs/2407.14133)
• Keywords: robust
• Abstract: Visual Language Models (VLMs) are essential for various tasks, particularly visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing \VLMs{}' visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the \ours{} model, designed to enhance the visual spatial reasoning abilities of VLMS. ZeroVLM employs Zero-1-to-3, a 3D reconstruction model for obtaining different views of the input images and incorporates a prompting mechanism to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our \ours{} achieves up to 19.48% accuracy improvement, which indicates the effectiveness of the 3D reconstruction and prompting mechanisms of our ZeroVLM.

Title: Visual Text Generation in the Wild

• Authors: Yuanzhi Zhu, Jiawei Liu, Feiyu Gao, Wenyu Liu, Xinggang Wang, Peng Wang, Fei Huang, Cong Yao, Zhibo Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14138
• Pdf URL: https://arxiv.org/pdf/2407.14138
• Copy Paste: [[2407.14138]] Visual Text Generation in the Wild(https://arxiv.org/abs/2407.14138)
• Keywords: diffusion, generative, large language model
• Abstract: Recently, with the rapid advancements of generative models, the field of visual text generation has witnessed significant progress. However, it is still challenging to render high-quality text images in real-world scenarios, as three critical criteria should be satisfied: (1) Fidelity: the generated text images should be photo-realistic and the contents are expected to be the same as specified in the given conditions; (2) Reasonability: the regions and contents of the generated text should cohere with the scene; (3) Utility: the generated text images can facilitate related tasks (e.g., text detection and recognition). Upon investigation, we find that existing methods, either rendering-based or diffusion-based, can hardly meet all these aspects simultaneously, limiting their application range. Therefore, we propose in this paper a visual text generator (termed SceneVTG), which can produce high-quality text images in the wild. Following a two-stage paradigm, SceneVTG leverages a Multimodal Large Language Model to recommend reasonable text regions and contents across multiple scales and levels, which are used by a conditional diffusion model as conditions to generate text images. Extensive experiments demonstrate that the proposed SceneVTG significantly outperforms traditional rendering-based methods and recent diffusion-based methods in terms of fidelity and reasonability. Besides, the generated images provide superior utility for tasks involving text detection and text recognition. Code and datasets are available at AdvancedLiterateMachinery.

Title: Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation

• Authors: Zhengyuan Xie, Haiquan Lu, Jia-wen Xiao, Enguang Wang, Le Zhang, Xialei Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14142
• Pdf URL: https://arxiv.org/pdf/2407.14142
• Copy Paste: [[2407.14142]] Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation(https://arxiv.org/abs/2407.14142)
• Keywords: segmentation
• Abstract: Class incremental semantic segmentation aims to preserve old knowledge while learning new tasks, however, it is impeded by catastrophic forgetting and background shift issues. Prior works indicate the pivotal importance of initializing new classifiers and mainly focus on transferring knowledge from the background classifier or preparing classifiers for future classes, neglecting the flexibility and variance of new classifiers. In this paper, we propose a new classifier pre-tuning~(NeST) method applied before the formal training process, learning a transformation from old classifiers to generate new classifiers for initialization rather than directly tuning the parameters of new classifiers. Our method can make new classifiers align with the backbone and adapt to the new data, preventing drastic changes in the feature extractor when learning new classes. Besides, we design a strategy considering the cross-task class similarity to initialize matrices used in the transformation, helping achieve the stability-plasticity trade-off. Experiments on Pascal VOC 2012 and ADE20K datasets show that the proposed strategy can significantly improve the performance of previous methods. The code is available at \url{this https URL}.

Title: Where is the Testbed for my Federated Learning Research?

• Authors: Janez Božič, Amândio R. Faustino, Boris Radovič, Marco Canini, Veljko Pejović
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.14154
• Pdf URL: https://arxiv.org/pdf/2407.14154
• Copy Paste: [[2407.14154]] Where is the Testbed for my Federated Learning Research?(https://arxiv.org/abs/2407.14154)
• Keywords: federate
• Abstract: Progressing beyond centralized AI is of paramount importance, yet, distributed AI solutions, in particular various federated learning (FL) algorithms, are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a certain solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of FL client devices and different platforms, with different datasets and data distribution, all while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence, to name a few. In this paper, we present CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed configuration space, with a large number of heterogeneous edge devices, ranging from single-board computers to smartphones, and provides real-time collection and visualization of a variety of metrics through automatic instrumentation. According to our evaluation, porting FL algorithms to CoLExT requires minimal involvement from the developer, and the instrumentation introduces minimal resource usage overhead. Furthermore, through an initial investigation involving popular FL algorithms running on CoLExT, we reveal previously unknown trade-offs, inefficiencies, and programming bugs.

Title: EVLM: An Efficient Vision-Language Model for Visual Understanding

• Authors: Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, Changyi Liu, Dewen Fan, Huihui Xiao, Jiahong Wu, Fan Yang, Size Li, Di Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14177
• Pdf URL: https://arxiv.org/pdf/2407.14177
• Copy Paste: [[2407.14177]] EVLM: An Efficient Vision-Language Model for Visual Understanding(https://arxiv.org/abs/2407.14177)
• Keywords: large language model
• Abstract: In the field of multi-modal language models, the majority of methods are built on an architecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt, directly feeding it into the language models alongside textual tokens. However, when dealing with long sequences of visual signals or inputs such as videos, the self-attention mechanism of language models can lead to significant computational overhead. Additionally, using single-layer ViT features makes it challenging for large language models to perceive visual signals fully. This paper proposes an efficient multi-modal language model to minimize computational costs while enabling the model to perceive visual signals as comprehensively as possible. Our method primarily includes: (1) employing cross-attention to image-text interaction similar to Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of Experts (MoE) mechanism to enhance model effectiveness. Our model achieves competitive scores on public multi-modal benchmarks and performs well in tasks such as image captioning and video captioning.

Title: Automatic Classification of News Subjects in Broadcast News: Application to a Gender Bias Representation Analysis

• Authors: Valentin Pelloin, Lena Dodson, Émile Chapuis, Nicolas Hervé, David Doukhan
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.14180
• Pdf URL: https://arxiv.org/pdf/2407.14180
• Copy Paste: [[2407.14180]] Automatic Classification of News Subjects in Broadcast News: Application to a Gender Bias Representation Analysis(https://arxiv.org/abs/2407.14180)
• Keywords: large language model
• Abstract: This paper introduces a computational framework designed to delineate gender distribution biases in topics covered by French TV and radio news. We transcribe a dataset of 11.7k hours, broadcasted in 2023 on 21 French channels. A Large Language Model (LLM) is used in few-shot conversation mode to obtain a topic classification on those transcriptions. Using the generated LLM annotations, we explore the finetuning of a specialized smaller classification model, to reduce the computational cost. To evaluate the performances of these models, we construct and annotate a dataset of 804 dialogues. This dataset is made available free of charge for research purposes. We show that women are notably underrepresented in subjects such as sports, politics and conflicts. Conversely, on topics such as weather, commercials and health, women have more speaking time than their overall average across all subjects. We also observe representations differences between private and public service channels.

Title: Normative Diffusion Autoencoders: Application to Amyotrophic Lateral Sclerosis

• Authors: Ayodeji Ijishakin, Adamos Hadjasavilou, Ahmed Abdulaal, Nina Montana-Brown, Florence Townend, Edoardo Spinelli, Massimo Fillipi, Federica Agosta, James Cole, Andrea Malaspina
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14191
• Pdf URL: https://arxiv.org/pdf/2407.14191
• Copy Paste: [[2407.14191]] Normative Diffusion Autoencoders: Application to Amyotrophic Lateral Sclerosis(https://arxiv.org/abs/2407.14191)
• Keywords: diffusion, generative
• Abstract: Predicting survival in Amyotrophic Lateral Sclerosis (ALS) is a challenging task. Magnetic resonance imaging (MRI) data provide in vivo insight into brain health, but the low prevalence of the condition and resultant data scarcity limit training set sizes for prediction models. Survival models are further hindered by the subtle and often highly localised profile of ALS-related neurodegeneration. Normative models present a solution as they increase statistical power by leveraging large healthy cohorts. Separately, diffusion models excel in capturing the semantics embedded within images including subtle signs of accelerated brain ageing, which may help predict survival in ALS. Here, we combine the benefits of generative and normative modelling by introducing the normative diffusion autoencoder framework. To our knowledge, this is the first use of normative modelling within a diffusion autoencoder, as well as the first application of normative modelling to ALS. Our approach outperforms generative and non-generative normative modelling benchmarks in ALS prognostication, demonstrating enhanced predictive accuracy in the context of ALS survival prediction and normative modelling in general.

Title: LeKUBE: A Legal Knowledge Update BEnchmark

• Authors: Changyue Wang, Weihang Su, Hu Yiran, Qingyao Ai, Yueyue Wu, Cheng Luo, Yiqun Liu, Min Zhang, Shaoping Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14192
• Pdf URL: https://arxiv.org/pdf/2407.14192
• Copy Paste: [[2407.14192]] LeKUBE: A Legal Knowledge Update BEnchmark(https://arxiv.org/abs/2407.14192)
• Keywords: large language model
• Abstract: Recent advances in Large Language Models (LLMs) have significantly shaped the applications of AI in multiple fields, including the studies of legal intelligence. Trained on extensive legal texts, including statutes and legal documents, the legal LLMs can capture important legal knowledge/concepts effectively and provide important support for downstream legal applications such as legal consultancy. Yet, the dynamic nature of legal statutes and interpretations also poses new challenges to the use of LLMs in legal applications. Particularly, how to update the legal knowledge of LLMs effectively and efficiently has become an important research problem in practice. Existing benchmarks for evaluating knowledge update methods are mostly designed for the open domain and cannot address the specific challenges of the legal domain, such as the nuanced application of new legal knowledge, the complexity and lengthiness of legal regulations, and the intricate nature of legal reasoning. To address this gap, we introduce the Legal Knowledge Update BEnchmark, i.e. LeKUBE, which evaluates knowledge update methods for legal LLMs across five dimensions. Specifically, we categorize the needs of knowledge updates in the legal domain with the help of legal professionals, and then hire annotators from law schools to create synthetic updates to the Chinese Criminal and Civil Code as well as sets of questions of which the answers would change after the updates. Through a comprehensive evaluation of state-of-the-art knowledge update methods, we reveal a notable gap between existing knowledge update methods and the unique needs of the legal domain, emphasizing the need for further research and development of knowledge update mechanisms tailored for legal LLMs.

Title: Double-Shot 3D Shape Measurement with a Dual-Branch Network

• Authors: Mingyang Lei, Jingfan Fan, Long Shao, Hong Song, Deqiang Xiao, Danni Ai, Tianyu Fu, Ying Gu, Jian Yang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.14198
• Pdf URL: https://arxiv.org/pdf/2407.14198
• Copy Paste: [[2407.14198]] Double-Shot 3D Shape Measurement with a Dual-Branch Network(https://arxiv.org/abs/2407.14198)
• Keywords: transformer
• Abstract: The structured light (SL)-based 3D measurement techniques with deep learning have been widely studied, among which speckle projection profilometry (SPP) and fringe projection profilometry (FPP) are two popular methods. However, they generally use a single projection pattern for reconstruction, resulting in fringe order ambiguity or poor reconstruction accuracy. To alleviate these problems, we propose a parallel dual-branch Convolutional Neural Network (CNN)-Transformer network (PDCNet), to take advantage of convolutional operations and self-attention mechanisms for processing different SL modalities. Within PDCNet, a Transformer branch is used to capture global perception in the fringe images, while a CNN branch is designed to collect local details in the speckle images. To fully integrate complementary features, we design a double-stream attention aggregation module (DAAM) that consist of a parallel attention subnetwork for aggregating multi-scale spatial structure information. This module can dynamically retain local and global representations to the maximum extent. Moreover, an adaptive mixture density head with bimodal Gaussian distribution is proposed for learning a representation that is precise near discontinuities. Compared to the standard disparity regression strategy, this adaptive mixture head can effectively improves performance at object boundaries. Extensive experiments demonstrate that our method can reduce fringe order ambiguity while producing high-accuracy results on a self-made dataset. We also show that the proposed architecture reveals the potential in infrared-visible image fusion task.

Title: Bucketed Ranking-based Losses for Efficient Training of Object Detectors

• Authors: Feyza Yavuz, Baris Can Cam, Adnan Harun Dogan, Kemal Oksuz, Emre Akbas, Sinan Kalkan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14204
• Pdf URL: https://arxiv.org/pdf/2407.14204
• Copy Paste: [[2407.14204]] Bucketed Ranking-based Losses for Efficient Training of Object Detectors(https://arxiv.org/abs/2407.14204)
• Keywords: robust, transformer
• Abstract: Ranking-based loss functions, such as Average Precision Loss and Rank&Sort Loss, outperform widely used score-based losses in object detection. These loss functions better align with the evaluation criteria, have fewer hyperparameters, and offer robustness against the imbalance between positive and negative classes. However, they require pairwise comparisons among $P$ positive and $N$ negative predictions, introducing a time complexity of $\mathcal{O}(PN)$, which is prohibitive since $N$ is often large (e.g., $10^8$ in ATSS). Despite their advantages, the widespread adoption of ranking-based losses has been hindered by their high time and space complexities. In this paper, we focus on improving the efficiency of ranking-based loss functions. To this end, we propose Bucketed Ranking-based Losses which group negative predictions into $B$ buckets ($B \ll N$) in order to reduce the number of pairwise comparisons so that time complexity can be reduced. Our method enhances the time complexity, reducing it to $\mathcal{O}(\max (N \log(N), P^2))$. To validate our method and show its generality, we conducted experiments on 2 different tasks, 3 different datasets, 7 different detectors. We show that Bucketed Ranking-based (BR) Losses yield the same accuracy with the unbucketed versions and provide $2\times$ faster training on average. We also train, for the first time, transformer-based object detectors using ranking-based losses, thanks to the efficiency of our BR. When we train CoDETR, a state-of-the-art transformer-based object detector, using our BR Loss, we consistently outperform its original results over several different backbones. Code is available at this https URL

Title: Watermark Smoothing Attacks against Language Models

• Authors: Hongyan Chang, Hamed Hassani, Reza Shokri
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14206
• Pdf URL: https://arxiv.org/pdf/2407.14206
• Copy Paste: [[2407.14206]] Watermark Smoothing Attacks against Language Models(https://arxiv.org/abs/2407.14206)
• Keywords: attack, robust, watermark, large language model
• Abstract: Watermarking is a technique used to embed a hidden signal in the probability distribution of text generated by large language models (LLMs), enabling attribution of the text to the originating model. We introduce smoothing attacks and show that existing watermarking methods are not robust against minor modifications of text. An adversary can use weaker language models to smooth out the distribution perturbations caused by watermarks without significantly compromising the quality of the generated text. The modified text resulting from the smoothing attack remains close to the distribution of text that the original model (without watermark) would have produced. Our attack reveals a fundamental limitation of a wide range of watermarking techniques.

Title: Longhorn: State Space Models are Amortized Online Learners

• Authors: Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, Qiang Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14207
• Pdf URL: https://arxiv.org/pdf/2407.14207
• Copy Paste: [[2407.14207]] Longhorn: State Space Models are Amortized Online Learners(https://arxiv.org/abs/2407.14207)
• Keywords: transformer, large language model
• Abstract: The most fundamental capability of modern AI methods such as Large Language Models (LLMs) is the ability to predict the next token in a long sequence of tokens, known as ``sequence modeling." Although the Transformers model is the current dominant approach to sequence modeling, its quadratic computational cost with respect to sequence length is a significant drawback. State-space models (SSMs) offer a promising alternative due to their linear decoding efficiency and high parallelizability during training. However, existing SSMs often rely on seemingly ad hoc linear recurrence designs. In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. This approach links SSM design to formulating precise online learning objectives, with state transition rules derived from optimizing these objectives. Based on this insight, we introduce a novel deep SSM architecture based on the implicit update for optimizing an online regression objective. Our experimental results show that our models outperform state-of-the-art SSMs, including the Mamba model, on standard sequence modeling benchmarks and language modeling tasks.

Title: Unlearning Concepts from Text-to-Video Diffusion Models

• Authors: Shiqi Liu, Yihua Tan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14209
• Pdf URL: https://arxiv.org/pdf/2407.14209
• Copy Paste: [[2407.14209]] Unlearning Concepts from Text-to-Video Diffusion Models(https://arxiv.org/abs/2407.14209)
• Keywords: diffusion
• Abstract: With the advancement of computer vision and natural language processing, text-to-video generation, enabled by text-to-video diffusion models, has become more prevalent. These models are trained using a large amount of data from the internet. However, the training data often contain copyrighted content, including cartoon character icons and artist styles, private portraits, and unsafe videos. Since filtering the data and retraining the model is challenging, methods for unlearning specific concepts from text-to-video diffusion models have been investigated. However, due to the high computational complexity and relative large optimization scale, there is little work on unlearning methods for text-to-video diffusion models. We propose a novel concept-unlearning method by transferring the unlearning capability of the text encoder of text-to-image diffusion models to text-to-video diffusion models. Specifically, the method optimizes the text encoder using few-shot unlearning, where several generated images are used. We then use the optimized text encoder in text-to-video diffusion models to generate videos. Our method costs low computation resources and has small optimization scale. We discuss the generated videos after unlearning a concept. The experiments demonstrates that our method can unlearn copyrighted cartoon characters, artist styles, objects and people's facial characteristics. Our method can unlearn a concept within about 100 seconds on an RTX 3070. Since there was no concept unlearning method for text-to-video diffusion models before, we make concept unlearning feasible and more accessible in the text-to-video domain.

Title: Fair Overlap Number of Balls (Fair-ONB): A Data-Morphology-based Undersampling Method for Bias Reduction

• Authors: José Daniel Pascual-Triana, Alberto Fernández, Paulo Novais, Francisco Herrera
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14210
• Pdf URL: https://arxiv.org/pdf/2407.14210
• Copy Paste: [[2407.14210]] Fair Overlap Number of Balls (Fair-ONB): A Data-Morphology-based Undersampling Method for Bias Reduction(https://arxiv.org/abs/2407.14210)
• Keywords: protect, fair
• Abstract: Given the magnitude of data generation currently, both in quantity and speed, the use of machine learning is increasingly important. When data include protected features that might give rise to discrimination, special care must be taken. Data quality is critical in these cases, as biases in training data can be reflected in classification models. This has devastating consequences and fails to comply with current regulations. Data-Centric Artificial Intelligence proposes dataset modifications to improve its quality. Instance selection via undersampling can foster balanced learning of classes and protected feature values in the classifier. When such undersampling is done close to the decision boundary, the effect on the classifier would be bolstered. This work proposes Fair Overlap Number of Balls (Fair-ONB), an undersampling method that harnesses the data morphology of the different data groups (obtained from the combination of classes and protected feature values) to perform guided undersampling in the areas where they overlap. It employs attributes of the ball coverage of the groups, such as the radius, number of covered instances and density, to select the most suitable areas for undersampling and reduce bias. Results show that the Fair-ONB method reduces bias with low impact on the classifier's predictive performance.

Title: Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognition

• Authors: Suvajit Patra, Arkadip Maitra, Megha Tiwari, K. Kumaran, Swathy Prabhu, Swami Punyeshwarananda, Soumitra Samanta
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14224
• Pdf URL: https://arxiv.org/pdf/2407.14224
• Copy Paste: [[2407.14224]] Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognition(https://arxiv.org/abs/2407.14224)
• Keywords: robust
• Abstract: Automatic Sign Language (SL) recognition is an important task in the computer vision community. To build a robust SL recognition system, we need a considerable amount of data which is lacking particularly in Indian sign language (ISL). In this paper, we propose a large-scale isolated ISL dataset and a novel SL recognition model based on skeleton graph structure. The dataset covers 2,002 daily used common words in the deaf community recorded by 20 (10 male and 10 female) deaf adult signers (contains 40033 videos). We propose a SL recognition model namely Hierarchical Windowed Graph Attention Network (HWGAT) by utilizing the human upper body skeleton graph structure. The HWGAT tries to capture distinctive motions by giving attention to different body parts induced by the human skeleton graph structure. The utility of the proposed dataset and the usefulness of our model are evaluated through extensive experiments. We pre-trained the proposed model on the proposed dataset and fine-tuned it across different sign language datasets further boosting the performance of 1.10, 0.46, 0.78, and 6.84 percentage points on INCLUDE, LSA64, AUTSL and WLASL respectively compared to the existing state-of-the-art skeleton-based models.

Title: ETSCL: An Evidence Theory-Based Supervised Contrastive Learning Framework for Multi-modal Glaucoma Grading

• Authors: Zhiyuan Yang, Bo Zhang, Yufei Shi, Ningze Zhong, Johnathan Loh, Huihui Fang, Yanwu Xu, Si Yong Yeo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14230
• Pdf URL: https://arxiv.org/pdf/2407.14230
• Copy Paste: [[2407.14230]] ETSCL: An Evidence Theory-Based Supervised Contrastive Learning Framework for Multi-modal Glaucoma Grading(https://arxiv.org/abs/2407.14230)
• Keywords: extraction
• Abstract: Glaucoma is one of the leading causes of vision impairment. Digital imaging techniques, such as color fundus photography (CFP) and optical coherence tomography (OCT), provide quantitative and noninvasive methods for glaucoma diagnosis. Recently, in the field of computer-aided glaucoma diagnosis, multi-modality methods that integrate the CFP and OCT modalities have achieved greater diagnostic accuracy compared to single-modality methods. However, it remains challenging to extract reliable features due to the high similarity of medical images and the unbalanced multi-modal data distribution. Moreover, existing methods overlook the uncertainty estimation of different modalities, leading to unreliable predictions. To address these challenges, we propose a novel framework, namely ETSCL, which consists of a contrastive feature extraction stage and a decision-level fusion stage. Specifically, the supervised contrastive loss is employed to enhance the discriminative power in the feature extraction process, resulting in more effective features. In addition, we utilize the Frangi vesselness algorithm as a preprocessing step to incorporate vessel information to assist in the prediction. In the decision-level fusion stage, an evidence theory-based multi-modality classifier is employed to combine multi-source information with uncertainty estimation. Extensive experiments demonstrate that our method achieves state-of-the-art performance. The code is available at \url{this https URL}.

Title: Realistic Evaluation of Test-Time Adaptation Algorithms: Unsupervised Hyperparameter Selection

• Authors: Sebastian Cygert, Damian Sójka, Tomasz Trzciński, Bartłomiej Twardowski
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14231
• Pdf URL: https://arxiv.org/pdf/2407.14231
• Copy Paste: [[2407.14231]] Realistic Evaluation of Test-Time Adaptation Algorithms: Unsupervised Hyperparameter Selection(https://arxiv.org/abs/2407.14231)
• Keywords: robust
• Abstract: Test-Time Adaptation (TTA) has recently emerged as a promising strategy for tackling the problem of machine learning model robustness under distribution shifts by adapting the model during inference without access to any labels. Because of task difficulty, hyperparameters strongly influence the effectiveness of adaptation. However, the literature has provided little exploration into optimal hyperparameter selection. In this work, we tackle this problem by evaluating existing TTA methods using surrogate-based hp-selection strategies (which do not assume access to the test labels) to obtain a more realistic evaluation of their performance. We show that some of the recent state-of-the-art methods exhibit inferior performance compared to the previous algorithms when using our more realistic evaluation setup. Further, we show that forgetting is still a problem in TTA as the only method that is robust to hp-selection resets the model to the initial state at every step. We analyze different types of unsupervised selection strategies, and while they work reasonably well in most scenarios, the only strategies that work consistently well use some kind of supervision (either by a limited number of annotated test samples or by using pretraining data). Our findings underscore the need for further research with more rigorous benchmarking by explicitly stating model selection strategies, to facilitate which we open-source our code.

Title: Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

• Authors: Bo Yuan, Danpei Zhao, Zhuoran Liu, Wentao Li, Tian Li
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.14242
• Pdf URL: https://arxiv.org/pdf/2407.14242
• Copy Paste: [[2407.14242]] Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images(https://arxiv.org/abs/2407.14242)
• Keywords: segmentation
• Abstract: Continual learning (CL) breaks off the one-way training manner and enables a model to adapt to new data, semantics and tasks continuously. However, current CL methods mainly focus on single tasks. Besides, CL models are plagued by catastrophic forgetting and semantic drift since the lack of old data, which often occurs in remote-sensing interpretation due to the intricate fine-grained semantics. In this paper, we propose Continual Panoptic Perception (CPP), a unified continual learning model that leverages multi-task joint learning covering pixel-level classification, instance-level segmentation and image-level perception for universal interpretation in remote sensing images. Concretely, we propose a collaborative cross-modal encoder (CCE) to extract the input image features, which supports pixel classification and caption generation synchronously. To inherit the knowledge from the old model without exemplar memory, we propose a task-interactive knowledge distillation (TKD) method, which leverages cross-modal optimization and task-asymmetric pseudo-labeling (TPL) to alleviate catastrophic forgetting. Furthermore, we also propose a joint optimization mechanism to achieve end-to-end multi-modal panoptic perception. Experimental results on the fine-grained panoptic perception dataset validate the effectiveness of the proposed model, and also prove that joint optimization can boost sub-task CL efficiency with over 13\% relative improvement on panoptic quality.

Title: Conditioning Chat-GPT for information retrieval: the Unipa-GPT case study

• Authors: Irene Siragusa, Roberto Pirrone
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14246
• Pdf URL: https://arxiv.org/pdf/2407.14246
• Copy Paste: [[2407.14246]] Conditioning Chat-GPT for information retrieval: the Unipa-GPT case study(https://arxiv.org/abs/2407.14246)
• Keywords: large language model
• Abstract: This paper illustrates the architecture and training of Unipa-GPT, a chatbot relying on a Large Language Model, developed for assisting students in choosing a bachelor/master degree course at the University of Palermo. Unipa-GPT relies on gpt-3.5-turbo, it was presented in the context of the European Researchers' Night (SHARPER night). In our experiments we adopted both the Retrieval Augmented Generation (RAG) approach and fine-tuning to develop the system. The whole architecture of Unipa-GPT is presented, both the RAG and the fine-tuned systems are compared, and a brief discussion on their performance is reported. Further comparison with other Large Language Models and the experimental results during the SHARPER night are illustrated.

Title: Continual Learning for Adaptable Car-Following in Dynamic Traffic Environments

• Authors: Xianda Chen, PakHin Tiu, Xu Han, Junjie Chen, Yuanfei Wu, Xinhu Zheng, Meixin Zhu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14247
• Pdf URL: https://arxiv.org/pdf/2407.14247
• Copy Paste: [[2407.14247]] Continual Learning for Adaptable Car-Following in Dynamic Traffic Environments(https://arxiv.org/abs/2407.14247)
• Keywords: robust
• Abstract: The continual evolution of autonomous driving technology requires car-following models that can adapt to diverse and dynamic traffic environments. Traditional learning-based models often suffer from performance degradation when encountering unseen traffic patterns due to a lack of continual learning capabilities. This paper proposes a novel car-following model based on continual learning that addresses this limitation. Our framework incorporates Elastic Weight Consolidation (EWC) and Memory Aware Synapses (MAS) techniques to mitigate catastrophic forgetting and enable the model to learn incrementally from new traffic data streams. We evaluate the performance of the proposed model on the Waymo and Lyft datasets which encompass various traffic scenarios. The results demonstrate that the continual learning techniques significantly outperform the baseline model, achieving 0\% collision rates across all traffic conditions. This research contributes to the advancement of autonomous driving technology by fostering the development of more robust and adaptable car-following models.

Title: Personalized Multi-tier Federated Learning

• Authors: Sourasekhar Banerjee, Ali Dadras, Alp Yurtsever, Monowar Bhuyan
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2407.14251
• Pdf URL: https://arxiv.org/pdf/2407.14251
• Copy Paste: [[2407.14251]] Personalized Multi-tier Federated Learning(https://arxiv.org/abs/2407.14251)
• Keywords: robust, federate
• Abstract: The key challenge of personalized federated learning (PerFL) is to capture the statistical heterogeneity properties of data with inexpensive communications and gain customized performance for participating devices. To address these, we introduced personalized federated learning in multi-tier architecture (PerMFL) to obtain optimized and personalized local models when there are known team structures across devices. We provide theoretical guarantees of PerMFL, which offers linear convergence rates for smooth strongly convex problems and sub-linear convergence rates for smooth non-convex problems. We conduct numerical experiments demonstrating the robust empirical performance of PerMFL, outperforming the state-of-the-art in multiple personalized federated learning tasks.

Title: Bridging the Gap: A Survey and Classification of Research-Informed Ethical Hacking Tools

• Authors: Paolo Modesti, Lewis Golightly, Louis Holmes, Chidimma Opara, Marco Moscini
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.14255
• Pdf URL: https://arxiv.org/pdf/2407.14255
• Copy Paste: [[2407.14255]] Bridging the Gap: A Survey and Classification of Research-Informed Ethical Hacking Tools(https://arxiv.org/abs/2407.14255)
• Keywords: security
• Abstract: The majority of Ethical Hacking (EH) tools utilised in penetration testing are developed by practitioners within the industry or underground communities. Similarly, academic researchers have also contributed to developing security tools. However, there appears to be limited awareness among practitioners of academic contributions in this domain, creating a significant gap between industry and academia's contributions to EH tools. This research paper aims to survey the current state of EH academic research, primarily focusing on research-informed security tools. We categorise these tools into process-based frameworks (such as PTES and Mitre ATT\&CK) and knowledge-based frameworks (such as CyBOK and ACM CCS). This classification provides a comprehensive overview of novel, research-informed tools, considering their functionality and application areas. The analysis covers licensing, release dates, source code availability, development activity, and peer review status, providing valuable insights into the current state of research in this field.

Title: Obfuscated Location Disclosure for Remote ID Enabled Drones

• Authors: Alessandro Brighente, Mauro Conti, Matthijs Schotsman, Savio Sciancalepore
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.14256
• Pdf URL: https://arxiv.org/pdf/2407.14256
• Copy Paste: [[2407.14256]] Obfuscated Location Disclosure for Remote ID Enabled Drones(https://arxiv.org/abs/2407.14256)
• Keywords: privacy
• Abstract: The Remote ID (RID) regulation recently introduced by several aviation authorities worldwide (including the US and EU) forces commercial drones to regularly (max. every second) broadcast plaintext messages on the wireless channel, providing information about the drone identifier and current location, among others. Although these regulations increase the accountability of drone operations and improve traffic management, they allow malicious users to track drones via the disclosed information, possibly leading to drone capture and severe privacy leaks. In this paper, we propose Obfuscated Location disclOsure for RID-enabled drones (OLO-RID), a solution modifying and extending the RID regulation while preserving drones' location privacy. Rather than disclosing the actual drone's location, drones equipped with OLO-RID disclose a differentially private obfuscated location in a mobile scenario. OLO-RID also extends RID messages with encrypted location information, accessible only by authorized entities and valuable to obtain the current drone's location in safety-critical use cases. We design, implement, and deploy OLO-RID on a Raspberry Pi 3 and release the code of our implementation as open-source. We also perform an extensive performance assessment of the runtime overhead of our solution in terms of processing, communication, memory, and energy consumption. We show that OLO-RID can generate RID messages on a constrained device in less than 0.16 s while also requiring a minimal energy toll on a relevant device (0.0236% of energy for a DJI Mini 2). We also evaluate the utility of the proposed approach in the context of three reference use cases involving the drones' location usage, demonstrating minimal performance degradation when trading off location privacy and utility for next-generation RID-compliant drone ecosystems.

Title: SparseCraft: Few-Shot Neural Reconstruction through Stereopsis Guided Geometric Linearization

• Authors: Mae Younes, Amine Ouasfi, Adnane Boukhayma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14257
• Pdf URL: https://arxiv.org/pdf/2407.14257
• Copy Paste: [[2407.14257]] SparseCraft: Few-Shot Neural Reconstruction through Stereopsis Guided Geometric Linearization(https://arxiv.org/abs/2407.14257)
• Keywords: robust
• Abstract: We present a novel approach for recovering 3D shape and view dependent appearance from a few colored images, enabling efficient 3D reconstruction and novel view synthesis. Our method learns an implicit neural representation in the form of a Signed Distance Function (SDF) and a radiance field. The model is trained progressively through ray marching enabled volumetric rendering, and regularized with learning-free multi-view stereo (MVS) cues. Key to our contribution is a novel implicit neural shape function learning strategy that encourages our SDF field to be as linear as possible near the level-set, hence robustifying the training against noise emanating from the supervision and regularization signals. Without using any pretrained priors, our method, called SparseCraft, achieves state-of-the-art performances both in novel-view synthesis and reconstruction from sparse views in standard benchmarks, while requiring less than 10 minutes for training.

Title: Voices in a Crowd: Searching for Clusters of Unique Perspectives

• Authors: Nikolas Vitsakis, Amit Parekh, Ioannis Konstas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14259
• Pdf URL: https://arxiv.org/pdf/2407.14259
• Copy Paste: [[2407.14259]] Voices in a Crowd: Searching for Clusters of Unique Perspectives(https://arxiv.org/abs/2407.14259)
• Keywords: robust
• Abstract: Language models have been shown to reproduce underlying biases existing in their training data, which is the majority perspective by default. Proposed solutions aim to capture minority perspectives by either modelling annotator disagreements or grouping annotators based on shared metadata, both of which face significant challenges. We propose a framework that trains models without encoding annotator metadata, extracts latent embeddings informed by annotator behaviour, and creates clusters of similar opinions, that we refer to as voices. Resulting clusters are validated post-hoc via internal and external quantitative metrics, as well a qualitative analysis to identify the type of voice that each cluster represents. Our results demonstrate the strong generalisation capability of our framework, indicated by resulting clusters being adequately robust, while also capturing minority perspectives based on different demographic factors throughout two distinct datasets.

Title: Hyperparameter Optimization for Driving Strategies Based on Reinforcement Learning

• Authors: Nihal Acharya Adde, Hanno Gottschalk, Andreas Ebert
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2407.14262
• Pdf URL: https://arxiv.org/pdf/2407.14262
• Copy Paste: [[2407.14262]] Hyperparameter Optimization for Driving Strategies Based on Reinforcement Learning(https://arxiv.org/abs/2407.14262)
• Keywords: robust
• Abstract: This paper focuses on hyperparameter optimization for autonomous driving strategies based on Reinforcement Learning. We provide a detailed description of training the RL agent in a simulation environment. Subsequently, we employ Efficient Global Optimization algorithm that uses Gaussian Process fitting for hyperparameter optimization in RL. Before this optimization phase, Gaussian process interpolation is applied to fit the surrogate model, for which the hyperparameter set is generated using Latin hypercube sampling. To accelerate the evaluation, parallelization techniques are employed. Following the hyperparameter optimization procedure, a set of hyperparameters is identified, resulting in a noteworthy enhancement in overall driving performance. There is a substantial increase of 4\% when compared to existing manually tuned parameters and the hyperparameters discovered during the initialization process using Latin hypercube sampling. After the optimization, we analyze the obtained results thoroughly and conduct a sensitivity analysis to assess the robustness and generalization capabilities of the learned autonomous driving strategies. The findings from this study contribute to the advancement of Gaussian process based Bayesian optimization to optimize the hyperparameters for autonomous driving in RL, providing valuable insights for the development of efficient and reliable autonomous driving systems.

Title: Predictive Simultaneous Interpretation: Harnessing Large Language Models for Democratizing Real-Time Multilingual Communication

• Authors: Kurando Iida, Kenjiro Mimura, Nobuo Ito
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14269
• Pdf URL: https://arxiv.org/pdf/2407.14269
• Copy Paste: [[2407.14269]] Predictive Simultaneous Interpretation: Harnessing Large Language Models for Democratizing Real-Time Multilingual Communication(https://arxiv.org/abs/2407.14269)
• Keywords: large language model
• Abstract: This study introduces a groundbreaking approach to simultaneous interpretation by directly leveraging the predictive capabilities of Large Language Models (LLMs). We present a novel algorithm that generates real-time translations by predicting speaker utterances and expanding multiple possibilities in a tree-like structure. This method demonstrates unprecedented flexibility and adaptability, potentially overcoming the structural differences between languages more effectively than existing systems. Our theoretical analysis, supported by illustrative examples, suggests that this approach could lead to more natural and fluent translations with minimal latency. The primary purpose of this paper is to share this innovative concept with the academic community, stimulating further research and development in this field. We discuss the theoretical foundations, potential advantages, and implementation challenges of this technique, positioning it as a significant step towards democratizing multilingual communication.

Title: OpenSU3D: Open World 3D Scene Understanding using Foundation Models

• Authors: Rafay Mohiuddin, Sai Manoj Prakhya, Fiona Collins, Ziyuan Liu, André Borrmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14279
• Pdf URL: https://arxiv.org/pdf/2407.14279
• Copy Paste: [[2407.14279]] OpenSU3D: Open World 3D Scene Understanding using Foundation Models(https://arxiv.org/abs/2407.14279)
• Keywords: robust, large language model
• Abstract: In this paper, we present a novel, scalable approach for constructing open set, instance-level 3D scene representations, advancing open world understanding of 3D environments. Existing methods require pre-constructed 3D scenes and face scalability issues due to per-point feature vector learning, limiting their efficacy with complex queries. Our method overcomes these limitations by incrementally building instance-level 3D scene representations using 2D foundation models, efficiently aggregating instance-level details such as masks, feature vectors, names, and captions. We introduce fusion schemes for feature vectors to enhance their contextual knowledge and performance on complex queries. Additionally, we explore large language models for robust automatic annotation and spatial reasoning tasks. We evaluate our proposed approach on multiple scenes from ScanNet and Replica datasets demonstrating zero-shot generalization capabilities, exceeding current state-of-the-art methods in open world 3D scene understanding.

Title: How to Blend Concepts in Diffusion Models

• Authors: Giorgio Longari, Lorenzo Olearo, Simone Melzi, Rafael Peñaloza, Alessandro Raganato
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14280
• Pdf URL: https://arxiv.org/pdf/2407.14280
• Copy Paste: [[2407.14280]] How to Blend Concepts in Diffusion Models(https://arxiv.org/abs/2407.14280)
• Keywords: diffusion
• Abstract: For the last decade, there has been a push to use multi-dimensional (latent) spaces to represent concepts; and yet how to manipulate these concepts or reason with them remains largely unclear. Some recent methods exploit multiple latent representations and their connection, making this research question even more entangled. Our goal is to understand how operations in the latent space affect the underlying concepts. To that end, we explore the task of concept blending through diffusion models. Diffusion models are based on a connection between a latent representation of textual prompts and a latent space that enables image reconstruction and generation. This task allows us to try different text-based combination strategies, and evaluate easily through a visual analysis. Our conclusion is that concept blending through space manipulation is possible, although the best strategy depends on the context of the blend.

Title: PACCOR4ESP: Embedded Device Security Attestation using Platform Attribute Certificates

• Authors: Thomas Grübl, Jan von der Assen, Markus Knecht, Burkhard Stiller
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.14286
• Pdf URL: https://arxiv.org/pdf/2407.14286
• Copy Paste: [[2407.14286]] PACCOR4ESP: Embedded Device Security Attestation using Platform Attribute Certificates(https://arxiv.org/abs/2407.14286)
• Keywords: secure, security, attack
• Abstract: Verifying the integrity of embedded device characteristics is required to ensure secure operation of a device. One central challenge is to securely extract and store device-specific configurations for future verification. Existing device attestation schemes suffer from notable limitations, including a lack of standardization and a failure to encompass all hardware and software aspects inherent to a platform. This paper proposes an extension of the NSA Cybersecurity Directorate's Platform Attribute Certificate Creator (PACCOR) for the ESP32, a widely-used microcontroller series. Platform Attribute Certificates store device characteristics as per the Trusted Computing Group's Platform Certificate Profile. As of today, there is little research on hybrid attestation schemes utilizing Platform Attribute Certificates on embedded devices, which this work addresses. This paper presents a collection of attacks that can be detected using PACCOR4ESP. The toolkit extracts security-relevant information from an ESP32-S3, such as the firmware hash, bootloader hash, GPIO pin configuration, and a reference to the endorsement key of the secure element, and automatically embeds it into a Platform Attribute Certificate. Lastly, this work shows how PACCOR4ESP can be integrated with existing embedded device attestation frameworks, such as RAS, CRAFT, and SEDA.

Title: CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units

• Authors: Yeeun Kang
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.14295
• Pdf URL: https://arxiv.org/pdf/2407.14295
• Copy Paste: [[2407.14295]] CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units(https://arxiv.org/abs/2407.14295)
• Keywords: segmentation
• Abstract: Multilingual code-switching research is often hindered by the lack and linguistically biased status of available datasets. To expand language representation, we synthesize code-switching data by replacing intonation units detected through PSST, a speech segmentation model fine-tuned from OpenAI's Whisper, using a speech-to-text translation dataset, CoVoST 2. With our dataset, CoVoSwitch, spanning 13 languages, we evaluate the code-switching translation performance of two multilingual translation models, M2M-100 418M and NLLB-200 600M. We reveal that the inclusion of code-switching units results in higher translation performance than monolingual settings and that models are better at code-switching translation into English than non-English. Further, low-resource languages gain most from integration of code-switched units when translating into English but much less when translating into non-English. Translations into low-resource languages also perform worse than even raw code-switched inputs. We find that systems excel at copying English tokens but struggle with non-English tokens, that the off-target problem in monolingual settings is also relevant in code-switching settings, and that models hallucinate in code-switching translation by introducing words absent in both of the original source sentences. CoVoSwitch and code are available at this https URL.

Title: Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition

• Authors: Yurong Zhang, Honghao Chen, Xinyu Zhang, Xiangxiang Chu, Li Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14302
• Pdf URL: https://arxiv.org/pdf/2407.14302
• Copy Paste: [[2407.14302]] Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition(https://arxiv.org/abs/2407.14302)
• Keywords: extraction
• Abstract: Parameter-efficient transfer learning (PETL) is a promising task, aiming to adapt the large-scale pre-trained model to downstream tasks with a relatively modest cost. However, current PETL methods struggle in compressing computational complexity and bear a heavy inference burden due to the complete forward process. This paper presents an efficient visual recognition paradigm, called Dynamic Adapter (Dyn-Adapter), that boosts PETL efficiency by subtly disentangling features in multiple levels. Our approach is simple: first, we devise a dynamic architecture with balanced early heads for multi-level feature extraction, along with adaptive training strategy. Second, we introduce a bidirectional sparsity strategy driven by the pursuit of powerful generalization ability. These qualities enable us to fine-tune efficiently and effectively: we reduce FLOPs during inference by 50%, while maintaining or even yielding higher recognition accuracy. Extensive experiments on diverse datasets and pretrained backbones demonstrate the potential of Dyn-Adapter serving as a general efficiency booster for PETL in vision recognition tasks.

Title: EmoCAM: Toward Understanding What Drives CNN-based Emotion Recognition

• Authors: Youssef Doulfoukar, Laurent Mertens, Joost Vennekens
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14314
• Pdf URL: https://arxiv.org/pdf/2407.14314
• Copy Paste: [[2407.14314]] EmoCAM: Toward Understanding What Drives CNN-based Emotion Recognition(https://arxiv.org/abs/2407.14314)
• Keywords: explainability, segmentation
• Abstract: Convolutional Neural Networks are particularly suited for image analysis tasks, such as Image Classification, Object Recognition or Image Segmentation. Like all Artificial Neural Networks, however, they are "black box" models, and suffer from poor explainability. This work is concerned with the specific downstream task of Emotion Recognition from images, and proposes a framework that combines CAM-based techniques with Object Detection on a corpus level to better understand on which image cues a particular model, in our case EmoNet, relies to assign a specific emotion to an image. We demonstrate that the model mostly focuses on human characteristics, but also explore the pronounced effect of specific image modifications.

Title: Multimodal Misinformation Detection using Large Vision-Language Models

• Authors: Sahar Tahmasebi, Eric Müller-Budack, Ralph Ewerth
• Subjects: cs.CL, cs.IR, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.14321
• Pdf URL: https://arxiv.org/pdf/2407.14321
• Copy Paste: [[2407.14321]] Multimodal Misinformation Detection using Large Vision-Language Models(https://arxiv.org/abs/2407.14321)
• Keywords: fair, large language model
• Abstract: The increasing proliferation of misinformation and its alarming impact have motivated both industry and academia to develop approaches for misinformation detection and fact checking. Recent advances on large language models (LLMs) have shown remarkable performance in various tasks, but whether and how LLMs could help with misinformation detection remains relatively underexplored. Most of existing state-of-the-art approaches either do not consider evidence and solely focus on claim related features or assume the evidence to be provided. Few approaches consider evidence retrieval as part of the misinformation detection but rely on fine-tuning models. In this paper, we investigate the potential of LLMs for misinformation detection in a zero-shot setting. We incorporate an evidence retrieval component into the process as it is crucial to gather pertinent information from various sources to detect the veracity of claims. To this end, we propose a novel re-ranking approach for multimodal evidence retrieval using both LLMs and large vision-language models (LVLM). The retrieved evidence samples (images and texts) serve as the input for an LVLM-based approach for multimodal fact verification (LVLM4FV). To enable a fair evaluation, we address the issue of incomplete ground truth for evidence samples in an existing evidence retrieval dataset by annotating a more complete set of evidence samples for both image and text retrieval. Our experimental results on two datasets demonstrate the superiority of the proposed approach in both evidence retrieval and fact verification tasks and also better generalization capability across dataset compared to the supervised baseline.

Title: Panoptic Segmentation of Mammograms with Text-To-Image Diffusion Model

• Authors: Kun Zhao, Jakub Prokop, Javier Montalt Tordera, Sadegh Mohammadi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14326
• Pdf URL: https://arxiv.org/pdf/2407.14326
• Copy Paste: [[2407.14326]] Panoptic Segmentation of Mammograms with Text-To-Image Diffusion Model(https://arxiv.org/abs/2407.14326)
• Keywords: diffusion, segmentation
• Abstract: Mammography is crucial for breast cancer surveillance and early diagnosis. However, analyzing mammography images is a demanding task for radiologists, who often review hundreds of mammograms daily, leading to overdiagnosis and overtreatment. Computer-Aided Diagnosis (CAD) systems have been developed to assist in this process, but their capabilities, particularly in lesion segmentation, remained limited. With the contemporary advances in deep learning their performance may be improved. Recently, vision-language diffusion models emerged, demonstrating outstanding performance in image generation and transferability to various downstream tasks. We aim to harness their capabilities for breast lesion segmentation in a panoptic setting, which encompasses both semantic and instance-level predictions. Specifically, we propose leveraging pretrained features from a Stable Diffusion model as inputs to a state-of-the-art panoptic segmentation architecture, resulting in accurate delineation of individual breast lesions. To bridge the gap between natural and medical imaging domains, we incorporated a mammography-specific MAM-E diffusion model and BiomedCLIP image and text encoders into this framework. We evaluated our approach on two recently published mammography datasets, CDD-CESM and VinDr-Mammo. For the instance segmentation task, we noted 40.25 AP0.1 and 46.82 AP0.05, as well as 25.44 PQ0.1 and 26.92 PQ0.05. For the semantic segmentation task, we achieved Dice scores of 38.86 and 40.92, respectively.

Title: Modality-Order Matters! A Novel Hierarchical Feature Fusion Method for CoSAm: A Code-Switched Autism Corpus

• Authors: Mohd Mujtaba Akhtar, Girish, Muskaan Singh, Orchid Chetia Phukan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14328
• Pdf URL: https://arxiv.org/pdf/2407.14328
• Copy Paste: [[2407.14328]] Modality-Order Matters! A Novel Hierarchical Feature Fusion Method for CoSAm: A Code-Switched Autism Corpus(https://arxiv.org/abs/2407.14328)
• Keywords: robust, transformer
• Abstract: Autism Spectrum Disorder (ASD) is a complex neuro-developmental challenge, presenting a spectrum of difficulties in social interaction, communication, and the expression of repetitive behaviors in different situations. This increasing prevalence underscores the importance of ASD as a major public health concern and the need for comprehensive research initiatives to advance our understanding of the disorder and its early detection methods. This study introduces a novel hierarchical feature fusion method aimed at enhancing the early detection of ASD in children through the analysis of code-switched speech (English and Hindi). Employing advanced audio processing techniques, the research integrates acoustic, paralinguistic, and linguistic information using Transformer Encoders. This innovative fusion strategy is designed to improve classification robustness and accuracy, crucial for early and precise ASD identification. The methodology involves collecting a code-switched speech corpus, CoSAm, from children diagnosed with ASD and a matched control group. The dataset comprises 61 voice recordings from 30 children diagnosed with ASD and 31 from neurotypical children, aged between 3 and 13 years, resulting in a total of 159.75 minutes of voice recordings. The feature analysis focuses on MFCCs and extensive statistical attributes to capture speech pattern variability and complexity. The best model performance is achieved using a hierarchical fusion technique with an accuracy of 98.75% using a combination of acoustic and linguistic features first, followed by paralinguistic features in a hierarchical manner.

Title: LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domains

• Authors: Raphael Hernandes
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.14344
• Pdf URL: https://arxiv.org/pdf/2407.14344
• Copy Paste: [[2407.14344]] LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domains(https://arxiv.org/abs/2407.14344)
• Keywords: large language model
• Abstract: This research investigates whether OpenAI's GPT-4, a state-of-the-art large language model, can accurately classify the political bias of news sources based solely on their URLs. Given the subjective nature of political labels, third-party bias ratings like those from Ad Fontes Media, AllSides, and Media Bias/Fact Check (MBFC) are often used in research to analyze news source diversity. This study aims to determine if GPT-4 can replicate these human ratings on a seven-degree scale ("far-left" to "far-right"). The analysis compares GPT-4's classifications against MBFC's, and controls for website popularity using Open PageRank scores. Findings reveal a high correlation ($\text{Spearman's } \rho = .89$, $n = 5,877$, $p < 0.001$) between GPT-4's and MBFC's ratings, indicating the model's potential reliability. However, GPT-4 abstained from classifying approximately $\frac{2}{3}$ of the dataset, particularly less popular and less biased sources. The study also identifies a slight leftward skew in GPT-4's classifications compared to MBFC's. The analysis suggests that while GPT-4 can be a scalable, cost-effective tool for political bias classification of news websites, but its use should complement human judgment to mitigate biases. Further research is recommended to explore the model's performance across different settings, languages, and additional datasets.

Title: Thinking Racial Bias in Fair Forgery Detection: Models, Datasets and Evaluations

• Authors: Decheng Liu, Zongqi Wang, Chunlei Peng, Nannan Wang, Ruimin Hu, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14367
• Pdf URL: https://arxiv.org/pdf/2407.14367
• Copy Paste: [[2407.14367]] Thinking Racial Bias in Fair Forgery Detection: Models, Datasets and Evaluations(https://arxiv.org/abs/2407.14367)
• Keywords: security, fair
• Abstract: Due to the successful development of deep image generation technology, forgery detection plays a more important role in social and economic security. Racial bias has not been explored thoroughly in the deep forgery detection field. In the paper, we first contribute a dedicated dataset called the Fair Forgery Detection (FairFD) dataset, where we prove the racial bias of public state-of-the-art (SOTA) methods. Different from existing forgery detection datasets, the self-construct FairFD dataset contains a balanced racial ratio and diverse forgery generation images with the largest-scale subjects. Additionally, we identify the problems with naive fairness metrics when benchmarking forgery detection models. To comprehensively evaluate fairness, we design novel metrics including Approach Averaged Metric and Utility Regularized Metric, which can avoid deceptive results. Extensive experiments conducted with nine representative forgery detection models demonstrate the value of the proposed dataset and the reasonability of the designed fairness metrics. We also conduct more in-depth analyses to offer more insights to inspire researchers in the community.

Title: Open Artificial Knowledge

• Authors: Vadim Borisov, Richard H. Schreiber
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14371
• Pdf URL: https://arxiv.org/pdf/2407.14371
• Copy Paste: [[2407.14371]] Open Artificial Knowledge(https://arxiv.org/abs/2407.14371)
• Keywords: privacy, large language model
• Abstract: The tremendous success of chat-based AI systems like ChatGPT, Claude, and Gemini stems from Large Language Models (LLMs) trained on vast amount of datasets. However, acquiring high-quality, diverse, and ethically sourced training data remains a significant challenge. We introduce the Open Artificial Knowledge (OAK) dataset, a large-scale resource of over 500 million tokens (at the moment of writing) designed to address this issue. OAK leverages an ensemble of state-of-the-art LLMs, including GPT4o, LLaMa3-70B, LLaMa3-8B, Mixtral-8x7B, Gemma-7B, and Gemma-2-9B , to generate high-quality text across diverse domains, guided by Wikipedia's main categories. Our methodology ensures broad knowledge coverage while maintaining coherence and factual accuracy. The OAK dataset aims to foster the development of more capable and aligned language models while addressing critical issues of data scarcity and privacy in LLM training, and it is freely available on this http URL.

Title: Improving GBDT Performance on Imbalanced Datasets: An Empirical Study of Class-Balanced Loss Functions

• Authors: Jiaqi Luo, Yuan Yuan, Shixin Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14381
• Pdf URL: https://arxiv.org/pdf/2407.14381
• Copy Paste: [[2407.14381]] Improving GBDT Performance on Imbalanced Datasets: An Empirical Study of Class-Balanced Loss Functions(https://arxiv.org/abs/2407.14381)
• Keywords: robust
• Abstract: Class imbalance remains a significant challenge in machine learning, particularly for tabular data classification tasks. While Gradient Boosting Decision Trees (GBDT) models have proven highly effective for such tasks, their performance can be compromised when dealing with imbalanced datasets. This paper presents the first comprehensive study on adapting class-balanced loss functions to three GBDT algorithms across various tabular classification tasks, including binary, multi-class, and multi-label classification. We conduct extensive experiments on multiple datasets to evaluate the impact of class-balanced losses on different GBDT models, establishing a valuable benchmark. Our results demonstrate the potential of class-balanced loss functions to enhance GBDT performance on imbalanced datasets, offering a robust approach for practitioners facing class imbalance challenges in real-world applications. Additionally, we introduce a Python package that facilitates the integration of class-balanced loss functions into GBDT workflows, making these advanced techniques accessible to a wider audience.

Title: Frontiers of Deep Learning: From Novel Application to Real-World Deployment

• Authors: Rui Xie
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14386
• Pdf URL: https://arxiv.org/pdf/2407.14386
• Copy Paste: [[2407.14386]] Frontiers of Deep Learning: From Novel Application to Real-World Deployment(https://arxiv.org/abs/2407.14386)
• Keywords: transformer
• Abstract: Deep learning continues to re-shape numerous fields, from natural language processing and imaging to data analytics and recommendation systems. This report studies two research papers that represent recent progress on deep learning from two largely different aspects: The first paper applied the transformer networks, which are typically used in language models, to improve the quality of synthetic aperture radar image by effectively reducing the speckle noise. The second paper presents an in-storage computing design solution to enable cost-efficient and high-performance implementations of deep learning recommendation systems. In addition to summarizing each paper in terms of motivation, key ideas and techniques, and evaluation results, this report also presents thoughts and discussions about possible future research directions. By carrying out in-depth study on these two representative papers and related references, this doctoral candidate has developed better understanding on the far-reaching impact and efficient implementation of deep learning models.

Title: DEAL: Disentangle and Localize Concept-level Explanations for VLMs

• Authors: Tang Li, Mengmeng Ma, Xi Peng
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14412
• Pdf URL: https://arxiv.org/pdf/2407.14412
• Copy Paste: [[2407.14412]] DEAL: Disentangle and Localize Concept-level Explanations for VLMs(https://arxiv.org/abs/2407.14412)
• Keywords: explainability
• Abstract: Large pre-trained Vision-Language Models (VLMs) have become ubiquitous foundational components of other models and downstream tasks. Although powerful, our empirical results reveal that such models might not be able to identify fine-grained concepts. Specifically, the explanations of VLMs with respect to fine-grained concepts are entangled and mislocalized. To address this issue, we propose to DisEntAngle and Localize (DEAL) the concept-level explanations for VLMs without human annotations. The key idea is encouraging the concept-level explanations to be distinct while maintaining consistency with category-level explanations. We conduct extensive experiments and ablation studies on a wide range of benchmark datasets and vision-language models. Our empirical results demonstrate that the proposed method significantly improves the concept-level explanations of the model in terms of disentanglability and localizability. Surprisingly, the improved explainability alleviates the model's reliance on spurious correlations, which further benefits the prediction accuracy.

Title: Improving classification of road surface conditions via road area extraction and contrastive learning

• Authors: Linh Trinh, Ali Anwar, Siegfried Mercelis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14418
• Pdf URL: https://arxiv.org/pdf/2407.14418
• Copy Paste: [[2407.14418]] Improving classification of road surface conditions via road area extraction and contrastive learning(https://arxiv.org/abs/2407.14418)
• Keywords: extraction, segmentation
• Abstract: Maintaining roads is crucial to economic growth and citizen well-being because roads are a vital means of transportation. In various countries, the inspection of road surfaces is still done manually, however, to automate it, research interest is now focused on detecting the road surface defects via the visual data. While, previous research has been focused on deep learning methods which tend to process the entire image and leads to heavy computational cost. In this study, we focus our attention on improving the classification performance while keeping the computational cost of our solution low. Instead of processing the whole image, we introduce a segmentation model to only focus the downstream classification model to the road surface in the image. Furthermore, we employ contrastive learning during model training to improve the road surface condition classification. Our experiments on the public RTK dataset demonstrate a significant improvement in our proposed method when compared to previous works.

Title: HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation

• Authors: Zezeng Li, Weimin Wang, WenHai Li, Na Lei, Xianfeng Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14419
• Pdf URL: https://arxiv.org/pdf/2407.14419
• Copy Paste: [[2407.14419]] HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation(https://arxiv.org/abs/2407.14419)
• Keywords: diffusion
• Abstract: Recent CLIP-guided 3D generation methods have achieved promising results but struggle with generating faithful 3D shapes that conform with input text due to the gap between text and image embeddings. To this end, this paper proposes HOTS3D which makes the first attempt to effectively bridge this gap by aligning text features to the image features with spherical optimal transport (SOT). However, in high-dimensional situations, solving the SOT remains a challenge. To obtain the SOT map for high-dimensional features obtained from CLIP encoding of two modalities, we mathematically formulate and derive the solution based on Villani's theorem, which can directly align two hyper-sphere distributions without manifold exponential maps. Furthermore, we implement it by leveraging input convex neural networks (ICNNs) for the optimal Kantorovich potential. With the optimally mapped features, a diffusion-based generator and a Nerf-based decoder are subsequently utilized to transform them into 3D shapes. Extensive qualitative and qualitative comparisons with state-of-the-arts demonstrate the superiority of the proposed HOTS3D for 3D shape generation, especially on the consistency with text semantics.

Title: Controllable and Efficient Multi-Class Pathology Nuclei Data Augmentation using Text-Conditioned Diffusion Models

• Authors: Hyun-Jic Oh, Won-Ki Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14426
• Pdf URL: https://arxiv.org/pdf/2407.14426
• Copy Paste: [[2407.14426]] Controllable and Efficient Multi-Class Pathology Nuclei Data Augmentation using Text-Conditioned Diffusion Models(https://arxiv.org/abs/2407.14426)
• Keywords: diffusion, generative, segmentation
• Abstract: In the field of computational pathology, deep learning algorithms have made significant progress in tasks such as nuclei segmentation and classification. However, the potential of these advanced methods is limited by the lack of available labeled data. Although image synthesis via recent generative models has been actively explored to address this challenge, existing works have barely addressed label augmentation and are mostly limited to single-class and unconditional label generation. In this paper, we introduce a novel two-stage framework for multi-class nuclei data augmentation using text-conditional diffusion models. In the first stage, we innovate nuclei label synthesis by generating multi-class semantic labels and corresponding instance maps through a joint diffusion model conditioned by text prompts that specify the label structure information. In the second stage, we utilize a semantic and text-conditional latent diffusion model to efficiently generate high-quality pathology images that align with the generated nuclei label images. We demonstrate the effectiveness of our method on large and diverse pathology nuclei datasets, with evaluations including qualitative and quantitative analyses, as well as assessments of downstream tasks.

Title: The Extrapolation Power of Implicit Models

• Authors: Juliette Decugis, Alicia Y. Tsai, Max Emerling, Ashwin Ganesh, Laurent El Ghaoui
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14430
• Pdf URL: https://arxiv.org/pdf/2407.14430
• Copy Paste: [[2407.14430]] The Extrapolation Power of Implicit Models(https://arxiv.org/abs/2407.14430)
• Keywords: robust
• Abstract: In this paper, we investigate the extrapolation capabilities of implicit deep learning models in handling unobserved data, where traditional deep neural networks may falter. Implicit models, distinguished by their adaptability in layer depth and incorporation of feedback within their computational graph, are put to the test across various extrapolation scenarios: out-of-distribution, geographical, and temporal shifts. Our experiments consistently demonstrate significant performance advantage with implicit models. Unlike their non-implicit counterparts, which often rely on meticulous architectural design for each task, implicit models demonstrate the ability to learn complex model structures without the need for task-specific design, highlighting their robustness in handling unseen data.

Title: Co-synthesis of Histopathology Nuclei Image-Label Pairs using a Context-Conditioned Joint Diffusion Model

• Authors: Seonghui Min, Hyun-Jic Oh, Won-Ki Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14434
• Pdf URL: https://arxiv.org/pdf/2407.14434
• Copy Paste: [[2407.14434]] Co-synthesis of Histopathology Nuclei Image-Label Pairs using a Context-Conditioned Joint Diffusion Model(https://arxiv.org/abs/2407.14434)
• Keywords: diffusion, generative, segmentation
• Abstract: In multi-class histopathology nuclei analysis tasks, the lack of training data becomes a main bottleneck for the performance of learning-based methods. To tackle this challenge, previous methods have utilized generative models to increase data by generating synthetic samples. However, existing methods often overlook the importance of considering the context of biological tissues (e.g., shape, spatial layout, and tissue type) in the synthetic data. Moreover, while generative models have shown superior performance in synthesizing realistic histopathology images, none of the existing methods are capable of producing image-label pairs at the same time. In this paper, we introduce a novel framework for co-synthesizing histopathology nuclei images and paired semantic labels using a context-conditioned joint diffusion model. We propose conditioning of a diffusion model using nucleus centroid layouts with structure-related text prompts to incorporate spatial and structural context information into the generation targets. Moreover, we enhance the granularity of our synthesized semantic labels by generating instance-wise nuclei labels using distance maps synthesized concurrently in conjunction with the images and semantic labels. We demonstrate the effectiveness of our framework in generating high-quality samples on multi-institutional, multi-organ, and multi-modality datasets. Our synthetic data consistently outperforms existing augmentation methods in the downstream tasks of nuclei segmentation and classification.

Title: Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders

• Authors: Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, Neel Nanda
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14435
• Pdf URL: https://arxiv.org/pdf/2407.14435
• Copy Paste: [[2407.14435]] Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders(https://arxiv.org/abs/2407.14435)
• Keywords: interpretability
• Abstract: Sparse autoencoders (SAEs) are a promising unsupervised approach for identifying causally relevant and interpretable linear features in a language model's (LM) activations. To be useful for downstream tasks, SAEs need to decompose LM activations faithfully; yet to be interpretable the decomposition must be sparse -- two objectives that are in tension. In this paper, we introduce JumpReLU SAEs, which achieve state-of-the-art reconstruction fidelity at a given sparsity level on Gemma 2 9B activations, compared to other recent advances such as Gated and TopK SAEs. We also show that this improvement does not come at the cost of interpretability through manual and automated interpretability studies. JumpReLU SAEs are a simple modification of vanilla (ReLU) SAEs -- where we replace the ReLU with a discontinuous JumpReLU activation function -- and are similarly efficient to train and run. By utilising straight-through-estimators (STEs) in a principled manner, we show how it is possible to train JumpReLU SAEs effectively despite the discontinuous JumpReLU function introduced in the SAE's forward pass. Similarly, we use STEs to directly train L0 to be sparse, instead of training on proxies such as L1, avoiding problems like shrinkage.

Title: Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding

• Authors: Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, Liqiang Nie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14439
• Pdf URL: https://arxiv.org/pdf/2407.14439
• Copy Paste: [[2407.14439]] Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding(https://arxiv.org/abs/2407.14439)
• Keywords: large language model
• Abstract: Cropping high-resolution document images into multiple sub-images is the most widely used approach for current Multimodal Large Language Models (MLLMs) to do document understanding. Most of current document understanding methods preserve all tokens within sub-images and treat them equally. This neglects their different informativeness and leads to a significant increase in the number of image tokens. To perform a more adaptive and efficient document understanding, we propose Token-level Correlation-guided Compression, a parameter-free and plug-and-play methodology to optimize token processing. Firstly, we propose an innovative approach for assessing the pattern repetitiveness based on the correlation between each patch tokens. This method identifies redundant tokens, allowing for the determination of the sub-image's information density. Secondly, we present a token-level sampling method that efficiently captures the most informative tokens by delving into the correlation between the [CLS] token and patch tokens. By integrating these strategies, we develop a plug-and-play adaptive compressor module that can be seamlessly incorporated into MLLMs utilizing cropping techniques. This module not only enhances the processing speed during training and inference but also maintains comparable performance. We conduct experiments with the SOTA document understanding model mPLUG-DocOwl1.5 and the effectiveness is demonstrated through extensive comparisons with other compression methods.

Title: PolyFormer: Scalable Node-wise Filters via Polynomial Graph Transformer

• Authors: Jiahong Ma, Mingguo He, Zhewei Wei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14459
• Pdf URL: https://arxiv.org/pdf/2407.14459
• Copy Paste: [[2407.14459]] PolyFormer: Scalable Node-wise Filters via Polynomial Graph Transformer(https://arxiv.org/abs/2407.14459)
• Keywords: transformer
• Abstract: Spectral Graph Neural Networks have demonstrated superior performance in graph representation learning. However, many current methods focus on employing shared polynomial coefficients for all nodes, i.e., learning node-unified filters, which limits the filters' flexibility for node-level tasks. The recent DSF attempts to overcome this limitation by learning node-wise coefficients based on positional encoding. However, the initialization and updating process of the positional encoding are burdensome, hindering scalability on large-scale graphs. In this work, we propose a scalable node-wise filter, PolyAttn. Leveraging the attention mechanism, PolyAttn can directly learn node-wise filters in an efficient manner, offering powerful representation capabilities. Building on PolyAttn, we introduce the whole model, named PolyFormer. In the lens of Graph Transformer models, PolyFormer, which calculates attention scores within nodes, shows great scalability. Moreover, the model captures spectral information, enhancing expressiveness while maintaining efficiency. With these advantages, PolyFormer offers a desirable balance between scalability and expressiveness for node-level tasks. Extensive experiments demonstrate that our proposed methods excel at learning arbitrary node-wise filters, showing superior performance on both homophilic and heterophilic graphs, and handling graphs containing up to 100 million nodes. The code is available at this https URL.

Title: SurvReLU: Inherently Interpretable Survival Analysis via Deep ReLU Networks

• Authors: Xiaotong Sun, Peijie Qiu, Shengfan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14463
• Pdf URL: https://arxiv.org/pdf/2407.14463
• Copy Paste: [[2407.14463]] SurvReLU: Inherently Interpretable Survival Analysis via Deep ReLU Networks(https://arxiv.org/abs/2407.14463)
• Keywords: interpretability
• Abstract: Survival analysis models time-to-event distributions with censorship. Recently, deep survival models using neural networks have dominated due to their representational power and state-of-the-art performance. However, their "black-box" nature hinders interpretability, which is crucial in real-world applications. In contrast, "white-box" tree-based survival models offer better interpretability but struggle to converge to global optima due to greedy expansion. In this paper, we bridge the gap between previous deep survival models and traditional tree-based survival models through deep rectified linear unit (ReLU) networks. We show that a deliberately constructed deep ReLU network (SurvReLU) can harness the interpretability of tree-based structures with the representational power of deep survival models. Empirical studies on both simulated and real survival benchmark datasets show the effectiveness of the proposed SurvReLU in terms of performance and interoperability. The code is available at \href{this https URL}{\color{magenta}{ this https URL}}.

Title: Check-Eval: A Checklist-based Approach for Evaluating Text Quality

• Authors: Jayr Pereira, Roberto Lotufo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14467
• Pdf URL: https://arxiv.org/pdf/2407.14467
• Copy Paste: [[2407.14467]] Check-Eval: A Checklist-based Approach for Evaluating Text Quality(https://arxiv.org/abs/2407.14467)
• Keywords: large language model
• Abstract: Evaluating the quality of text generated by large language models (LLMs) remains a significant challenge. Traditional metrics often fail to align well with human judgments, particularly in tasks requiring creativity and nuance. In this paper, we propose Check-Eval, a novel evaluation framework leveraging LLMs to assess the quality of generated text through a checklist-based approach. Check-Eval can be employed as both a reference-free and reference-dependent evaluation method, providing a structured and interpretable assessment of text quality. The framework consists of two main stages: checklist generation and checklist evaluation. We validate Check-Eval on two benchmark datasets: Portuguese Legal Semantic Textual Similarity and SummEval. Our results demonstrate that Check-Eval achieves higher correlations with human judgments compared to existing metrics, such as G-Eval and GPTScore, underscoring its potential as a more reliable and effective evaluation framework for natural language generation tasks. The code for our experiments is available at https://anonymous.4open.science/r/check-eval-0DB4.

Title: MLMT-CNN for Object Detection and Segmentation in Multi-layer and Multi-spectral Images

• Authors: Majedaldein Almahasneh, Adeline Paiement, Xianghua Xie, Jean Aboudarham
• Subjects: cs.CV, physics.space-ph
• Abstract URL: https://arxiv.org/abs/2407.14473
• Pdf URL: https://arxiv.org/pdf/2407.14473
• Copy Paste: [[2407.14473]] MLMT-CNN for Object Detection and Segmentation in Multi-layer and Multi-spectral Images(https://arxiv.org/abs/2407.14473)
• Keywords: segmentation
• Abstract: Precisely localising solar Active Regions (AR) from multi-spectral images is a challenging but important task in understanding solar activity and its influence on space weather. A main challenge comes from each modality capturing a different location of the 3D objects, as opposed to typical multi-spectral imaging scenarios where all image bands observe the same scene. Thus, we refer to this special multi-spectral scenario as multi-layer. We present a multi-task deep learning framework that exploits the dependencies between image bands to produce 3D AR localisation (segmentation and detection) where different image bands (and physical locations) have their own set of results. Furthermore, to address the difficulty of producing dense AR annotations for training supervised machine learning (ML) algorithms, we adapt a training strategy based on weak labels (i.e. bounding boxes) in a recursive manner. We compare our detection and segmentation stages against baseline approaches for solar image analysis (multi-channel coronal hole detection, SPOCA for ARs) and state-of-the-art deep learning methods (Faster RCNN, U-Net). Additionally, both detection a nd segmentation stages are quantitatively validated on artificially created data of similar spatial configurations made from annotated multi-modal magnetic resonance images. Our framework achieves an average of 0.72 IoU (segmentation) and 0.90 F1 score (detection) across all modalities, comparing to the best performing baseline methods with scores of 0.53 and 0.58, respectively, on the artificial dataset, and 0.84 F1 score in the AR detection task comparing to baseline of 0.82 F1 score. Our segmentation results are qualitatively validated by an expert on real ARs.

Title: Contrastive Learning with Counterfactual Explanations for Radiology Report Generation

• Authors: Mingjie Li, Haokun Lin, Liang Qiu, Xiaodan Liang, Ling Chen, Abdulmotaleb Elsaddik, Xiaojun Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14474
• Pdf URL: https://arxiv.org/pdf/2407.14474
• Copy Paste: [[2407.14474]] Contrastive Learning with Counterfactual Explanations for Radiology Report Generation(https://arxiv.org/abs/2407.14474)
• Keywords: large language model
• Abstract: Due to the common content of anatomy, radiology images with their corresponding reports exhibit high similarity. Such inherent data bias can predispose automatic report generation models to learn entangled and spurious representations resulting in misdiagnostic reports. To tackle these, we propose a novel \textbf{Co}unter\textbf{F}actual \textbf{E}xplanations-based framework (CoFE) for radiology report generation. Counterfactual explanations serve as a potent tool for understanding how decisions made by algorithms can be changed by asking ``what if'' scenarios. By leveraging this concept, CoFE can learn non-spurious visual representations by contrasting the representations between factual and counterfactual images. Specifically, we derive counterfactual images by swapping a patch between positive and negative samples until a predicted diagnosis shift occurs. Here, positive and negative samples are the most semantically similar but have different diagnosis labels. Additionally, CoFE employs a learnable prompt to efficiently fine-tune the pre-trained large language model, encapsulating both factual and counterfactual content to provide a more generalizable prompt representation. Extensive experiments on two benchmarks demonstrate that leveraging the counterfactual explanations enables CoFE to generate semantically coherent and factually complete reports and outperform in terms of language generation and clinical efficacy metrics.

Title: A review on vision-based motion estimation

• Authors: Hongyi Liu, Haifeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14478
• Pdf URL: https://arxiv.org/pdf/2407.14478
• Copy Paste: [[2407.14478]] A review on vision-based motion estimation(https://arxiv.org/abs/2407.14478)
• Keywords: robust
• Abstract: Compared to contact sensors-based motion measurement, vision-based motion measurement has advantages of low cost and high efficiency and have been under active development in the past decades. This paper provides a review on existing motion measurement methods. In addition to the development of each branch of vision-based motion measurement methods, this paper also discussed the advantages and disadvantages of existing methods. Based on this discussion, it was identified that existing methods have a common limitation in optimally balancing accuracy and robustness. To address issue, we developed the Gaussian kernel-based motion measurement method. Preliminary study shows that the developed method can achieve high accuracy on simple synthesized images.

Title: Evaluating the Reliability of Self-Explanations in Large Language Models

• Authors: Korbinian Randl, John Pavlopoulos, Aron Henriksson, Tony Lindgren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14487
• Pdf URL: https://arxiv.org/pdf/2407.14487
• Copy Paste: [[2407.14487]] Evaluating the Reliability of Self-Explanations in Large Language Models(https://arxiv.org/abs/2407.14487)
• Keywords: explainability, large language model
• Abstract: This paper investigates the reliability of explanations generated by large language models (LLMs) when prompted to explain their previous output. We evaluate two kinds of such self-explanations - extractive and counterfactual - using three state-of-the-art LLMs (2B to 8B parameters) on two different classification tasks (objective and subjective). Our findings reveal, that, while these self-explanations can correlate with human judgement, they do not fully and accurately follow the model's decision process, indicating a gap between perceived and actual model reasoning. We show that this gap can be bridged because prompting LLMs for counterfactual explanations can produce faithful, informative, and easy-to-verify results. These counterfactuals offer a promising alternative to traditional explainability methods (e.g. SHAP, LIME), provided that prompts are tailored to specific tasks and checked for validity.

Title: InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques

• Authors: Rohan Gupta, Iván Arcuschin, Thomas Kwa, Adrià Garriga-Alonso
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14494
• Pdf URL: https://arxiv.org/pdf/2407.14494
• Copy Paste: [[2407.14494]] InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques(https://arxiv.org/abs/2407.14494)
• Keywords: interpretability, transformer
• Abstract: Mechanistic interpretability methods aim to identify the algorithm a neural network implements, but it is difficult to validate such methods when the true algorithm is unknown. This work presents InterpBench, a collection of semi-synthetic yet realistic transformers with known circuits for evaluating these techniques. We train these neural networks using a stricter version of Interchange Intervention Training (IIT) which we call Strict IIT (SIIT). Like the original, SIIT trains neural networks by aligning their internal computation with a desired high-level causal model, but it also prevents non-circuit nodes from affecting the model's output. We evaluate SIIT on sparse transformers produced by the Tracr tool and find that SIIT models maintain Tracr's original circuit while being more realistic. SIIT can also train transformers with larger circuits, like Indirect Object Identification (IOI). Finally, we use our benchmark to evaluate existing circuit discovery techniques.

Title: Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery

• Authors: Sukrut Rao, Sweta Mahajan, Moritz Böhle, Bernt Schiele
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14499
• Pdf URL: https://arxiv.org/pdf/2407.14499
• Copy Paste: [[2407.14499]] Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery(https://arxiv.org/abs/2407.14499)
• Keywords: extraction, interpretability
• Abstract: Concept Bottleneck Models (CBMs) have recently been proposed to address the 'black-box' problem of deep neural networks, by first mapping images to a human-understandable concept space and then linearly combining concepts for classification. Such models typically require first coming up with a set of concepts relevant to the task and then aligning the representations of a feature extractor to map to these concepts. However, even with powerful foundational feature extractors like CLIP, there are no guarantees that the specified concepts are detectable. In this work, we leverage recent advances in mechanistic interpretability and propose a novel CBM approach -- called Discover-then-Name-CBM (DN-CBM) -- that inverts the typical paradigm: instead of pre-selecting concepts based on the downstream classification task, we use sparse autoencoders to first discover concepts learnt by the model, and then name them and train linear probes for classification. Our concept extraction strategy is efficient, since it is agnostic to the downstream task, and uses concepts already known to the model. We perform a comprehensive evaluation across multiple datasets and CLIP architectures and show that our method yields semantically meaningful concepts, assigns appropriate names to them that make them easy to interpret, and yields performant and interpretable CBMs. Code available at this https URL.

Title: ViLLa: Video Reasoning Segmentation with Large Language Model

• Authors: Rongkun Zheng, Lu Qi, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14500
• Pdf URL: https://arxiv.org/pdf/2407.14500
• Copy Paste: [[2407.14500]] ViLLa: Video Reasoning Segmentation with Large Language Model(https://arxiv.org/abs/2407.14500)
• Keywords: large language model, segmentation
• Abstract: Although video perception models have made remarkable advancements in recent years, they still heavily rely on explicit text descriptions or pre-defined categories to identify target instances before executing video perception tasks. These models, however, fail to proactively comprehend and reason the user's intentions via textual input. Even though previous works attempt to investigate solutions to incorporate reasoning with image segmentation, they fail to reason with videos due to the video's complexity in object motion. To bridge the gap between image and video, in this work, we propose a new video segmentation task - video reasoning segmentation. The task is designed to output tracklets of segmentation masks given a complex input text query. What's more, to promote research in this unexplored area, we construct a reasoning video segmentation benchmark. Finally, we present ViLLa: Video reasoning segmentation with a Large Language Model, which incorporates the language generation capabilities of multimodal Large Language Models (LLMs) while retaining the capabilities of detecting, segmenting, and tracking multiple instances. We use a temporal-aware context aggregation module to incorporate contextual visual cues to text embeddings and propose a video-frame decoder to build temporal correlations across segmentation tokens. Remarkably, our ViLLa demonstrates capability in handling complex reasoning and referring video segmentation. Also, our model shows impressive ability in different temporal understanding benchmarks. Both quantitative and qualitative experiments show our method effectively unlocks new video reasoning segmentation capabilities for multimodal LLMs. The code and dataset will be available at this https URL.

Title: M2D2M: Multi-Motion Generation from Text with Discrete Diffusion Models

• Authors: Seunggeun Chi, Hyung-gun Chi, Hengbo Ma, Nakul Agarwal, Faizan Siddiqui, Karthik Ramani, Kwonjoon Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14502
• Pdf URL: https://arxiv.org/pdf/2407.14502
• Copy Paste: [[2407.14502]] M2D2M: Multi-Motion Generation from Text with Discrete Diffusion Models(https://arxiv.org/abs/2407.14502)
• Keywords: diffusion
• Abstract: We introduce the Multi-Motion Discrete Diffusion Models (M2D2M), a novel approach for human motion generation from textual descriptions of multiple actions, utilizing the strengths of discrete diffusion models. This approach adeptly addresses the challenge of generating multi-motion sequences, ensuring seamless transitions of motions and coherence across a series of actions. The strength of M2D2M lies in its dynamic transition probability within the discrete diffusion model, which adapts transition probabilities based on the proximity between motion tokens, encouraging mixing between different modes. Complemented by a two-phase sampling strategy that includes independent and joint denoising steps, M2D2M effectively generates long-term, smooth, and contextually coherent human motion sequences, utilizing a model trained for single-motion generation. Extensive experiments demonstrate that M2D2M surpasses current state-of-the-art benchmarks for motion generation from text descriptions, showcasing its efficacy in interpreting language semantics and generating dynamic, realistic motions.

Title: Nonlinear Schr\"odinger Network

• Authors: Yiming Zhou, Callen MacPhee, Tingyi Zhou, Bahram Jalali
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.14504
• Pdf URL: https://arxiv.org/pdf/2407.14504
• Copy Paste: [[2407.14504]] Nonlinear Schr\"odinger Network(https://arxiv.org/abs/2407.14504)
• Keywords: interpretability
• Abstract: Deep neural networks (DNNs) have achieved exceptional performance across various fields by learning complex nonlinear mappings from large-scale datasets. However, they encounter challenges such as high computational costs and limited interpretability. To address these issues, hybrid approaches that integrate physics with AI are gaining interest. This paper introduces a novel physics-based AI model called the "Nonlinear Schrödinger Network", which treats the Nonlinear Schrödinger Equation (NLSE) as a general-purpose trainable model for learning complex patterns including nonlinear mappings and memory effects from data. Existing physics-informed machine learning methods use neural networks to approximate the solutions of partial differential equations (PDEs). In contrast, our approach directly treats the PDE as a trainable model to obtain general nonlinear mappings that would otherwise require neural networks. As a physics-inspired approach, it offers a more interpretable and parameter-efficient alternative to traditional black-box neural networks, achieving comparable or better accuracy in time series classification tasks while significantly reducing the number of required parameters. Notably, the trained Nonlinear Schrödinger Network is interpretable, with all parameters having physical meanings as properties of a virtual physical system that transforms the data to a more separable space. This interpretability allows for insight into the underlying dynamics of the data transformation process. Applications to time series forecasting have also been explored. While our current implementation utilizes the NLSE, the proposed method of using physics equations as trainable models to learn nonlinear mappings from data is not limited to the NLSE and may be extended to other master equations of physics.

Title: T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation

• Authors: Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.14505
• Pdf URL: https://arxiv.org/pdf/2407.14505
• Copy Paste: [[2407.14505]] T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation(https://arxiv.org/abs/2407.14505)
• Keywords: generative
• Abstract: Text-to-video (T2V) generation models have advanced significantly, yet their ability to compose different objects, attributes, actions, and motions into a video remains unexplored. Previous text-to-video benchmarks also neglect this important ability for evaluation. In this work, we conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation. T2V-CompBench encompasses diverse aspects of compositionality, including consistent attribute binding, dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy. We further carefully design evaluation metrics of MLLM-based metrics, detection-based metrics, and tracking-based metrics, which can better reflect the compositional text-to-video generation quality of seven proposed categories with 700 text prompts. The effectiveness of the proposed metrics is verified by correlation with human evaluations. We also benchmark various text-to-video generative models and conduct in-depth analysis across different models and different compositional categories. We find that compositional text-to-video generation is highly challenging for current models, and we hope that our attempt will shed light on future research in this direction.

Title: On Pre-training of Multimodal Language Models Customized for Chart Understanding

• Authors: Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, Leonid Sigal
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14506
• Pdf URL: https://arxiv.org/pdf/2407.14506
• Copy Paste: [[2407.14506]] On Pre-training of Multimodal Language Models Customized for Chart Understanding(https://arxiv.org/abs/2407.14506)
• Keywords: robust, large language model
• Abstract: Recent studies customizing Multimodal Large Language Models (MLLMs) for domain-specific tasks have yielded promising results, especially in the field of scientific chart comprehension. These studies generally utilize visual instruction tuning with specialized datasets to enhance question and answer (QA) accuracy within the chart domain. However, they often neglect the fundamental discrepancy between natural image-caption pre-training data and digital chart image-QA data, particularly in the models' capacity to extract underlying numeric values from charts. This paper tackles this oversight by exploring the training processes necessary to improve MLLMs' comprehension of charts. We present three key findings: (1) Incorporating raw data values in alignment pre-training markedly improves comprehension of chart data. (2) Replacing images with their textual representation randomly during end-to-end fine-tuning transfer the language reasoning capability to chart interpretation skills. (3) Requiring the model to first extract the underlying chart data and then answer the question in the fine-tuning can further improve the accuracy. Consequently, we introduce CHOPINLLM, an MLLM tailored for in-depth chart comprehension. CHOPINLLM effectively interprets various types of charts, including unannotated ones, while maintaining robust reasoning abilities. Furthermore, we establish a new benchmark to evaluate MLLMs' understanding of different chart types across various comprehension levels. Experimental results show that CHOPINLLM exhibits strong performance in understanding both annotated and unannotated charts across a wide range of types.

Title: Internal Consistency and Self-Feedback in Large Language Models: A Survey

• Authors: Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Feiyu Xiong, Zhiyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.14507
• Pdf URL: https://arxiv.org/pdf/2407.14507
• Copy Paste: [[2407.14507]] Internal Consistency and Self-Feedback in Large Language Models: A Survey(https://arxiv.org/abs/2407.14507)
• Keywords: large language model
• Abstract: Large language models (LLMs) are expected to respond accurately but often exhibit deficient reasoning or generate hallucinatory content. To address these, studies prefixed with ``Self-'' such as Self-Consistency, Self-Improve, and Self-Refine have been initiated. They share a commonality: involving LLMs evaluating and updating itself to mitigate the issues. Nonetheless, these efforts lack a unified perspective on summarization, as existing surveys predominantly focus on categorization without examining the motivations behind these works. In this paper, we summarize a theoretical framework, termed Internal Consistency, which offers unified explanations for phenomena such as the lack of reasoning and the presence of hallucinations. Internal Consistency assesses the coherence among LLMs' latent layer, decoding layer, and response layer based on sampling methodologies. Expanding upon the Internal Consistency framework, we introduce a streamlined yet effective theoretical framework capable of mining Internal Consistency, named Self-Feedback. The Self-Feedback framework consists of two modules: Self-Evaluation and Self-Update. This framework has been employed in numerous studies. We systematically classify these studies by tasks and lines of work; summarize relevant evaluation methods and benchmarks; and delve into the concern, ``Does Self-Feedback Really Work?'' We propose several critical viewpoints, including the ``Hourglass Evolution of Internal Consistency'', ``Consistency Is (Almost) Correctness'' hypothesis, and ``The Paradox of Latent and Explicit Reasoning''. Furthermore, we outline promising directions for future research. We have open-sourced the experimental code, reference list, and statistical data, available at \url{this https URL}.

Title: DEPICT: Diffusion-Enabled Permutation Importance for Image Classification Tasks

• Authors: Sarah Jabbour, Gregory Kondas, Ella Kazerooni, Michael Sjoding, David Fouhey, Jenna Wiens
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.14509
• Pdf URL: https://arxiv.org/pdf/2407.14509
• Copy Paste: [[2407.14509]] DEPICT: Diffusion-Enabled Permutation Importance for Image Classification Tasks(https://arxiv.org/abs/2407.14509)
• Keywords: diffusion
• Abstract: We propose a permutation-based explanation method for image classifiers. Current image-model explanations like activation maps are limited to instance-based explanations in the pixel space, making it difficult to understand global model behavior. In contrast, permutation based explanations for tabular data classifiers measure feature importance by comparing model performance on data before and after permuting a feature. We propose an explanation method for image-based models that permutes interpretable concepts across dataset images. Given a dataset of images labeled with specific concepts like captions, we permute a concept across examples in the text space and then generate images via a text-conditioned diffusion model. Feature importance is then reflected by the change in model performance relative to unpermuted data. When applied to a set of concepts, the method generates a ranking of feature importance. We show this approach recovers underlying model feature importance on synthetic and real-world image classification tasks.