2024-09-09

Title: Post-quantum encryption algorithms of high-degree 3-variable polynomial congruences: BS cryptosystems and BS key generation

• Authors: Nicholas J.Daras
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.03758
• Pdf URL: https://arxiv.org/pdf/2409.03758
• Copy Paste: [[2409.03758]] Post-quantum encryption algorithms of high-degree 3-variable polynomial congruences: BS cryptosystems and BS key generation(https://arxiv.org/abs/2409.03758)
• Keywords: secure, security
• Abstract: We will construct post-quantum encryption algorithms based on three-variable polynomial Beal-Schur congruence. After giving a proof of Beal's conjecture and citing some applications of it to selected cases where the discrete logarithm and some of its generalizations are unsolvable problems, we will investigate the formulation and validity of an appropriate version of the Beal's conjecture on finite fields of integers. In contrast to the infinite case, we will show that the corresponding Beal-Schur congruence equation $x^{p}+y^{q}\equiv z^{r} (mod \mathcal{N})$ has non-trivial solutions into the finite field $\mathbb{Z}_{\mathcal{N}} $, for all sufficiently large primes $\mathcal{N}$ that do not divide the product $xyz$, under certain mutual divisibility conditions of the exponents $p$, $q$ and $r$. We will apply this result to generate the so-called BS cryptosystems, i.e., simple and secure post-quantum encryption algorithms based on the Beal-Schur congruence equation, as well as new cryptographic key generation methods, whose post-quantum algorithmic encryption security relies on having an infinite number of options for the parameters $p$, $q$, $r$, $\mathcal{N}$.

Title: A Dataset for Mechanical Mechanisms

• Authors: Farshid Ghezelbash, Amir Hossein Eskandari, Amir J Bidhendi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03763
• Pdf URL: https://arxiv.org/pdf/2409.03763
• Copy Paste: [[2409.03763]] A Dataset for Mechanical Mechanisms(https://arxiv.org/abs/2409.03763)
• Keywords: diffusion, generative
• Abstract: This study introduces a dataset consisting of approximately 9,000 images of mechanical mechanisms and their corresponding descriptions, aimed at supporting research in mechanism design. The dataset consists of a diverse collection of 2D and 3D sketches, meticulously curated to ensure relevance and quality. We demonstrate the application of this dataset by fine-tuning two models: 1) Stable Diffusion (for generating new mechanical designs), and 2) BLIP-2 (for captioning these designs). While the results from Stable Diffusion show promise, particularly in generating coherent 3D sketches, the model struggles with 2D sketches and occasionally produces nonsensical outputs. These limitations underscore the need for further development, particularly in expanding the dataset and refining model architectures. Nonetheless, this work serves as a step towards leveraging generative AI in mechanical design, highlighting both the potential and current limitations of these approaches.

Title: AI and Entrepreneurship: Facial Recognition Technology Detects Entrepreneurs, Outperforming Human Experts

• Authors: Martin Obschonka, Christian Fisch, Tharindu Fernando, Clinton Fookes
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.03765
• Pdf URL: https://arxiv.org/pdf/2409.03765
• Copy Paste: [[2409.03765]] AI and Entrepreneurship: Facial Recognition Technology Detects Entrepreneurs, Outperforming Human Experts(https://arxiv.org/abs/2409.03765)
• Keywords: robust
• Abstract: Occupational outcomes like entrepreneurship are generally considered personal information that individuals should have the autonomy to disclose. With the advancing capability of artificial intelligence (AI) to infer private details from widely available human-centric data, such as social media, it is crucial to investigate whether AI can accurately extract private occupational information from such data. In this study, we demonstrate that deep neural networks can classify individuals as entrepreneurs based on a single facial image with high accuracy in data sourced from Crunchbase, a premier source for entrepreneurship data. Utilizing a dataset comprising facial images of 40,728 individuals, including both entrepreneurs and non-entrepreneurs, we trained a Convolutional Neural Network (CNN) and evaluated its classification performance. While human experts (n=650) and trained participants (n=133) were unable to classify entrepreneurs with accuracy above chance levels (>50%), the AI model achieved a classification accuracy of 79.51%. Several robustness tests show that this high level of accuracy is maintained under various conditions.

Title: Assessing the Uncertainty and Robustness of Object Detection Models for Detecting Stickers on Laptops

• Authors: Chengjie Lu, Jiahui Wu, Shaukat Ali, Mikkel Labori Olsen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03782
• Pdf URL: https://arxiv.org/pdf/2409.03782
• Copy Paste: [[2409.03782]] Assessing the Uncertainty and Robustness of Object Detection Models for Detecting Stickers on Laptops(https://arxiv.org/abs/2409.03782)
• Keywords: robust, diffusion
• Abstract: Refurbishing laptops extends their lives while contributing to reducing electronic waste, which promotes building a sustainable future. To this end, the Danish Technological Institute (DTI) focuses on the research and development of several applications, including laptop refurbishing. This has several steps, including cleaning, which involves identifying and removing stickers from laptop surfaces. DTI trained six sticker detection models (SDMs) based on open-source object detection models to identify such stickers precisely so these stickers can be removed automatically. However, given the diversity in types of stickers (e.g., shapes, colors, locations), identification of the stickers is highly uncertain, thereby requiring explicit quantification of uncertainty associated with the identified stickers. Such uncertainty quantification can help reduce risks in removing stickers, which, for example, could otherwise result in damaging laptop surfaces. For uncertainty quantification, we adopted the Monte Carlo Dropout method to evaluate the six SDMs from DTI using three datasets: the original image dataset from DTI and two datasets generated with vision language models, i.e., DALL-E-3 and Stable Diffusion-3. In addition, we presented novel robustness metrics concerning detection accuracy and uncertainty to assess the robustness of the SDMs based on adversarial datasets generated from the three datasets using a dense adversary method. Our evaluation results show that different SDMs perform differently regarding different metrics. Based on the results, we provide SDM selection guidelines and lessons learned from various perspectives.

Title: HSF: Defending against Jailbreak Attacks with Hidden State Filtering

• Authors: Cheng Qian, Hainan Zhang, Lei Sha, Zhiming Zheng
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03788
• Pdf URL: https://arxiv.org/pdf/2409.03788
• Copy Paste: [[2409.03788]] HSF: Defending against Jailbreak Attacks with Hidden State Filtering(https://arxiv.org/abs/2409.03788)
• Keywords: defense, attack
• Abstract: With the growing deployment of LLMs in daily applications like chatbots and content generation, efforts to ensure outputs align with human values and avoid harmful content have intensified. However, increasingly sophisticated jailbreak attacks threaten this alignment, aiming to induce unsafe outputs. Current defense efforts either focus on prompt rewriting or detection, which are limited in effectiveness due to the various design of jailbreak prompts, or on output control and detection, which are computationally expensive as they require LLM inference. Therefore, designing a pre-inference defense method that resists diverse jailbreak prompts is crucial for preventing LLM jailbreak attacks. We observe that jailbreak attacks, safe queries, and harmful queries exhibit different clustering patterns within the LLM's hidden state representation space. This suggests that by leveraging the LLM's hidden state representational capabilities, we can analyze the LLM's forthcoming behavior and proactively intervene for defense. In this paper, we propose a jailbreak attack defense strategy based on a Hidden State Filter (HSF), a lossless architectural defense mechanism that enables the model to preemptively identify and reject adversarial inputs before the inference process begins. We activate its defensive potential through an additional plugin module, effectively framing the defense task as a classification problem. Experimental results on two benchmark datasets, utilizing three different LLMs, show that HSF significantly enhances resilience against six cutting-edge jailbreak attacks. It significantly reduces the success rate of jailbreak attacks while minimally impacting responses to benign user queries, with negligible inference overhead, and outperforming defense baselines.Our code and data are available at https://anonymous.4open.science/r/Hidden-State-Filtering-8652/

Title: BreachSeek: A Multi-Agent Automated Penetration Tester

• Authors: Ibrahim Alshehri, Adnan Alshehri, Abdulrahman Almalki, Majed Bamardouf, Alaqsa Akbar
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03789
• Pdf URL: https://arxiv.org/pdf/2409.03789
• Copy Paste: [[2409.03789]] BreachSeek: A Multi-Agent Automated Penetration Tester(https://arxiv.org/abs/2409.03789)
• Keywords: security, attack, large language model
• Abstract: The increasing complexity and scale of modern digital environments have exposed significant gaps in traditional cybersecurity penetration testing methods, which are often time-consuming, labor-intensive, and unable to rapidly adapt to emerging threats. There is a critical need for an automated solution that can efficiently identify and exploit vulnerabilities across diverse systems without extensive human intervention. BreachSeek addresses this challenge by providing an AI-driven multi-agent software platform that leverages Large Language Models (LLMs) integrated through LangChain and LangGraph in Python. This system enables autonomous agents to conduct thorough penetration testing by identifying vulnerabilities, simulating a variety of cyberattacks, executing exploits, and generating comprehensive security reports. In preliminary evaluations, BreachSeek successfully exploited vulnerabilities in exploitable machines within local networks, demonstrating its practical effectiveness. Future developments aim to expand its capabilities, positioning it as an indispensable tool for cybersecurity professionals.

Title: Unveiling the Digital Fingerprints: Analysis of Internet attacks based on website fingerprints

• Authors: Blerim Rexha, Arbena Musa, Kamer Vishi, Edlira Martiri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.03791
• Pdf URL: https://arxiv.org/pdf/2409.03791
• Copy Paste: [[2409.03791]] Unveiling the Digital Fingerprints: Analysis of Internet attacks based on website fingerprints(https://arxiv.org/abs/2409.03791)
• Keywords: privacy, protect, attack
• Abstract: Parallel to our physical activities our virtual presence also leaves behind our unique digital fingerprints, while navigating on the Internet. These digital fingerprints have the potential to unveil users' activities encompassing browsing history, utilized applications, and even devices employed during these engagements. Many Internet users tend to use web browsers that provide the highest privacy protection and anonymization such as Tor. The success of such privacy protection depends on the Tor feature to anonymize end-user IP addresses and other metadata that constructs the website fingerprint. In this paper, we show that using the newest machine learning algorithms an attacker can deanonymize Tor traffic by applying such techniques. In our experimental framework, we establish a baseline and comparative reference point using a publicly available dataset from Universidad Del Cauca, Colombia. We capture network packets across 11 days, while users navigate specific web pages, recording data in .pcapng format through the Wireshark network capture tool. Excluding extraneous packets, we employ various machine learning algorithms in our analysis. The results show that the Gradient Boosting Machine algorithm delivers the best outcomes in binary classification, achieving an accuracy of 0.8363. In the realm of multi-class classification, the Random Forest algorithm attains an accuracy of 0.6297.

Title: Safeguarding AI Agents: Developing and Analyzing Safety Architectures

• Authors: Ishaan Domkundwar, Mukunda N S
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03793
• Pdf URL: https://arxiv.org/pdf/2409.03793
• Copy Paste: [[2409.03793]] Safeguarding AI Agents: Developing and Analyzing Safety Architectures(https://arxiv.org/abs/2409.03793)
• Keywords: security, attack, robust, large language model
• Abstract: AI agents, specifically powered by large language models, have demonstrated exceptional capabilities in various applications where precision and efficacy are necessary. However, these agents come with inherent risks, including the potential for unsafe or biased actions, vulnerability to adversarial attacks, lack of transparency, and tendency to generate hallucinations. As AI agents become more prevalent in critical sectors of the industry, the implementation of effective safety protocols becomes increasingly important. This paper addresses the critical need for safety measures in AI systems, especially ones that collaborate with human teams. We propose and evaluate three frameworks to enhance safety protocols in AI agent systems: an LLM-powered input-output filter, a safety agent integrated within the system, and a hierarchical delegation-based system with embedded safety checks. Our methodology involves implementing these frameworks and testing them against a set of unsafe agentic use cases, providing a comprehensive evaluation of their effectiveness in mitigating risks associated with AI agent deployment. We conclude that these frameworks can significantly strengthen the safety and security of AI agent systems, minimizing potential harmful actions or outputs. Our work contributes to the ongoing effort to create safe and reliable AI applications, particularly in automated operations, and provides a foundation for developing robust guardrails to ensure the responsible use of AI agents in real-world applications.

Title: Security Implications and Mitigation Strategies in MPLS Networks

• Authors: Ayush Thakur
• Subjects: cs.CR, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.03795
• Pdf URL: https://arxiv.org/pdf/2409.03795
• Copy Paste: [[2409.03795]] Security Implications and Mitigation Strategies in MPLS Networks(https://arxiv.org/abs/2409.03795)
• Keywords: security, attack
• Abstract: Multiprotocol Label Switching (MPLS) is a high-performance telecommunications technology that directs data from one network node to another based on short path labels rather than long network addresses. Its efficiency and scalability have made it a popular choice for large-scale and enterprise networks. However, as MPLS networks grow and evolve, they encounter various security challenges. This paper explores the security implications associated with MPLS networks, including risks such as label spoofing, traffic interception, and denial of service attacks. Additionally, it evaluates advanced mitigation strategies to address these vulnerabilities, leveraging mathematical models and security protocols to enhance MPLS network resilience. By integrating theoretical analysis with practical solutions, this paper aims to provide a comprehensive understanding of MPLS security and propose effective methods for safeguarding network infrastructure.

Title: Protecting Activity Sensing Data Privacy Using Hierarchical Information Dissociation

• Authors: Guangjing Wang, Hanqing Guo, Yuanda Wang, Bocheng Chen, Ce Zhou, Qiben Yan
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03796
• Pdf URL: https://arxiv.org/pdf/2409.03796
• Copy Paste: [[2409.03796]] Protecting Activity Sensing Data Privacy Using Hierarchical Information Dissociation(https://arxiv.org/abs/2409.03796)
• Keywords: privacy, protect, diffusion
• Abstract: Smartphones and wearable devices have been integrated into our daily lives, offering personalized services. However, many apps become overprivileged as their collected sensing data contains unnecessary sensitive information. For example, mobile sensing data could reveal private attributes (e.g., gender and age) and unintended sensitive features (e.g., hand gestures when entering passwords). To prevent sensitive information leakage, existing methods must obtain private labels and users need to specify privacy policies. However, they only achieve limited control over information disclosure. In this work, we present Hippo to dissociate hierarchical information including private metadata and multi-grained activity information from the sensing data. Hippo achieves fine-grained control over the disclosure of sensitive information without requiring private labels. Specifically, we design a latent guidance-based diffusion model, which generates multi-grained versions of raw sensor data conditioned on hierarchical latent activity features. Hippo enables users to control the disclosure of sensitive information in sensing data, ensuring their privacy while preserving the necessary features to meet the utility requirements of applications. Hippo is the first unified model that achieves two goals: perturbing the sensitive attributes and controlling the disclosure of sensitive information in mobile sensing data. Extensive experiments show that Hippo can anonymize personal attributes and transform activity information at various resolutions across different types of sensing data.

Title: Interpretable Cyber Threat Detection for Enterprise Industrial Networks: A Computational Design Science Approach

• Authors: Prabhat Kumar, A.K.M. Najmul Islam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.03798
• Pdf URL: https://arxiv.org/pdf/2409.03798
• Copy Paste: [[2409.03798]] Interpretable Cyber Threat Detection for Enterprise Industrial Networks: A Computational Design Science Approach(https://arxiv.org/abs/2409.03798)
• Keywords: secure, security, privacy, interpretability, generative
• Abstract: Enterprise industrial networks face threats that risk data and operations. However, designing efficient threat detection system is challenging due to data scarcity, especially where privacy is a concern. The complexity of enterprise industrial network data adds to this challenge, causing high false positives and interpretation issues. Towards this, we use IS computational design science paradigm to develop a two-stage cyber threat detection system for enterprise-level IS that are both secure and capable of adapting to evolving technological and business environments. The first stage generates synthetic industrial network data using a modified generative adversarial network. The second stage develops a novel bidirectional gated recurrent unit and a modified attention mechanism for effective threat detection. We also use shapley additive explanations and a decision tree technique for enhancing interpretability. Our analysis on two public datasets shows the frameworks high precision in threat detection and offers practical cybersecurity solutions and methodological advancements.

Title: Neural Entropy

• Authors: Akhil Premkumar
• Subjects: cs.LG, cond-mat.stat-mech, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.03817
• Pdf URL: https://arxiv.org/pdf/2409.03817
• Copy Paste: [[2409.03817]] Neural Entropy(https://arxiv.org/abs/2409.03817)
• Keywords: diffusion, generative
• Abstract: We examine the connection between deep learning and information theory through the paradigm of diffusion models. Using well-established principles from non-equilibrium thermodynamics we can characterize the amount of information required to reverse a diffusive process. Neural networks store this information and operate in a manner reminiscent of Maxwell's demon during the generative stage. We illustrate this cycle using a novel diffusion scheme we call the entropy matching model, wherein the information conveyed to the network during training exactly corresponds to the entropy that must be negated during reversal. We demonstrate that this entropy can be used to analyze the encoding efficiency and storage capacity of the network. This conceptual picture blends elements of stochastic optimal control, thermodynamics, information theory, and optimal transport, and raises the prospect of applying diffusion models as a test bench to understand neural networks.

Title: Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoption

• Authors: Wenchao Dong, Assem Zhunis, Dongyoung Jeong, Hyojin Chin, Jiyoung Han, Meeyoung Cha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03843
• Pdf URL: https://arxiv.org/pdf/2409.03843
• Copy Paste: [[2409.03843]] Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoption(https://arxiv.org/abs/2409.03843)
• Keywords: large language model
• Abstract: Drawing parallels between human cognition and artificial intelligence, we explored how large language models (LLMs) internalize identities imposed by targeted prompts. Informed by Social Identity Theory, these identity assignments lead LLMs to distinguish between "we" (the ingroup) and "they" (the outgroup). This self-categorization generates both ingroup favoritism and outgroup bias. Nonetheless, existing literature has predominantly focused on ingroup favoritism, often overlooking outgroup bias, which is a fundamental source of intergroup prejudice and discrimination. Our experiment addresses this gap by demonstrating that outgroup bias manifests as strongly as ingroup favoritism. Furthermore, we successfully mitigated the inherent pro-liberal, anti-conservative bias in LLMs by guiding them to adopt the perspectives of the initially disfavored group. These results were replicated in the context of gender bias. Our findings highlight the potential to develop more equitable and balanced language models.

Title: Sirius: Contextual Sparsity with Correction for Efficient LLMs

• Authors: Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Victoria Lin, Beidi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03856
• Pdf URL: https://arxiv.org/pdf/2409.03856
• Copy Paste: [[2409.03856]] Sirius: Contextual Sparsity with Correction for Efficient LLMs(https://arxiv.org/abs/2409.03856)
• Keywords: large language model
• Abstract: With the blossom of large language models (LLMs), inference efficiency becomes increasingly important. Various approximation methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without quality degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, CS significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models often share general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces Sirius, an efficient correction mechanism, which significantly recovers CS models quality on reasoning tasks while maintaining its efficiency gain. Sirius is evaluated on 6 models with 8 difficult generation tasks in reasoning, math, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for Sirius and show that Sirius achieves roughly 20% reduction in latency for 8B model on-chip and 35% reduction for 70B model offloading. We open-source our implementation of Sirius at this https URL.

Title: Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?

• Authors: Peizhong Ju, Haibo Yang, Jia Liu, Yingbin Liang, Ness Shroff
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03863
• Pdf URL: https://arxiv.org/pdf/2409.03863
• Copy Paste: [[2409.03863]] Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?(https://arxiv.org/abs/2409.03863)
• Keywords: federate
• Abstract: Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms along with their optimization analyses have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention. This lack of investigation can be attributed to the complex interplay between data heterogeneity and infrequent communication due to the local updates within the FL framework. This motivates us to investigate a fundamental question in FL: Can we quantify the impact of data heterogeneity and local updates on the generalization performance for FL as the learning process evolves? To this end, we conduct a comprehensive theoretical study of FL's generalization performance using a linear model as the first step, where the data heterogeneity is considered for both the stationary and online/non-stationary cases. By providing closed-form expressions of the model error, we rigorously quantify the impact of the number of the local updates (denoted as $K$) under three settings ($K=1$, $K<\infty$, and $K=\infty$) and show how the generalization performance evolves with the number of rounds $t$. Our investigation also provides a comprehensive understanding of how different configurations (including the number of model parameters $p$ and the number of training samples $n$) contribute to the overall generalization performance, thus shedding new insights (such as benign overfitting) for implementing FL over networks.

Title: The Influence of Faulty Labels in Data Sets on Human Pose Estimation

• Authors: Arnold Schwarz, Levente Hernadi, Felix Bießmann, Kristian Hildebrand
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.03887
• Pdf URL: https://arxiv.org/pdf/2409.03887
• Copy Paste: [[2409.03887]] The Influence of Faulty Labels in Data Sets on Human Pose Estimation(https://arxiv.org/abs/2409.03887)
• Keywords: robust
• Abstract: In this study we provide empirical evidence demonstrating that the quality of training data impacts model performance in Human Pose Estimation (HPE). Inaccurate labels in widely used data sets, ranging from minor errors to severe mislabeling, can negatively influence learning and distort performance metrics. We perform an in-depth analysis of popular HPE data sets to show the extent and nature of label inaccuracies. Our findings suggest that accounting for the impact of faulty labels will facilitate the development of more robust and accurate HPE models for a variety of real-world applications. We show improved performance with cleansed data.

Title: MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition

• Authors: Mallika Garg, Debashis Ghosh, Pyari Mohan Pradhan
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.03890
• Pdf URL: https://arxiv.org/pdf/2409.03890
• Copy Paste: [[2409.03890]] MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition(https://arxiv.org/abs/2409.03890)
• Keywords: transformer
• Abstract: In this paper, we introduce a novel Multiscale Video Transformer Network (MVTN) for dynamic hand gesture recognition, since multiscale features can extract features with variable size, pose, and shape of hand which is a challenge in hand gesture recognition. The proposed model incorporates a multiscale feature hierarchy to capture diverse levels of detail and context within hand gestures which enhances the model's ability. This multiscale hierarchy is obtained by extracting different dimensions of attention in different transformer stages with initial stages to model high-resolution features and later stages to model low-resolution features. Our approach also leverages multimodal data, utilizing depth maps, infrared data, and surface normals along with RGB images from NVGesture and Briareo datasets. Experiments show that the proposed MVTN achieves state-of-the-art results with less computational complexity and parameters. The source code is available at this https URL.

Title: Understanding Fairness Metrics in Recommender Systems: A Healthcare Perspective

• Authors: Veronica Kecki, Alan Said
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.03893
• Pdf URL: https://arxiv.org/pdf/2409.03893
• Copy Paste: [[2409.03893]] Understanding Fairness Metrics in Recommender Systems: A Healthcare Perspective(https://arxiv.org/abs/2409.03893)
• Keywords: fair
• Abstract: Fairness in AI-driven decision-making systems has become a critical concern, especially when these systems directly affect human lives. This paper explores the public's comprehension of fairness in healthcare recommendations. We conducted a survey where participants selected from four fairness metrics -- Demographic Parity, Equal Accuracy, Equalized Odds, and Positive Predictive Value -- across different healthcare scenarios to assess their understanding of these concepts. Our findings reveal that fairness is a complex and often misunderstood concept, with a generally low level of public understanding regarding fairness metrics in recommender systems. This study highlights the need for enhanced information and education on algorithmic fairness to support informed decision-making in using these systems. Furthermore, the results suggest that a one-size-fits-all approach to fairness may be insufficient, pointing to the importance of context-sensitive designs in developing equitable AI systems.

Title: On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments

• Authors: Muxing Wang, Pengkun Yang, Lili Su
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.03897
• Pdf URL: https://arxiv.org/pdf/2409.03897
• Copy Paste: [[2409.03897]] On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments(https://arxiv.org/abs/2409.03897)
• Keywords: federate
• Abstract: Large-scale multi-agent systems are often deployed across wide geographic areas, where agents interact with heterogeneous environments. There is an emerging interest in understanding the role of heterogeneity in the performance of the federated versions of classic reinforcement learning algorithms. In this paper, we study synchronous federated Q-learning, which aims to learn an optimal Q-function by having $K$ agents average their local Q-estimates per $E$ iterations. We observe an interesting phenomenon on the convergence speeds in terms of $K$ and $E$. Similar to the homogeneous environment settings, there is a linear speed-up concerning $K$ in reducing the errors that arise from sampling randomness. Yet, in sharp contrast to the homogeneous settings, $E>1$ leads to significant performance degradation. Specifically, we provide a fine-grained characterization of the error evolution in the presence of environmental heterogeneity, which decay to zero as the number of iterations $T$ increases. The slow convergence of having $E>1$ turns out to be fundamental rather than an artifact of our analysis. We prove that, for a wide range of stepsizes, the $\ell_{\infty}$ norm of the error cannot decay faster than $\Theta (E/T)$. In addition, our experiments demonstrate that the convergence exhibits an interesting two-phase phenomenon. For any given stepsize, there is a sharp phase-transition of the convergence: the error decays rapidly in the beginning yet later bounces up and stabilizes. Provided that the phase-transition time can be estimated, choosing different stepsizes for the two phases leads to faster overall convergence.

Title: On-board Satellite Image Classification for Earth Observation: A Comparative Study of Pre-Trained Vision Transformer Models

• Authors: Thanh-Dung Le, Vu Nguyen Ha, Ti Ti Nguyen, Geoffrey Eappen, Prabhu Thiruvasagam, Luis M. Garces-Socarras, Hong-fu Chou, Jorge L. Gonzalez-Rios, Juan Carlos Merlano-Duncan, Symeon Chatzinotas
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.03901
• Pdf URL: https://arxiv.org/pdf/2409.03901
• Copy Paste: [[2409.03901]] On-board Satellite Image Classification for Earth Observation: A Comparative Study of Pre-Trained Vision Transformer Models(https://arxiv.org/abs/2409.03901)
• Keywords: robust, transformer
• Abstract: Remote sensing image classification is a critical component of Earth observation (EO) systems, traditionally dominated by convolutional neural networks (CNNs) and other deep learning techniques. However, the advent of Transformer-based architectures and large-scale pre-trained models has significantly shifted, offering enhanced performance and efficiency. This study focuses on identifying the most effective pre-trained model for land use classification in onboard satellite processing, emphasizing achieving high accuracy, computational efficiency, and robustness against noisy data conditions commonly encountered during satellite-based inference. Through extensive experimentation, we compared traditional CNN-based models, ResNet-based models, and various pre-trained vision Transformer models. Our findings demonstrate that pre-trained Transformer models, particularly MobileViTV2 and EfficientViT-M2, outperform models trained from scratch in accuracy and efficiency. These models achieve high performance with reduced computational requirements and exhibit greater resilience during inference under noisy conditions. While MobileViTV2 excelled on clean validation data, EfficientViT-M2 proved more robust when handling noise, making it the most suitable model for onboard satellite Earth observation tasks. In conclusion, EfficientViT-M2 is the optimal choice for reliable and efficient remote sensing image classification in satellite operations, achieving 98.76\% accuracy, precision, and recall. Specifically, EfficientViT-M2 delivered the highest performance across all metrics, excelled in training efficiency (1,000s) and inference time (10s), and demonstrated greater robustness (overall robustness score at 0.79).

Title: WaterMAS: Sharpness-Aware Maximization for Neural Network Watermarking

• Authors: Carl De Sousa Trias, Mihai Mitrea, Attilio Fiandrotti, Marco Cagnazzo, Sumanta Chaudhuri, Enzo Tartaglione
• Subjects: cs.LG, cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.03902
• Pdf URL: https://arxiv.org/pdf/2409.03902
• Copy Paste: [[2409.03902]] WaterMAS: Sharpness-Aware Maximization for Neural Network Watermarking(https://arxiv.org/abs/2409.03902)
• Keywords: security, protect, attack, robust, watermark, segmentation
• Abstract: Nowadays, deep neural networks are used for solving complex tasks in several critical applications and protecting both their integrity and intellectual property rights (IPR) has become of utmost importance. To this end, we advance WaterMAS, a substitutive, white-box neural network watermarking method that improves the trade-off among robustness, imperceptibility, and computational complexity, while making provisions for increased data payload and security. WasterMAS insertion keeps unchanged the watermarked weights while sharpening their underlying gradient space. The robustness is thus ensured by limiting the attack's strength: even small alterations of the watermarked weights would impact the model's performance. The imperceptibility is ensured by inserting the watermark during the training process. The relationship among the WaterMAS data payload, imperceptibility, and robustness properties is discussed. The secret key is represented by the positions of the weights conveying the watermark, randomly chosen through multiple layers of the model. The security is evaluated by investigating the case in which an attacker would intercept the key. The experimental validations consider 5 models and 2 tasks (VGG16, ResNet18, MobileNetV3, SwinT for CIFAR10 image classification, and DeepLabV3 for Cityscapes image segmentation) as well as 4 types of attacks (Gaussian noise addition, pruning, fine-tuning, and quantization). The code will be released open-source upon acceptance of the article.

Title: CACER: Clinical Concept Annotations for Cancer Events and Relations

• Authors: Yujuan Fu, Giridhar Kaushik Ramachandran, Ahmad Halwani, Bridget T. McInnes, Fei Xia, Kevin Lybarger, Meliha Yetisgen, Özlem Uzuner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03905
• Pdf URL: https://arxiv.org/pdf/2409.03905
• Copy Paste: [[2409.03905]] CACER: Clinical Concept Annotations for Cancer Events and Relations(https://arxiv.org/abs/2409.03905)
• Keywords: extraction, transformer
• Abstract: Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. We present Clinical Concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48,000 medical problems and drug events and 10,000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction (IE) models such as BERT, Flan-T5, Llama3, and GPT-4 using fine-tuning and in-context learning (ICL). In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks. The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, LLMs offer no performance advantage over the smaller BERT models. The results emphasize the need for annotated training data to optimize models. Multiple fine-tuned transformer models achieved performance comparable to IAA for several extraction tasks.

Title: The Role of Generative Systems in Historical Photography Management: A Case Study on Catalan Archives

• Authors: Èric Śanchez, Adrià Molina, Oriol Ramos Terrades
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03911
• Pdf URL: https://arxiv.org/pdf/2409.03911
• Copy Paste: [[2409.03911]] The Role of Generative Systems in Historical Photography Management: A Case Study on Catalan Archives(https://arxiv.org/abs/2409.03911)
• Keywords: generative
• Abstract: The use of image analysis in automated photography management is an increasing trend in heritage institutions. Such tools alleviate the human cost associated with the manual and expensive annotation of new data sources while facilitating fast access to the citizenship through online indexes and search engines. However, available tagging and description tools are usually designed around modern photographs in English, neglecting historical corpora in minoritized languages, each of which exhibits intrinsic particularities. The primary objective of this research is to study the quantitative contribution of generative systems in the description of historical sources. This is done by contextualizing the task of captioning historical photographs from the Catalan archives as a case study. Our findings provide practitioners with tools and directions on transfer learning for captioning models based on visual adaptation and linguistic proximity.

Title: Data-Efficient Generation for Dataset Distillation

• Authors: Zhe Li, Weitong Zhang, Sarah Cechnicka, Bernhard Kainz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03929
• Pdf URL: https://arxiv.org/pdf/2409.03929
• Copy Paste: [[2409.03929]] Data-Efficient Generation for Dataset Distillation(https://arxiv.org/abs/2409.03929)
• Keywords: diffusion
• Abstract: While deep learning techniques have proven successful in image-related tasks, the exponentially increased data storage and computation costs become a significant challenge. Dataset distillation addresses these challenges by synthesizing only a few images for each class that encapsulate all essential information. Most current methods focus on matching. The problems lie in the synthetic images not being human-readable and the dataset performance being insufficient for downstream learning tasks. Moreover, the distillation time can quickly get out of bounds when the number of synthetic images per class increases even slightly. To address this, we train a class conditional latent diffusion model capable of generating realistic synthetic images with labels. The sampling time can be reduced to several tens of images per seconds. We demonstrate that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set. Our approach achieved rank \(1\) in The First Dataset Distillation Challenge at ECCV 2024 on the CIFAR100 and TinyImageNet datasets.

Title: Experimentation in Content Moderation using RWKV

• Authors: Umut Yildirim, Rohan Dutta, Burak Yildirim, Atharva Vaidya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.03939
• Pdf URL: https://arxiv.org/pdf/2409.03939
• Copy Paste: [[2409.03939]] Experimentation in Content Moderation using RWKV(https://arxiv.org/abs/2409.03939)
• Keywords: large language model
• Abstract: This paper investigates the RWKV model's efficacy in content moderation through targeted experimentation. We introduce a novel dataset specifically designed for distillation into smaller models, enhancing content moderation practices. This comprehensive dataset encompasses images, videos, sounds, and text data that present societal challenges. Leveraging advanced Large Language Models (LLMs), we generated an extensive set of responses -- 558,958 for text and 83,625 for images -- to train and refine content moderation systems. Our core experimentation involved fine-tuning the RWKV model, capitalizing on its CPU-efficient architecture to address large-scale content moderation tasks. By highlighting the dataset's potential for knowledge distillation, this study not only demonstrates RWKV's capability in improving the accuracy and efficiency of content moderation systems but also paves the way for developing more compact, resource-efficient models in this domain. Datasets and models can be found in HuggingFace: this https URL

Title: HUMOS: Human Motion Model Conditioned on Body Shape

• Authors: Shashank Tripathi, Omid Taheri, Christoph Lassner, Michael J. Black, Daniel Holden, Carsten Stoll
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03944
• Pdf URL: https://arxiv.org/pdf/2409.03944
• Copy Paste: [[2409.03944]] HUMOS: Human Motion Model Conditioned on Body Shape(https://arxiv.org/abs/2409.03944)
• Keywords: generative
• Abstract: Generating realistic human motion is essential for many computer vision and graphics applications. The wide variety of human body shapes and sizes greatly impacts how people move. However, most existing motion models ignore these differences, relying on a standardized, average body. This leads to uniform motion across different body types, where movements don't match their physical characteristics, limiting diversity. To solve this, we introduce a new approach to develop a generative motion model based on body shape. We show that it's possible to train this model using unpaired data by applying cycle consistency, intuitive physics, and stability constraints, which capture the relationship between identity and movement. The resulting model generates diverse, physically plausible, and dynamically stable human motions that are both quantitatively and qualitatively more realistic than current state-of-the-art methods. More details are available on our project page this https URL.

Title: FODA-PG for Enhanced Medical Imaging Narrative Generation: Adaptive Differentiation of Normal and Abnormal Attributes

• Authors: Kai Shu, Yuzhuo Jia, Ziyang Zhang, Jiechao Gao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.03947
• Pdf URL: https://arxiv.org/pdf/2409.03947
• Copy Paste: [[2409.03947]] FODA-PG for Enhanced Medical Imaging Narrative Generation: Adaptive Differentiation of Normal and Abnormal Attributes(https://arxiv.org/abs/2409.03947)
• Keywords: transformer
• Abstract: Automatic Medical Imaging Narrative generation aims to alleviate the workload of radiologists by producing accurate clinical descriptions directly from radiological images. However, the subtle visual nuances and domain-specific terminology in medical images pose significant challenges compared to generic image captioning tasks. Existing approaches often neglect the vital distinction between normal and abnormal findings, leading to suboptimal performance. In this work, we propose FODA-PG, a novel Fine-grained Organ-Disease Adaptive Partitioning Graph framework that addresses these limitations through domain-adaptive learning. FODA-PG constructs a granular graphical representation of radiological findings by separating disease-related attributes into distinct "disease-specific" and "disease-free" categories based on their clinical significance and location. This adaptive partitioning enables our model to capture the nuanced differences between normal and pathological states, mitigating the impact of data biases. By integrating this fine-grained semantic knowledge into a powerful transformer-based architecture and providing rigorous mathematical justifications for its effectiveness, FODA-PG generates precise and clinically coherent reports with enhanced generalization capabilities. Extensive experiments on the IU-Xray and MIMIC-CXR benchmarks demonstrate the superiority of our approach over state-of-the-art methods, highlighting the importance of domain adaptation in medical report generation.

Title: Boundary feature fusion network for tooth image segmentation

• Authors: Dongping Zhang, Zheng Li, Fangao Zeng, Yutong Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.03982
• Pdf URL: https://arxiv.org/pdf/2409.03982
• Copy Paste: [[2409.03982]] Boundary feature fusion network for tooth image segmentation(https://arxiv.org/abs/2409.03982)
• Keywords: extraction, segmentation
• Abstract: Tooth segmentation is a critical technology in the field of medical image segmentation, with applications ranging from orthodontic treatment to human body identification and dental pathology assessment. Despite the development of numerous tooth image segmentation models by researchers, a common shortcoming is the failure to account for the challenges of blurred tooth boundaries. Dental diagnostics require precise delineation of tooth boundaries. This paper introduces an innovative tooth segmentation network that integrates boundary information to address the issue of indistinct boundaries between teeth and adjacent tissues. This network's core is its boundary feature extraction module, which is designed to extract detailed boundary information from high-level features. Concurrently, the feature cross-fusion module merges detailed boundary and global semantic information in a synergistic way, allowing for stepwise layer transfer of feature information. This method results in precise tooth segmentation. In the most recent STS Data Challenge, our methodology was rigorously tested and received a commendable overall score of 0.91. When compared to other existing approaches, this score demonstrates our method's significant superiority in segmenting tooth boundaries.

Title: An Efficient and Generalizable Symbolic Regression Method for Time Series Analysis

• Authors: Yi Xie, Tianyu Qiu, Yun Xiong, Xiuqi Huang, Xiaofeng Gao, Chao Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.03986
• Pdf URL: https://arxiv.org/pdf/2409.03986
• Copy Paste: [[2409.03986]] An Efficient and Generalizable Symbolic Regression Method for Time Series Analysis(https://arxiv.org/abs/2409.03986)
• Keywords: interpretability
• Abstract: Time series analysis and prediction methods currently excel in quantitative analysis, offering accurate future predictions and diverse statistical indicators, but generally falling short in elucidating the underlying evolution patterns of time series. To gain a more comprehensive understanding and provide insightful explanations, we utilize symbolic regression techniques to derive explicit expressions for the non-linear dynamics in the evolution of time series variables. However, these techniques face challenges in computational efficiency and generalizability across diverse real-world time series data. To overcome these challenges, we propose \textbf{N}eural-\textbf{E}nhanced \textbf{Mo}nte-Carlo \textbf{T}ree \textbf{S}earch (NEMoTS) for time series. NEMoTS leverages the exploration-exploitation balance of Monte-Carlo Tree Search (MCTS), significantly reducing the search space in symbolic regression and improving expression quality. Furthermore, by integrating neural networks with MCTS, NEMoTS not only capitalizes on their superior fitting capabilities to concentrate on more pertinent operations post-search space reduction, but also replaces the complex and time-consuming simulation process, thereby substantially improving computational efficiency and generalizability in time series analysis. NEMoTS offers an efficient and comprehensive approach to time series analysis. Experiments with three real-world datasets demonstrate NEMoTS's significant superiority in performance, efficiency, reliability, and interpretability, making it well-suited for large-scale real-world time series data.

Title: Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance

• Authors: RenMing Huang, Shaochong Liu, Yunqiang Pei, Peng Wang, Guoqing Wang, Yang Yang, Hengtao Shen
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.03996
• Pdf URL: https://arxiv.org/pdf/2409.03996
• Copy Paste: [[2409.03996]] Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance(https://arxiv.org/abs/2409.03996)
• Keywords: robust, diffusion
• Abstract: In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions.

Title: DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes

• Authors: Jianbiao Mei, Yukai Ma, Xuemeng Yang, Licheng Wen, Tiantian Wei, Min Dou, Botian Shi, Yong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04003
• Pdf URL: https://arxiv.org/pdf/2409.04003
• Copy Paste: [[2409.04003]] DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes(https://arxiv.org/abs/2409.04003)
• Keywords: diffusion
• Abstract: Recent advances in diffusion models have significantly enhanced the cotrollable generation of streetscapes for and facilitated downstream perception and planning tasks. However, challenges such as maintaining temporal coherence, generating long videos, and accurately modeling driving scenes persist. Accordingly, we propose DreamForge, an advanced diffusion-based autoregressive video generation model designed for the long-term generation of 3D-controllable and extensible video. In terms of controllability, our DreamForge supports flexible conditions such as text descriptions, camera poses, 3D bounding boxes, and road layouts, while also providing perspective guidance to produce driving scenes that are both geometrically and contextually accurate. For consistency, we ensure inter-view consistency through cross-view attention and temporal coherence via an autoregressive architecture enhanced with motion cues. Codes will be available at this https URL.

Title: One-Shot Diffusion Mimicker for Handwritten Text Generation

• Authors: Gang Dai, Yifan Zhang, Quhui Ke, Qiangya Guo, Shuangping Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04004
• Pdf URL: https://arxiv.org/pdf/2409.04004
• Copy Paste: [[2409.04004]] One-Shot Diffusion Mimicker for Handwritten Text Generation(https://arxiv.org/abs/2409.04004)
• Keywords: extraction, diffusion
• Abstract: Existing handwritten text generation methods often require more than ten handwriting samples as style references. However, in practical applications, users tend to prefer a handwriting generation model that operates with just a single reference sample for its convenience and efficiency. This approach, known as "one-shot generation", significantly simplifies the process but poses a significant challenge due to the difficulty of accurately capturing a writer's style from a single sample, especially when extracting fine details from the characters' edges amidst sparse foreground and undesired background noise. To address this problem, we propose a One-shot Diffusion Mimicker (One-DM) to generate handwritten text that can mimic any calligraphic style with only one reference sample. Inspired by the fact that high-frequency information of the individual sample often contains distinct style patterns (e.g., character slant and letter joining), we develop a novel style-enhanced module to improve the style extraction by incorporating high-frequency components from a single sample. We then fuse the style features with the text content as a merged condition for guiding the diffusion model to produce high-quality handwritten text images. Extensive experiments demonstrate that our method can successfully generate handwriting scripts with just one sample reference in multiple languages, even outperforming previous methods using over ten samples. Our source code is available at this https URL.

Title: Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens for Text-to-Any-Task

• Authors: Jing Wang, Ao Ma, Jiasong Feng, Dawei Leng, Yuhui Yin, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04005
• Pdf URL: https://arxiv.org/pdf/2409.04005
• Copy Paste: [[2409.04005]] Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens for Text-to-Any-Task(https://arxiv.org/abs/2409.04005)
• Keywords: diffusion, transformer
• Abstract: The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy Token Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently. Specifically, in each transformer block, we randomly sample one token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 48% reduction compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code is available at this https URL.

Title: PlantSeg: A Large-Scale In-the-wild Dataset for Plant Disease Segmentation

• Authors: Tianqi Wei, Zhi Chen, Xin Yu, Scott Chapman, Paul Melloy, Zi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04038
• Pdf URL: https://arxiv.org/pdf/2409.04038
• Copy Paste: [[2409.04038]] PlantSeg: A Large-Scale In-the-wild Dataset for Plant Disease Segmentation(https://arxiv.org/abs/2409.04038)
• Keywords: robust, segmentation
• Abstract: Plant diseases pose significant threats to agriculture. It necessitates proper diagnosis and effective treatment to safeguard crop yields. To automate the diagnosis process, image segmentation is usually adopted for precisely identifying diseased regions, thereby advancing precision agriculture. Developing robust image segmentation models for plant diseases demands high-quality annotations across numerous images. However, existing plant disease datasets typically lack segmentation labels and are often confined to controlled laboratory settings, which do not adequately reflect the complexity of natural environments. Motivated by this fact, we established PlantSeg, a large-scale segmentation dataset for plant diseases. PlantSeg distinguishes itself from existing datasets in three key aspects. (1) Annotation type: Unlike the majority of existing datasets that only contain class labels or bounding boxes, each image in PlantSeg includes detailed and high-quality segmentation masks, associated with plant types and disease names. (2) Image source: Unlike typical datasets that contain images from laboratory settings, PlantSeg primarily comprises in-the-wild plant disease images. This choice enhances the practical applicability, as the trained models can be applied for integrated disease management. (3) Scale: PlantSeg is extensive, featuring 11,400 images with disease segmentation masks and an additional 8,000 healthy plant images categorized by plant type. Extensive technical experiments validate the high quality of PlantSeg's annotations. This dataset not only allows researchers to evaluate their image classification methods but also provides a critical foundation for developing and benchmarking advanced plant disease segmentation algorithms.

Title: A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakage

• Authors: Huan Yang, Deyu Zhang, Yudong Zhao, Yuanchun Li, Yunxin Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04040
• Pdf URL: https://arxiv.org/pdf/2409.04040
• Copy Paste: [[2409.04040]] A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakage(https://arxiv.org/abs/2409.04040)
• Keywords: secure, privacy, attack
• Abstract: Running LLMs on end devices has garnered significant attention recently due to their advantages in privacy preservation. With the advent of lightweight LLM models and specially designed GPUs, on-device LLM inference has achieved the necessary accuracy and performance metrics. However, we have identified that LLM inference on GPUs can leak privacy-sensitive intermediate information, specifically the KV pairs. An attacker could exploit these KV pairs to reconstruct the entire user conversation, leading to significant vulnerabilities. Existing solutions, such as Fully Homomorphic Encryption (FHE) and Trusted Execution Environments (TEE), are either too computation-intensive or resource-limited. To address these issues, we designed KV-Shield, which operates in two phases. In the initialization phase, it permutes the weight matrices so that all KV pairs are correspondingly permuted. During the runtime phase, the attention vector is inversely permuted to ensure the correctness of the layer output. All permutation-related operations are executed within the TEE, ensuring that insecure GPUs cannot access the original KV pairs, thus preventing conversation reconstruction. Finally, we theoretically analyze the correctness of KV-Shield, along with its advantages and overhead.

Title: Towards Safer Online Spaces: Simulating and Assessing Intervention Strategies for Eating Disorder Discussions

• Authors: Louis Penafiel, Hsien-Te Kao, Isabel Erickson, David Chu, Robert McCormack, Kristina Lerman, Svitlana Volkova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04043
• Pdf URL: https://arxiv.org/pdf/2409.04043
• Copy Paste: [[2409.04043]] Towards Safer Online Spaces: Simulating and Assessing Intervention Strategies for Eating Disorder Discussions(https://arxiv.org/abs/2409.04043)
• Keywords: generative
• Abstract: Eating disorders are complex mental health conditions that affect millions of people around the world. Effective interventions on social media platforms are crucial, yet testing strategies in situ can be risky. We present a novel LLM-driven experimental testbed for simulating and assessing intervention strategies in ED-related discussions. Our framework generates synthetic conversations across multiple platforms, models, and ED-related topics, allowing for controlled experimentation with diverse intervention approaches. We analyze the impact of various intervention strategies on conversation dynamics across four dimensions: intervention type, generative model, social media platform, and ED-related community/topic. We employ cognitive domain analysis metrics, including sentiment, emotions, etc., to evaluate the effectiveness of interventions. Our findings reveal that civility-focused interventions consistently improve positive sentiment and emotional tone across all dimensions, while insight-resetting approaches tend to increase negative emotions. We also uncover significant biases in LLM-generated conversations, with cognitive metrics varying notably between models (Claude-3 Haiku $>$ Mistral $>$ GPT-3.5-turbo $>$ LLaMA3) and even between versions of the same model. These variations highlight the importance of model selection in simulating realistic discussions related to ED. Our work provides valuable information on the complex dynamics of ED-related discussions and the effectiveness of various intervention strategies.

Title: Exploring User Privacy Awareness on GitHub: An Empirical Study

• Authors: Costanza Alfieri, Juri Di Rocco, Phuong T. Nguyen, Paola Inverardi
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.04048
• Pdf URL: https://arxiv.org/pdf/2409.04048
• Copy Paste: [[2409.04048]] Exploring User Privacy Awareness on GitHub: An Empirical Study(https://arxiv.org/abs/2409.04048)
• Keywords: security, privacy, protect, large language model
• Abstract: GitHub provides developers with a practical way to distribute source code and collaboratively work on common projects. To enhance account security and privacy, GitHub allows its users to manage access permissions, review audit logs, and enable two-factor authentication. However, despite the endless effort, the platform still faces various issues related to the privacy of its users. This paper presents an empirical study delving into the GitHub ecosystem. Our focus is on investigating the utilization of privacy settings on the platform and identifying various types of sensitive information disclosed by users. Leveraging a dataset comprising 6,132 developers, we report and analyze their activities by means of comments on pull requests. Our findings indicate an active engagement by users with the available privacy settings on GitHub. Notably, we observe the disclosure of different forms of private information within pull request comments. This observation has prompted our exploration into sensitivity detection using a large language model and BERT, to pave the way for a personalized privacy assistant. Our work provides insights into the utilization of existing privacy protection tools, such as privacy settings, along with their inherent limitations. Essentially, we aim to advance research in this field by providing both the motivation for creating such privacy protection tools and a proposed methodology for personalizing them.

Title: Self-Harmonized Chain of Thought

• Authors: Ziqi Jin, Wei Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04057
• Pdf URL: https://arxiv.org/pdf/2409.04057
• Copy Paste: [[2409.04057]] Self-Harmonized Chain of Thought(https://arxiv.org/abs/2409.04057)
• Keywords: large language model
• Abstract: Chain-of-Thought (CoT) prompting reveals that large language models are capable of performing complex reasoning via intermediate steps. CoT prompting is primarily categorized into three approaches. The first approach utilizes straightforward prompts like ``Let's think step by step'' to generate a sequential thought process before yielding an answer. The second approach makes use of human-crafted, step-by-step demonstrations to guide the model's reasoning process. The third automates the generation of reasoned demonstrations with the 'Let's think step by step'.This approach sometimes leads to reasoning errors, highlighting the need to diversify demonstrations to mitigate its misleading effects. However, diverse demonstrations pose challenges for effective representations. In this work, we propose ECHO, a self-harmonized chain-of-thought prompting method. It consolidates diverse solution paths into a uniform and effective solution pattern.ECHO demonstrates the best overall performance across three reasoning domains.

Title: D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detection

• Authors: Kentaro Hirahara, Chikahito Nakane, Hajime Ebisawa, Tsuyoshi Kuroda, Yohei Iwaki, Tomoyoshi Utsumi, Yuichiro Nomura, Makoto Koike, Hiroshi Mineno
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04060
• Pdf URL: https://arxiv.org/pdf/2409.04060
• Copy Paste: [[2409.04060]] D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detection(https://arxiv.org/abs/2409.04060)
• Keywords: diffusion, generative
• Abstract: In an agricultural field, plant phenotyping using object detection models is gaining attention. However, collecting the training data necessary to create generic and high-precision models is extremely challenging due to the difficulty of annotation and the diversity of domains. Furthermore, it is difficult to transfer training data across different crops, and although machine learning models effective for specific environments, conditions, or crops have been developed, they cannot be widely applied in actual fields. In this study, we propose a generative data augmentation method (D4) for vineyard shoot detection. D4 uses a pre-trained text-guided diffusion model based on a large number of original images culled from video data collected by unmanned ground vehicles or other means, and a small number of annotated datasets. The proposed method generates new annotated images with background information adapted to the target domain while retaining annotation information necessary for object detection. In addition, D4 overcomes the lack of training data in agriculture, including the difficulty of annotation and diversity of domains. We confirmed that this generative data augmentation method improved the mean average precision by up to 28.65% for the BBox detection task and the average precision by up to 13.73% for the keypoint detection task for vineyard shoot detection. Our generative data augmentation method D4 is expected to simultaneously solve the cost and domain diversity issues of training data generation in agriculture and improve the generalization performance of detection models.

Title: AnyMatch -- Efficient Zero-Shot Entity Matching with a Small Language Model

• Authors: Zeyu Zhang, Paul Groth, Iacer Calixto, Sebastian Schelter
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2409.04073
• Pdf URL: https://arxiv.org/pdf/2409.04073
• Copy Paste: [[2409.04073]] AnyMatch -- Efficient Zero-Shot Entity Matching with a Small Language Model(https://arxiv.org/abs/2409.04073)
• Keywords: large language model
• Abstract: Entity matching (EM) is the problem of determining whether two records refer to same real-world entity, which is crucial in data integration, e.g., for product catalogs or address databases. A major drawback of many EM approaches is their dependence on labelled examples. We thus focus on the challenging setting of zero-shot entity matching where no labelled examples are available for an unseen target dataset. Recently, large language models (LLMs) have shown promising results for zero-shot EM, but their low throughput and high deployment cost limit their applicability and scalability. We revisit the zero-shot EM problem with AnyMatch, a small language model fine-tuned in a transfer learning setup. We propose several novel data selection techniques to generate fine-tuning data for our model, e.g., by selecting difficult pairs to match via an AutoML filter, by generating additional attribute-level examples, and by controlling label imbalance in the data. We conduct an extensive evaluation of the prediction quality and deployment cost of our model, in a comparison to thirteen baselines on nine benchmark datasets. We find that AnyMatch provides competitive prediction quality despite its small parameter size: it achieves the second-highest F1 score overall, and outperforms several other approaches that employ models with hundreds of billions of parameters. Furthermore, our approach exhibits major cost benefits: the average prediction quality of AnyMatch is within 4.4% of the state-of-the-art method MatchGPT with the proprietary trillion-parameter model GPT-4, yet AnyMatch requires four orders of magnitude less parameters and incurs a 3,899 times lower inference cost (in dollars per 1,000 tokens).

Title: UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity

• Authors: Yicheng Fu, Raviteja Anantha, Prabal Vashisht, Jianpeng Cheng, Etai Littwin
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04081
• Pdf URL: https://arxiv.org/pdf/2409.04081
• Copy Paste: [[2409.04081]] UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity(https://arxiv.org/abs/2409.04081)
• Keywords: privacy, large language model
• Abstract: Generating user intent from a sequence of user interface (UI) actions is a core challenge in comprehensive UI understanding. Recent advancements in multimodal large language models (MLLMs) have led to substantial progress in this area, but their demands for extensive model parameters, computing power, and high latency makes them impractical for scenarios requiring lightweight, on-device solutions with low latency or heightened privacy. Additionally, the lack of high-quality datasets has hindered the development of such lightweight models. To address these challenges, we propose UI-JEPA, a novel framework that employs masking strategies to learn abstract UI embeddings from unlabeled data through self-supervised learning, combined with an LLM decoder fine-tuned for user intent prediction. We also introduce two new UI-grounded multimodal datasets, "Intent in the Wild" (IIW) and "Intent in the Tame" (IIT), designed for few-shot and zero-shot UI understanding tasks. IIW consists of 1.7K videos across 219 intent categories, while IIT contains 914 videos across 10 categories. We establish the first baselines for these datasets, showing that representations learned using a JEPA-style objective, combined with an LLM decoder, can achieve user intent predictions that match the performance of state-of-the-art large MLLMs, but with significantly reduced annotation and deployment resources. Measured by intent similarity scores, UI-JEPA outperforms GPT-4 Turbo and Claude 3.5 Sonnet by 10.0% and 7.2% respectively, averaged across two datasets. Notably, UI-JEPA accomplishes the performance with a 50.5x reduction in computational cost and a 6.6x improvement in latency in the IIW dataset. These results underscore the effectiveness of UI-JEPA, highlighting its potential for lightweight, high-performance UI understanding.

Title: SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation

• Authors: Yi Tian, Juan Andrade-Cetto
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04082
• Pdf URL: https://arxiv.org/pdf/2409.04082
• Copy Paste: [[2409.04082]] SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation(https://arxiv.org/abs/2409.04082)
• Keywords: robust, transformer
• Abstract: Event cameras generate asynchronous and sparse event streams capturing changes in light intensity. They offer significant advantages over conventional frame-based cameras, such as a higher dynamic range and an extremely faster data rate, making them particularly useful in scenarios involving fast motion or challenging lighting conditions. Spiking neural networks (SNNs) share similar asynchronous and sparse characteristics and are well-suited for processing data from event cameras. Inspired by the potential of transformers and spike-driven transformers (spikeformers) in other computer vision tasks, we propose two solutions for fast and robust optical flow estimation for event cameras: STTFlowNet and SDformerFlow. STTFlowNet adopts a U-shaped artificial neural network (ANN) architecture with spatiotemporal shifted window self-attention (swin) transformer encoders, while SDformerFlow presents its fully spiking counterpart, incorporating swin spikeformer encoders. Furthermore, we present two variants of the spiking version with different neuron models. Our work is the first to make use of spikeformers for dense optical flow estimation. We conduct end-to-end training for all models using supervised learning. Our results yield state-of-the-art performance among SNN-based event optical flow methods on both the DSEC and MVSEC datasets, and show significant reduction in power consumption compared to the equivalent ANNs.

Title: UNIT: Unifying Image and Text Recognition in One Vision Encoder

• Authors: Yi Zhu, Yanpeng Zhou, Chunwei Wang, Yang Cao, Jianhua Han, Lu Hou, Hang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04095
• Pdf URL: https://arxiv.org/pdf/2409.04095
• Copy Paste: [[2409.04095]] UNIT: Unifying Image and Text Recognition in One Vision Encoder(https://arxiv.org/abs/2409.04095)
• Keywords: robust, transformer
• Abstract: Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual recognition. To address this limitation, we propose UNIT, a novel training framework aimed at UNifying Image and Text recognition within a single model. Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities. The training process comprises two stages: intra-scale pretraining and inter-scale finetuning. During intra-scale pretraining, UNIT learns unified representations from multi-scale inputs, where images and documents are at their commonly used resolution, to enable fundamental recognition capability. In the inter-scale finetuning stage, the model introduces scale-exchanged data, featuring images and documents at resolutions different from the most commonly used ones, to enhance its scale robustness. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment. Experiments across multiple benchmarks confirm that our method significantly outperforms existing methods on document-related tasks (e.g., OCR and DocQA) while maintaining the performances on natural images, demonstrating its ability to substantially enhance text recognition without compromising its core image recognition capabilities.

Title: Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

• Authors: Chenglei Si, Diyi Yang, Tatsunori Hashimoto
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04109
• Pdf URL: https://arxiv.org/pdf/2409.04109
• Copy Paste: [[2409.04109]] Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers(https://arxiv.org/abs/2409.04109)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery, with a growing number of works proposing research agents that autonomously generate and validate new ideas. Despite this, no evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas, let alone perform the entire research process. We address this by establishing an experimental design that evaluates research idea generation while controlling for confounders and performs the first head-to-head comparison between expert NLP researchers and an LLM ideation agent. By recruiting over 100 NLP researchers to write novel ideas and blind reviews of both LLM and human ideas, we obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility. Studying our agent baselines closely, we identify open problems in building and evaluating research agents, including failures of LLM self-evaluation and their lack of diversity in generation. Finally, we acknowledge that human judgements of novelty can be difficult, even by experts, and propose an end-to-end study design which recruits researchers to execute these ideas into full projects, enabling us to study whether these novelty and feasibility judgements result in meaningful differences in research outcome.

Title: Active-Passive Federated Learning for Vertically Partitioned Multi-view Data

• Authors: Jiyuan Liu, Xinwang Liu, Siqi Wang, Xingchen Hu, Qing Liao, Xinhang Wan, Yi Zhang, Xin Lv, Kunlun He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04111
• Pdf URL: https://arxiv.org/pdf/2409.04111
• Copy Paste: [[2409.04111]] Active-Passive Federated Learning for Vertically Partitioned Multi-view Data(https://arxiv.org/abs/2409.04111)
• Keywords: federate
• Abstract: Vertical federated learning is a natural and elegant approach to integrate multi-view data vertically partitioned across devices (clients) while preserving their privacies. Apart from the model training, existing methods requires the collaboration of all clients in the model inference. However, the model inference is probably maintained for service in a long time, while the collaboration, especially when the clients belong to different organizations, is unpredictable in real-world scenarios, such as concellation of contract, network unavailablity, etc., resulting in the failure of them. To address this issue, we, at the first attempt, propose a flexible Active-Passive Federated learning (APFed) framework. Specifically, the active client is the initiator of a learning task and responsible to build the complete model, while the passive clients only serve as assistants. Once the model built, the active client can make inference independently. In addition, we instance the APFed framework into two classification methods with employing the reconstruction loss and the contrastive loss on passive clients, respectively. Meanwhile, the two methods are tested in a set of experiments and achieves desired results, validating their effectiveness.

Title: Multi-Programming Language Ensemble for Code Generation in Large Language Model

• Authors: Tengfei Xue, Xuefeng Li, Tahir Azim, Roman Smirnov, Jianhui Yu, Arash Sadrieh, Babak Pahlavan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04114
• Pdf URL: https://arxiv.org/pdf/2409.04114
• Copy Paste: [[2409.04114]] Multi-Programming Language Ensemble for Code Generation in Large Language Model(https://arxiv.org/abs/2409.04114)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have significantly improved code generation, particularly in one-pass code generation. However, most existing approaches focus solely on generating code in a single programming language, overlooking the potential of leveraging the multi-language capabilities of LLMs. LLMs have varying patterns of errors across different languages, suggesting that a more robust approach could be developed by leveraging these multi-language outputs. In this study, we propose Multi-Programming Language Ensemble (MPLE), a novel ensemble-based method that utilizes code generation across multiple programming languages to enhance overall performance. By treating each language-specific code generation process as an individual "weak expert" and effectively integrating their outputs, our method mitigates language-specific errors and biases. This multi-language ensemble strategy leverages the complementary strengths of different programming languages, enabling the model to produce more accurate and robust code. Our approach can be seamlessly integrated with commonly used techniques such as the reflection algorithm and Monte Carlo tree search to improve code generation quality further. Experimental results show that our framework consistently enhances baseline performance by up to 17.92% on existing benchmarks (HumanEval and HumanEval-plus), with a standout result of 96.25% accuracy on the HumanEval benchmark, achieving new state-of-the-art results across various LLM models. The code will be released at this https URL

Title: Smooth-edged Perturbations Improve Perturbation-based Image Explanations

• Authors: Gustav Grund Pihlgren, Kary Främling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04116
• Pdf URL: https://arxiv.org/pdf/2409.04116
• Copy Paste: [[2409.04116]] Smooth-edged Perturbations Improve Perturbation-based Image Explanations(https://arxiv.org/abs/2409.04116)
• Keywords: segmentation
• Abstract: Perturbation-based post-hoc image explanation methods are commonly used to explain image prediction models by perturbing parts of the input to measure how those parts affect the output. Due to the intractability of perturbing each pixel individually, images are typically attributed to larger segments. The Randomized Input Sampling for Explanations (RISE) method solved this issue by using smooth perturbation masks. While this method has proven effective and popular, it has not been investigated which parts of the method are responsible for its success. This work tests many combinations of mask sampling, segmentation techniques, smoothing, and attribution calculation. The results show that the RISE-style pixel attribution is beneficial to all evaluated methods. Furthermore, it is shown that attribution calculation is the least impactful parameter. The implementation of this work is available online: this https URL.

Title: Prompt-based Personality Profiling: Reinforcement Learning for Relevance Filtering

• Authors: Jan Hofmann, Cornelia Sindermann, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04122
• Pdf URL: https://arxiv.org/pdf/2409.04122
• Copy Paste: [[2409.04122]] Prompt-based Personality Profiling: Reinforcement Learning for Relevance Filtering(https://arxiv.org/abs/2409.04122)
• Keywords: transformer, large language model
• Abstract: Author profiling is the task of inferring characteristics about individuals by analyzing content they share. Supervised machine learning still dominates automatic systems that perform this task, despite the popularity of prompting large language models to address natural language understanding tasks. One reason is that the classification instances consist of large amounts of posts, potentially a whole user profile, which may exceed the input length of Transformers. Even if a model can use a large context window, the entirety of posts makes the application of API-accessed black box systems costly and slow, next to issues which come with such "needle-in-the-haystack" tasks. To mitigate this limitation, we propose a new method for author profiling which aims at distinguishing relevant from irrelevant content first, followed by the actual user profiling only with relevant data. To circumvent the need for relevance-annotated data, we optimize this relevance filter via reinforcement learning with a reward function that utilizes the zero-shot capabilities of large language models. We evaluate our method for Big Five personality trait prediction on two Twitter corpora. On publicly available real-world data with a skewed label distribution, our method shows similar efficacy to using all posts in a user profile, but with a substantially shorter context. An evaluation on a version of these data balanced with artificial posts shows that the filtering to relevant posts leads to a significantly improved accuracy of the predictions.

Title: Secure Traffic Sign Recognition: An Attention-Enabled Universal Image Inpainting Mechanism against Light Patch Attacks

• Authors: Hangcheng Cao, Longzhi Yuan, Guowen Xu, Ziyang He, Zhengru Fang, Yuguang Fang
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.04133
• Pdf URL: https://arxiv.org/pdf/2409.04133
• Copy Paste: [[2409.04133]] Secure Traffic Sign Recognition: An Attention-Enabled Universal Image Inpainting Mechanism against Light Patch Attacks(https://arxiv.org/abs/2409.04133)
• Keywords: secure, security, attack, steal
• Abstract: Traffic sign recognition systems play a crucial role in assisting drivers to make informed decisions while driving. However, due to the heavy reliance on deep learning technologies, particularly for future connected and autonomous driving, these systems are susceptible to adversarial attacks that pose significant safety risks to both personal and public transportation. Notably, researchers recently identified a new attack vector to deceive sign recognition systems: projecting well-designed adversarial light patches onto traffic signs. In comparison with traditional adversarial stickers or graffiti, these emerging light patches exhibit heightened aggression due to their ease of implementation and outstanding stealthiness. To effectively counter this security threat, we propose a universal image inpainting mechanism, namely, SafeSign. It relies on attention-enabled multi-view image fusion to repair traffic signs contaminated by adversarial light patches, thereby ensuring the accurate sign recognition. Here, we initially explore the fundamental impact of malicious light patches on the local and global feature spaces of authentic traffic signs. Then, we design a binary mask-based U-Net image generation pipeline outputting diverse contaminated sign patterns, to provide our image inpainting model with needed training data. Following this, we develop an attention mechanism-enabled neural network to jointly utilize the complementary information from multi-view images to repair contaminated signs. Finally, extensive experiments are conducted to evaluate SafeSign's effectiveness in resisting potential light patch-based attacks, bringing an average accuracy improvement of 54.8% in three widely-used sign recognition models

Title: Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformers

• Authors: Gorka Abad, Stjepan Picek, Lorenzo Cavallaro, Aitor Urbieta
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04142
• Pdf URL: https://arxiv.org/pdf/2409.04142
• Copy Paste: [[2409.04142]] Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformers(https://arxiv.org/abs/2409.04142)
• Keywords: attack, robust, transformer
• Abstract: Due to the high cost of training, large model (LM) practitioners commonly use pretrained models downloaded from untrusted sources, which could lead to owning compromised models. In-context learning is the ability of LMs to perform multiple tasks depending on the prompt or context. This can enable new attacks, such as backdoor attacks with dynamic behavior depending on how models are prompted. In this paper, we leverage the ability of vision transformers (ViTs) to perform different tasks depending on the prompts. Then, through data poisoning, we investigate two new threats: i) task-specific backdoors where the attacker chooses a target task to attack, and only the selected task is compromised at test time under the presence of the trigger. At the same time, any other task is not affected, even if prompted with the trigger. We succeeded in attacking every tested model, achieving up to 89.90\% degradation on the target task. ii) We generalize the attack, allowing the backdoor to affect \emph{any} task, even tasks unseen during the training phase. Our attack was successful on every tested model, achieving a maximum of $13\times$ degradation. Finally, we investigate the robustness of prompts and fine-tuning as techniques for removing the backdoors from the model. We found that these methods fall short and, in the best case, reduce the degradation from 89.90\% to 73.46\%.

Title: Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation

• Authors: Luis Mayer, Christian Heumann, Matthias Aßenmacher
• Subjects: cs.CL, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.04164
• Pdf URL: https://arxiv.org/pdf/2409.04164
• Copy Paste: [[2409.04164]] Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation(https://arxiv.org/abs/2409.04164)
• Keywords: large language model
• Abstract: In recent years, large language models (LLMs) have emerged as powerful tools with potential applications in various fields, including software engineering. Within the scope of this research, we evaluate five different state-of-the-art LLMs - Bard, BingChat, ChatGPT, Llama2, and Code Llama - concerning their capabilities for text-to-code generation. In an empirical study, we feed prompts with textual descriptions of coding problems sourced from the programming website LeetCode to the models with the task of creating solutions in Python. Subsequently, the quality of the generated outputs is assessed using the testing functionalities of LeetCode. The results indicate large differences in performance between the investigated models. ChatGPT can handle these typical programming challenges by far the most effectively, surpassing even code-specialized models like Code Llama. To gain further insights, we measure the runtime as well as the memory usage of the generated outputs and compared them to the other code submissions on Leetcode. A detailed error analysis, encompassing a comparison of the differences concerning correct indentation and form of the generated code as well as an assignment of the incorrectly solved tasks to certain error categories allows us to obtain a more nuanced picture of the results and potential for improvement. The results also show a clear pattern of increasingly incorrect produced code when the models are facing a lot of context in the form of longer prompts.

Title: Do Android App Developers Accurately Report Collection of Privacy-Related Data?

• Authors: Mugdha Khedkar, Ambuj Kumar Mondal, Eric Bodden
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04167
• Pdf URL: https://arxiv.org/pdf/2409.04167
• Copy Paste: [[2409.04167]] Do Android App Developers Accurately Report Collection of Privacy-Related Data?(https://arxiv.org/abs/2409.04167)
• Keywords: privacy, protect
• Abstract: Many Android applications collect data from users. The European Union's General Data Protection Regulation (GDPR) requires vendors to faithfully disclose which data their apps collect. This task is complicated because many apps use third-party code for which the same information is not readily available. Hence we ask: how accurately do current Android apps fulfill these requirements? In this work, we first expose a multi-layered definition of privacy-related data to correctly report data collection in Android apps. We further create a dataset of privacy-sensitive data classes that may be used as input by an Android app. This dataset takes into account data collected both through the user interface and system APIs. We manually examine the data safety sections of 70 Android apps to observe how data collection is reported, identifying instances of over- and under-reporting. Additionally, we develop a prototype to statically extract and label privacy-related data collected via app source code, user interfaces, and permissions. Comparing the prototype's results with the data safety sections of 20 apps reveals reporting discrepancies. Using the results from two Messaging and Social Media apps (Signal and Instagram), we discuss how app developers under-report and over-report data collection, respectively, and identify inaccurately reported data categories. Our results show that app developers struggle to accurately report data collection, either due to Google's abstract definition of collected data or insufficient existing tool support.

Title: From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks

• Authors: Andreas Stephan, Dawei Zhu, Matthias Aßenmacher, Xiaoyu Shen, Benjamin Roth
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04168
• Pdf URL: https://arxiv.org/pdf/2409.04168
• Copy Paste: [[2409.04168]] From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks(https://arxiv.org/abs/2409.04168)
• Keywords: large language model
• Abstract: To reduce the need for human annotations, large language models (LLMs) have been proposed as judges of the quality of other candidate models. LLM judges are typically evaluated by measuring the correlation with human judgments on generation tasks such as summarization or machine translation. In contrast, we study LLM judges on mathematical reasoning tasks. These tasks require multi-step reasoning, and the correctness of their solutions is verifiable, enabling a more objective evaluation. We perform a detailed performance analysis and find that the used judges are mostly unable to improve task performance but are able to pick the better model. Our analysis uncovers a strong correlation between judgment performance and the candidate model task performance. We observe that judges tend to choose the model of higher quality even if its answer is incorrect. Further, we show that it is possible to use statistics, such as the task performances of the individual models, to predict judgment performance. In an ablation, we either swap or mask the candidate answers and observe that judges often keep the original judgment, providing evidence that judges incorporate writing style in their judgments. In summary, we find that regularities in the judgments are quantifiable using statistical measures and provide various angles on exploiting them.

Title: Combining LLMs and Knowledge Graphs to Reduce Hallucinations in Question Answering

• Authors: Larissa Pusch, Tim O. F. Conrad
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04181
• Pdf URL: https://arxiv.org/pdf/2409.04181
• Copy Paste: [[2409.04181]] Combining LLMs and Knowledge Graphs to Reduce Hallucinations in Question Answering(https://arxiv.org/abs/2409.04181)
• Keywords: large language model
• Abstract: Advancements in natural language processing have revolutionized the way we can interact with digital information systems, such as databases, making them more accessible. However, challenges persist, especially when accuracy is critical, as in the biomedical domain. A key issue is the hallucination problem, where models generate information unsupported by the underlying data, potentially leading to dangerous misinformation. This paper presents a novel approach designed to bridge this gap by combining Large Language Models (LLM) and Knowledge Graphs (KG) to improve the accuracy and reliability of question-answering systems, on the example of a biomedical KG. Built on the LangChain framework, our method incorporates a query checker that ensures the syntactical and semantic validity of LLM-generated queries, which are then used to extract information from a Knowledge Graph, substantially reducing errors like hallucinations. We evaluated the overall performance using a new benchmark dataset of 50 biomedical questions, testing several LLMs, including GPT-4 Turbo and llama3:70b. Our results indicate that while GPT-4 Turbo outperforms other models in generating accurate queries, open-source models like llama3:70b show promise with appropriate prompt engineering. To make this approach accessible, a user-friendly web-based interface has been developed, allowing users to input natural language queries, view generated and corrected Cypher queries, and verify the resulting paths for accuracy. Overall, this hybrid approach effectively addresses common issues such as data gaps and hallucinations, offering a reliable and intuitive solution for question answering systems. The source code for generating the results of this paper and for the user-interface can be found in our Git repository: this https URL

Title: GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding

• Authors: Ziyin Zhang, Hang Yu, Shijie Li, Peng Di, Jianguo Li, Rui Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04183
• Pdf URL: https://arxiv.org/pdf/2409.04183
• Copy Paste: [[2409.04183]] GALLa: Graph Aligned Large Language Models for Improved Source Code Understanding(https://arxiv.org/abs/2409.04183)
• Keywords: transformer, large language model
• Abstract: Programming languages possess rich semantic information such as data flow that is represented by graphs and not available from the surface form of source code. Recent code language models have scaled to billions of parameters, but model source code solely as text tokens while ignoring any other structural information. Conversely, models that do encode structural information of code make modifications to the Transformer architecture, limiting their scale and compatibility with pretrained LLMs. In this work, we take the best of both worlds with GALLa - Graph Aligned Large Language Model. GALLa utilizes graph neural networks and cross-modal alignment technologies to inject the structural information of code into LLMs as an auxiliary task during finetuning. This framework is both model-agnostic and task-agnostic, as it can be applied to any code LLM for any code downstream task, and requires the structural graph data only at training time from a corpus unrelated to the finetuning data, while incurring no cost at inference time over the baseline LLM. Experiments on five code tasks with four different baseline LLMs ranging in size from 350M to 8B validate the effectiveness of GALLa, demonstrating consistent improvement over the baseline, even for powerful models such as LLaMA3.

Title: Residual Stream Analysis with Multi-Layer SAEs

• Authors: Tim Lawson, Lucy Farnik, Conor Houghton, Laurence Aitchison
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04185
• Pdf URL: https://arxiv.org/pdf/2409.04185
• Copy Paste: [[2409.04185]] Residual Stream Analysis with Multi-Layer SAEs(https://arxiv.org/abs/2409.04185)
• Keywords: transformer
• Abstract: Sparse autoencoders (SAEs) are a promising approach to interpreting the internal representations of transformer language models. However, standard SAEs are trained separately on each transformer layer, making it difficult to use them to study how information flows across layers. To solve this problem, we introduce the multi-layer SAE (MLSAE): a single SAE trained on the residual stream activation vectors from every transformer layer simultaneously. The residual stream is usually understood as preserving information across layers, so we expected to, and did, find individual SAE features that are active at multiple layers. Interestingly, while a single SAE feature is active at different layers for different prompts, for a single prompt, we find that a single feature is far more likely to be active at a single layer. For larger underlying models, we find that the cosine similarities between adjacent layers in the residual stream are higher, so we expect more features to be active at multiple layers. These results show that MLSAEs are a promising method to study information flow in transformers. We release our code to train and analyze MLSAEs at this https URL.

Title: LITE: A Paradigm Shift in Multi-Object Tracking with Efficient ReID Feature Integration

• Authors: Jumabek Alikhanov, Dilshod Obidov, Hakil Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04187
• Pdf URL: https://arxiv.org/pdf/2409.04187
• Copy Paste: [[2409.04187]] LITE: A Paradigm Shift in Multi-Object Tracking with Efficient ReID Feature Integration(https://arxiv.org/abs/2409.04187)
• Keywords: extraction, fair
• Abstract: The Lightweight Integrated Tracking-Feature Extraction (LITE) paradigm is introduced as a novel multi-object tracking (MOT) approach. It enhances ReID-based trackers by eliminating inference, pre-processing, post-processing, and ReID model training costs. LITE uses real-time appearance features without compromising speed. By integrating appearance feature extraction directly into the tracking pipeline using standard CNN-based detectors such as YOLOv8m, LITE demonstrates significant performance improvements. The simplest implementation of LITE on top of classic DeepSORT achieves a HOTA score of 43.03% at 28.3 FPS on the MOT17 benchmark, making it twice as fast as DeepSORT on MOT17 and four times faster on the more crowded MOT20 dataset, while maintaining similar accuracy. Additionally, a new evaluation framework for tracking-by-detection approaches reveals that conventional trackers like DeepSORT remain competitive with modern state-of-the-art trackers when evaluated under fair conditions. The code will be available post-publication at this https URL.

Title: Reassessing the Validity of Spurious Correlations Benchmarks

• Authors: Samuel J. Bell, Diane Bouchacourt, Levent Sagun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04188
• Pdf URL: https://arxiv.org/pdf/2409.04188
• Copy Paste: [[2409.04188]] Reassessing the Validity of Spurious Correlations Benchmarks(https://arxiv.org/abs/2409.04188)
• Keywords: robust
• Abstract: Neural networks can fail when the data contains spurious correlations. To understand this phenomenon, researchers have proposed numerous spurious correlations benchmarks upon which to evaluate mitigation methods. However, we observe that these benchmarks exhibit substantial disagreement, with the best methods on one benchmark performing poorly on another. We explore this disagreement, and examine benchmark validity by defining three desiderata that a benchmark should satisfy in order to meaningfully evaluate methods. Our results have implications for both benchmarks and mitigations: we find that certain benchmarks are not meaningful measures of method performance, and that several methods are not sufficiently robust for widespread use. We present a simple recipe for practitioners to choose methods using the most similar benchmark to their given problem.

Title: Mind The Gap: Can Air-Gaps Keep Your Private Data Secure?

• Authors: Mordechai Guri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04190
• Pdf URL: https://arxiv.org/pdf/2409.04190
• Copy Paste: [[2409.04190]] Mind The Gap: Can Air-Gaps Keep Your Private Data Secure?(https://arxiv.org/abs/2409.04190)
• Keywords: secure, security, privacy, protect, attack, biometric
• Abstract: Personal data has become one of the most valuable assets and lucrative targets for attackers in the modern digital world. This includes personal identification information (PII), medical records, legal information, biometric data, and private communications. To protect it from hackers, 'air-gap' measures might be employed. This protective strategy keeps sensitive data in networks entirely isolated (physically and logically) from the Internet. Creating a physical 'air gap' between internal networks and the outside world safeguards sensitive data from theft and online threats. Air-gap networks are relevant today to governmental organizations, healthcare industries, finance sectors, intellectual property and legal firms, and others. In this paper, we dive deep into air-gap security in light of modern cyberattacks and data privacy. Despite this level of protection, publicized incidents from the last decade show that even air-gap networks are not immune to breaches. Motivated and capable adversaries can use sophisticated attack vectors to penetrate the air-gapped networks, leaking sensitive data outward. We focus on different aspects of air gap security. First, we overview cyber incidents that target air-gap networks, including infamous ones such Agent.btz. Second, we introduce the adversarial attack model and different attack vectors attackers may use to compromise air-gap networks. Third, we present the techniques attackers can apply to leak data out of air-gap networks and introduce more innovative ones based on our recent research. Finally, we propose the necessary countermeasures to protect the data, both defensive and preventive.

Title: GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers

• Authors: Lorenza Prospero, Abdullah Hamdi, Joao F. Henriques, Christian Rupprecht
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04196
• Pdf URL: https://arxiv.org/pdf/2409.04196
• Copy Paste: [[2409.04196]] GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers(https://arxiv.org/abs/2409.04196)
• Keywords: diffusion, transformer
• Abstract: Reconstructing realistic 3D human models from monocular images has significant applications in creative industries, human-computer interfaces, and healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene representation composed of a mixture of Gaussians. Predicting such mixtures for a human from a single input image is challenging, as it is a non-uniform density (with a many-to-one relationship with input pixels) with strict physical constraints. At the same time, it needs to be flexible to accommodate a variety of clothes and poses. Our key observation is that the vertices of standardized human meshes (such as SMPL) can provide an adequate density and approximate initial position for Gaussians. We can then train a transformer model to jointly predict comparatively small adjustments to these positions, as well as the other Gaussians' attributes and the SMPL parameters. We show empirically that this combination (using only multi-view supervision) can achieve fast inference of 3D human models from a single image without test-time optimization, expensive diffusion models, or 3D points supervision. We also show that it can improve 3D pose estimation by better fitting human models that account for clothes and other variations. The code is available on the project website this https URL .

Title: Introducing Gating and Context into Temporal Action Detection

• Authors: Aglind Reka, Diana Laura Borza, Dominick Reilly, Michal Balazia, Francois Bremond
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04205
• Pdf URL: https://arxiv.org/pdf/2409.04205
• Copy Paste: [[2409.04205]] Introducing Gating and Context into Temporal Action Detection(https://arxiv.org/abs/2409.04205)
• Keywords: extraction, transformer
• Abstract: Temporal Action Detection (TAD), the task of localizing and classifying actions in untrimmed video, remains challenging due to action overlaps and variable action durations. Recent findings suggest that TAD performance is dependent on the structural design of transformers rather than on the self-attention mechanism. Building on this insight, we propose a refined feature extraction process through lightweight, yet effective operations. First, we employ a local branch that employs parallel convolutions with varying window sizes to capture both fine-grained and coarse-grained temporal features. This branch incorporates a gating mechanism to select the most relevant features. Second, we introduce a context branch that uses boundary frames as key-value pairs to analyze their relationship with the central frame through cross-attention. The proposed method captures temporal dependencies and improves contextual understanding. Evaluations of the gating mechanism and context branch on challenging datasets (THUMOS14 and EPIC-KITCHEN 100) show a consistent improvement over the baseline and existing methods.

Title: Learning to Learn Transferable Generative Attack for Person Re-Identification

• Authors: Yuan Bian, Min Liu, Xueping Wang, Yunfeng Ma, Yaonan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04208
• Pdf URL: https://arxiv.org/pdf/2409.04208
• Copy Paste: [[2409.04208]] Learning to Learn Transferable Generative Attack for Person Re-Identification(https://arxiv.org/abs/2409.04208)
• Keywords: attack, robust, generative
• Abstract: Deep learning-based person re-identification (re-id) models are widely employed in surveillance systems and inevitably inherit the vulnerability of deep networks to adversarial attacks. Existing attacks merely consider cross-dataset and cross-model transferability, ignoring the cross-test capability to perturb models trained in different domains. To powerfully examine the robustness of real-world re-id models, the Meta Transferable Generative Attack (MTGA) method is proposed, which adopts meta-learning optimization to promote the generative attacker producing highly transferable adversarial examples by learning comprehensively simulated transfer-based cross-model\&dataset\&test black-box meta attack tasks. Specifically, cross-model\&dataset black-box attack tasks are first mimicked by selecting different re-id models and datasets for meta-train and meta-test attack processes. As different models may focus on different feature regions, the Perturbation Random Erasing module is further devised to prevent the attacker from learning to only corrupt model-specific features. To boost the attacker learning to possess cross-test transferability, the Normalization Mix strategy is introduced to imitate diverse feature embedding spaces by mixing multi-domain statistics of target models. Extensive experiments show the superiority of MTGA, especially in cross-model\&dataset and cross-model\&dataset\&test attacks, our MTGA outperforms the SOTA methods by 21.5\% and 11.3\% on mean mAP drop rate, respectively. The code of MTGA will be released after the paper is accepted.

Title: UniDet3D: Multi-dataset Indoor 3D Object Detection

• Authors: Maksim Kolodiazhnyi, Anna Vorontsova, Matvey Skripkin, Danila Rukhovich, Anton Konushin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04234
• Pdf URL: https://arxiv.org/pdf/2409.04234
• Copy Paste: [[2409.04234]] UniDet3D: Multi-dataset Indoor 3D Object Detection(https://arxiv.org/abs/2409.04234)
• Keywords: transformer
• Abstract: Growing customer demand for smart solutions in robotics and augmented reality has attracted considerable attention to 3D object detection from point clouds. Yet, existing indoor datasets taken individually are too small and insufficiently diverse to train a powerful and general 3D object detection model. In the meantime, more general approaches utilizing foundation models are still inferior in quality to those based on supervised training for a specific task. In this work, we propose \ours{}, a simple yet effective 3D object detection model, which is trained on a mixture of indoor datasets and is capable of working in various indoor environments. By unifying different label spaces, \ours{} enables learning a strong representation across multiple datasets through a supervised joint training scheme. The proposed network architecture is built upon a vanilla transformer encoder, making it easy to run, customize and extend the prediction pipeline for practical use. Extensive experiments demonstrate that \ours{} obtains significant gains over existing 3D object detection methods in 6 indoor benchmarks: ScanNet (+1.1 mAP50), ARKitScenes (+19.4 mAP25), S3DIS (+9.1 mAP50), MultiScan (+9.3 mAP50), 3RScan (+3.2 mAP50), and ScanNet++ (+2.7 mAP50). Code is available at this https URL .

Title: AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspective

• Authors: Guoqiang Zhang, Richard Heusdens
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04275
• Pdf URL: https://arxiv.org/pdf/2409.04275
• Copy Paste: [[2409.04275]] AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspective(https://arxiv.org/abs/2409.04275)
• Keywords: transformer
• Abstract: In this paper, we extend the standard Attention in transformer by exploiting the consensus discrepancy from a distributed optimization perspective, referred to as AttentionX. It is noted that %the popular distributed optimization algorithm \cite{Boyd11ADMM} and the primal-dual method of multipliers (PDMM) \cite{Zhang16PDMM} is designed to iteratively solve a broad class of distributed optimization problems over a pear-to-pear (P2P) network, where neighbouring nodes gradually reach consensus as specified by predefined linear edge-constraints in the optimization process. In particular, at each iteration of PDMM, each node in a network first performs information-gathering from neighbours and then performs local information-fusion. From a high-level point of view, the $KQ$-softmax-based weighted summation of $V$-representations in Attention corresponds information-gathering from neighbours while the feature-processing via the feed-forward network (FFN) in transformer corresponds to local information fusion. PDMM exploits the Lagrangian multipliers to capture the historical consensus discrepancy in the form of residual errors of the linear edge-constraints, which plays a crucial role for the algorithm to converge. Inspired by PDMM, we propose AttentionX to incorporate the consensus discrepancy in the output update-expression of the standard Attention. The consensus discrepancy in AttentionX refers to the difference between the weighted summation of $V$-representations and scaled $V$-representions themselves. Experiments on ViT and nanoGPT show promising performance.

Title: CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysis

• Authors: William Knottenbelt, Zeyu Gao, Rebecca Wray, Woody Zhidong Zhang, Jiashuai Liu, Mireia Crispin-Ortuzar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04290
• Pdf URL: https://arxiv.org/pdf/2409.04290
• Copy Paste: [[2409.04290]] CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysis(https://arxiv.org/abs/2409.04290)
• Keywords: interpretability
• Abstract: Survival analysis is a branch of statistics used for modeling the time until a specific event occurs and is widely used in medicine, engineering, finance, and many other fields. When choosing survival models, there is typically a trade-off between performance and interpretability, where the highest performance is achieved by black-box models based on deep learning. This is a major problem in fields such as medicine where practitioners are reluctant to blindly trust black-box models to make important patient decisions. Kolmogorov-Arnold Networks (KANs) were recently proposed as an interpretable and accurate alternative to multi-layer perceptrons (MLPs). We introduce CoxKAN, a Cox proportional hazards Kolmogorov-Arnold Network for interpretable, high-performance survival analysis. We evaluate the proposed CoxKAN on 4 synthetic datasets and 9 real medical datasets. The synthetic experiments demonstrate that CoxKAN accurately recovers interpretable symbolic formulae for the hazard function, and effectively performs automatic feature selection. Evaluation on the 9 real datasets show that CoxKAN consistently outperforms the Cox proportional hazards model and achieves performance that is superior or comparable to that of tuned MLPs. Furthermore, we find that CoxKAN identifies complex interactions between predictor variables that would be extremely difficult to recognise using existing survival methods, and automatically finds symbolic formulae which uncover the precise effect of important biomarkers on patient risk.

Title: FS-MedSAM2: Exploring the Potential of SAM2 for Few-Shot Medical Image Segmentation without Fine-tuning

• Authors: Yunhao Bai, Qinji Yu, Boxiang Yun, Dakai Jin, Yingda Xia, Yan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04298
• Pdf URL: https://arxiv.org/pdf/2409.04298
• Copy Paste: [[2409.04298]] FS-MedSAM2: Exploring the Potential of SAM2 for Few-Shot Medical Image Segmentation without Fine-tuning(https://arxiv.org/abs/2409.04298)
• Keywords: segmentation
• Abstract: The Segment Anything Model 2 (SAM2) has recently demonstrated exceptional performance in zero-shot prompt segmentation for natural images and videos. However, it faces significant challenges when applied to medical images. Since its release, many attempts have been made to adapt SAM2's segmentation capabilities to the medical imaging domain. These efforts typically involve using a substantial amount of labeled data to fine-tune the model's weights. In this paper, we explore SAM2 from a different perspective via making the full use of its trained memory attention module and its ability of processing mask prompts. We introduce FS-MedSAM2, a simple yet effective framework that enables SAM2 to achieve superior medical image segmentation in a few-shot setting, without the need for fine-tuning. Our framework outperforms the current state-of-the-arts on two publicly available medical image datasets. The code is available at this https URL.

Title: Advancing SEM Based Nano-Scale Defect Analysis in Semiconductor Manufacturing for Advanced IC Nodes

• Authors: Bappaditya Dey, Matthias Monden, Victor Blanco, Sandip Halder, Stefan De Gendt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04310
• Pdf URL: https://arxiv.org/pdf/2409.04310
• Copy Paste: [[2409.04310]] Advancing SEM Based Nano-Scale Defect Analysis in Semiconductor Manufacturing for Advanced IC Nodes(https://arxiv.org/abs/2409.04310)
• Keywords: segmentation
• Abstract: In this research, we introduce a unified end-to-end Automated Defect Classification-Detection-Segmentation (ADCDS) framework for classifying, detecting, and segmenting multiple instances of semiconductor defects for advanced nodes. This framework consists of two modules: (a) a defect detection module, followed by (b) a defect segmentation module. The defect detection module employs Deformable DETR to aid in the classification and detection of nano-scale defects, while the segmentation module utilizes BoxSnake. BoxSnake facilitates box-supervised instance segmentation of nano-scale defects, supported by the former module. This simplifies the process by eliminating the laborious requirement for ground-truth pixel-wise mask annotation by human experts, which is typically associated with training conventional segmentation models. We have evaluated the performance of our ADCDS framework using two distinct process datasets from real wafers, as ADI and AEI, specifically focusing on Line-space patterns. We have demonstrated the applicability and significance of our proposed methodology, particularly in the nano-scale segmentation and generation of binary defect masks, using the challenging ADI SEM dataset where ground-truth pixelwise segmentation annotations were unavailable. Furthermore, we have presented a comparative analysis of our proposed framework against previous approaches to demonstrate its effectiveness. Our proposed framework achieved an overall mAP@IoU0.5 of 72.19 for detection and 78.86 for segmentation on the ADI dataset. Similarly, for the AEI dataset, these metrics were 90.38 for detection and 95.48 for segmentation. Thus, our proposed framework effectively fulfils the requirements of advanced defect analysis while addressing significant constraints.

Title: Learning vs Retrieval: The Role of In-Context Examples in Regression with LLMs

• Authors: Aliakbar Nafar, Kristen Brent Venable, Parisa Kordjamshidi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04318
• Pdf URL: https://arxiv.org/pdf/2409.04318
• Copy Paste: [[2409.04318]] Learning vs Retrieval: The Role of In-Context Examples in Regression with LLMs(https://arxiv.org/abs/2409.04318)
• Keywords: robust, generative, large language model
• Abstract: Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can perform regression on real-world datasets and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.

Title: How to Identify Good Superpixels for Deforestation Detection on Tropical Rainforests

• Authors: Isabela Borlido, Eduardo Bouhid, Victor Sundermann, Hugo Resende, Alvaro Luiz Fazenda, Fabio Faria, Silvio Jamil F. Guimarães
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04330
• Pdf URL: https://arxiv.org/pdf/2409.04330
• Copy Paste: [[2409.04330]] How to Identify Good Superpixels for Deforestation Detection on Tropical Rainforests(https://arxiv.org/abs/2409.04330)
• Keywords: segmentation
• Abstract: The conservation of tropical forests is a topic of significant social and ecological relevance due to their crucial role in the global ecosystem. Unfortunately, deforestation and degradation impact millions of hectares annually, requiring government or private initiatives for effective forest monitoring. However, identifying deforested regions in satellite images is challenging due to data imbalance, image resolution, low-contrast regions, and occlusion. Superpixel segmentation can overcome these drawbacks, reducing workload and preserving important image boundaries. However, most works for remote sensing images do not exploit recent superpixel methods. In this work, we evaluate 16 superpixel methods in satellite images to support a deforestation detection system in tropical forests. We also assess the performance of superpixel methods for the target task, establishing a relationship with segmentation methodological evaluation. According to our results, ERS, GMMSP, and DISF perform best on UE, BR, and SIRS, respectively, whereas ERS has the best trade-off with CO and Reg. In classification, SH, DISF, and ISF perform best on RGB, UMDA, and PCA compositions, respectively. According to our experiments, superpixel methods with better trade-offs between delineation, homogeneity, compactness, and regularity are more suitable for identifying good superpixels for deforestation detection tasks.

Title: AGR: Age Group fairness Reward for Bias Mitigation in LLMs

• Authors: Shuirong Cao, Ruoxi Cheng, Zhiqiang Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04340
• Pdf URL: https://arxiv.org/pdf/2409.04340
• Copy Paste: [[2409.04340]] AGR: Age Group fairness Reward for Bias Mitigation in LLMs(https://arxiv.org/abs/2409.04340)
• Keywords: fair
• Abstract: LLMs can exhibit age biases, resulting in unequal treatment of individuals across age groups. While much research has addressed racial and gender biases, age bias remains little explored. The scarcity of instruction-tuning and preference datasets for age bias hampers its detection and measurement, and existing fine-tuning methods seldom address age-related fairness. In this paper, we construct age bias preference datasets and instruction-tuning datasets for RLHF. We introduce ARG, an age fairness reward to reduce differences in the response quality of LLMs across different age groups. Extensive experiments demonstrate that this reward significantly improves response accuracy and reduces performance disparities across age groups. Our source code and datasets are available at the anonymous \href{https://anonymous.4open.science/r/FairRLHF-D445/readme.md}{link}.

Title: Towards Fine-Grained Webpage Fingerprinting at Scale

• Authors: Xiyuan Zhao, Xinhao Deng, Qi Li, Yunpeng Liu, Zhuotao Liu, Kun Sun, Ke Xu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04341
• Pdf URL: https://arxiv.org/pdf/2409.04341
• Copy Paste: [[2409.04341]] Towards Fine-Grained Webpage Fingerprinting at Scale(https://arxiv.org/abs/2409.04341)
• Keywords: attack
• Abstract: Website Fingerprinting (WF) attacks can effectively identify the websites visited by Tor clients via analyzing encrypted traffic patterns. Existing attacks focus on identifying different websites, but their accuracy dramatically decreases when applied to identify fine-grained webpages, especially when distinguishing among different subpages of the same website. WebPage Fingerprinting (WPF) attacks face the challenges of highly similar traffic patterns and a much larger scale of webpages. Furthermore, clients often visit multiple webpages concurrently, increasing the difficulty of extracting the traffic patterns of each webpage from the obfuscated traffic. In this paper, we propose Oscar, a WPF attack based on multi-label metric learning that identifies different webpages from obfuscated traffic by transforming the feature space. Oscar can extract the subtle differences among various webpages, even those with similar traffic patterns. In particular, Oscar combines proxy-based and sample-based metric learning losses to extract webpage features from obfuscated traffic and identify multiple webpages. We prototype Oscar and evaluate its performance using traffic collected from 1,000 monitored webpages and over 9,000 unmonitored webpages in the real world. Oscar demonstrates an 88.6% improvement in the multi-label metric Recall@5 compared to the state-of-the-art attacks.

Title: Serp-Mamba: Advancing High-Resolution Retinal Vessel Segmentation with Selective State-Space Model

• Authors: Hongqiu Wang, Yixian Chen, Wu Chen, Huihui Xu, Haoyu Zhao, Bin Sheng, Huazhu Fu, Guang Yang, Lei Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04356
• Pdf URL: https://arxiv.org/pdf/2409.04356
• Copy Paste: [[2409.04356]] Serp-Mamba: Advancing High-Resolution Retinal Vessel Segmentation with Selective State-Space Model(https://arxiv.org/abs/2409.04356)
• Keywords: segmentation
• Abstract: Ultra-Wide-Field Scanning Laser Ophthalmoscopy (UWF-SLO) images capture high-resolution views of the retina with typically 200 spanning degrees. Accurate segmentation of vessels in UWF-SLO images is essential for detecting and diagnosing fundus disease. Recent studies have revealed that the selective State Space Model (SSM) in Mamba performs well in modeling long-range dependencies, which is crucial for capturing the continuity of elongated vessel structures. Inspired by this, we propose the first Serpentine Mamba (Serp-Mamba) network to address this challenging task. Specifically, we recognize the intricate, varied, and delicate nature of the tubular structure of vessels. Furthermore, the high-resolution of UWF-SLO images exacerbates the imbalance between the vessel and background categories. Based on the above observations, we first devise a Serpentine Interwoven Adaptive (SIA) scan mechanism, which scans UWF-SLO images along curved vessel structures in a snake-like crawling manner. This approach, consistent with vascular texture transformations, ensures the effective and continuous capture of curved vascular structure features. Second, we propose an Ambiguity-Driven Dual Recalibration (ADDR) module to address the category imbalance problem intensified by high-resolution images. Our ADDR module delineates pixels by two learnable thresholds and refines ambiguous pixels through a dual-driven strategy, thereby accurately distinguishing vessels and background regions. Experiment results on three datasets demonstrate the superior performance of our Serp-Mamba on high-resolution vessel segmentation. We also conduct a series of ablation studies to verify the impact of our designs. Our code shall be released upon publication of this work.

Title: Connectivity-Inspired Network for Context-Aware Recognition

• Authors: Gianluca Carloni, Sara Colantonio
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.04360
• Pdf URL: https://arxiv.org/pdf/2409.04360
• Copy Paste: [[2409.04360]] Connectivity-Inspired Network for Context-Aware Recognition(https://arxiv.org/abs/2409.04360)
• Keywords: robust
• Abstract: The aim of this paper is threefold. We inform the AI practitioner about the human visual system with an extensive literature review; we propose a novel biologically motivated neural network for image classification; and, finally, we present a new plug-and-play module to model context awareness. We focus on the effect of incorporating circuit motifs found in biological brains to address visual recognition. Our convolutional architecture is inspired by the connectivity of human cortical and subcortical streams, and we implement bottom-up and top-down modulations that mimic the extensive afferent and efferent connections between visual and cognitive areas. Our Contextual Attention Block is simple and effective and can be integrated with any feed-forward neural network. It infers weights that multiply the feature maps according to their causal influence on the scene, modeling the co-occurrence of different objects in the image. We place our module at different bottlenecks to infuse a hierarchical context awareness into the model. We validated our proposals through image classification experiments on benchmark data and found a consistent improvement in performance and the robustness of the produced explanations via class activation. Our code is available at this https URL.

Title: Deanonymizing Ethereum Validators: The P2P Network Has a Privacy Issue

• Authors: Lioba Heimbach, Yann Vonlanthen, Juan Villacis, Lucianna Kiffer, Roger Wattenhofer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04366
• Pdf URL: https://arxiv.org/pdf/2409.04366
• Copy Paste: [[2409.04366]] Deanonymizing Ethereum Validators: The P2P Network Has a Privacy Issue(https://arxiv.org/abs/2409.04366)
• Keywords: security, privacy, protect
• Abstract: Many blockchain networks aim to preserve the anonymity of validators in the peer-to-peer (P2P) network, ensuring that no adversary can link a validator's identifier to the IP address of a peer due to associated privacy and security concerns. This work demonstrates that the Ethereum P2P network does not offer this anonymity. We present a methodology that enables any node in the network to identify validators hosted on connected peers and empirically verify the feasibility of our proposed method. Using data collected from four nodes over three days, we locate more than 15% of Ethereum validators in the P2P network. The insights gained from our deanonymization technique provide valuable information on the distribution of validators across peers, their geographic locations, and hosting organizations. We further discuss the implications and risks associated with the lack of anonymity in the P2P network and propose methods to help validators protect their privacy. The Ethereum Foundation has awarded us a bug bounty, acknowledging the impact of our results.

Title: Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challenges

• Authors: Parameswaran Kamalaruban, Yulu Pi, Stuart Burrell, Eleanor Drage, Piotr Skalski, Jason Wong, David Sutton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04373
• Pdf URL: https://arxiv.org/pdf/2409.04373
• Copy Paste: [[2409.04373]] Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challenges(https://arxiv.org/abs/2409.04373)
• Keywords: protect, fair
• Abstract: Ensuring fairness in transaction fraud detection models is vital due to the potential harms and legal implications of biased decision-making. Despite extensive research on algorithmic fairness, there is a notable gap in the study of bias in fraud detection models, mainly due to the field's unique challenges. These challenges include the need for fairness metrics that account for fraud data's imbalanced nature and the tradeoff between fraud protection and service quality. To address this gap, we present a comprehensive fairness evaluation of transaction fraud models using public synthetic datasets, marking the first algorithmic bias audit in this domain. Our findings reveal three critical insights: (1) Certain fairness metrics expose significant bias only after normalization, highlighting the impact of class imbalance. (2) Bias is significant in both service quality-related parity metrics and fraud protection-related parity metrics. (3) The fairness through unawareness approach, which involved removing sensitive attributes such as gender, does not improve bias mitigation within these datasets, likely due to the presence of correlated proxies. We also discuss socio-technical fairness-related challenges in transaction fraud models. These insights underscore the need for a nuanced approach to fairness in fraud detection, balancing protection and service quality, and moving beyond simple bias mitigation strategies. Future work must focus on refining fairness metrics and developing methods tailored to the unique complexities of the transaction fraud domain.

Title: Empirical Bayesian image restoration by Langevin sampling with a denoising diffusion implicit prior

• Authors: Charlesquin Kemajou Mbakam, Jean-Francois Giovannelli, Marcelo Pereyra
• Subjects: cs.CV, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.04384
• Pdf URL: https://arxiv.org/pdf/2409.04384
• Copy Paste: [[2409.04384]] Empirical Bayesian image restoration by Langevin sampling with a denoising diffusion implicit prior(https://arxiv.org/abs/2409.04384)
• Keywords: diffusion
• Abstract: Score-based diffusion methods provide a powerful strategy to solve image restoration tasks by flexibly combining a pre-trained foundational prior model with a likelihood function specified during test time. Such methods are predominantly derived from two stochastic processes: reversing Ornstein-Uhlenbeck, which underpins the celebrated denoising diffusion probabilistic models (DDPM) and denoising diffusion implicit models (DDIM), and the Langevin diffusion process. The solutions delivered by DDPM and DDIM are often remarkably realistic, but they are not always consistent with measurements because of likelihood intractability issues and the associated required approximations. Alternatively, using a Langevin process circumvents the intractable likelihood issue, but usually leads to restoration results of inferior quality and longer computing times. This paper presents a novel and highly computationally efficient image restoration method that carefully embeds a foundational DDPM denoiser within an empirical Bayesian Langevin algorithm, which jointly calibrates key model hyper-parameters as it estimates the model's posterior mean. Extensive experimental results on three canonical tasks (image deblurring, super-resolution, and inpainting) demonstrate that the proposed approach improves on state-of-the-art strategies both in image estimation accuracy and computing time.

Title: Question-Answering Dense Video Events

• Authors: Hangyu Qin, Junbin Xiao, Angela Yao
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.04388
• Pdf URL: https://arxiv.org/pdf/2409.04388
• Copy Paste: [[2409.04388]] Question-Answering Dense Video Events(https://arxiv.org/abs/2409.04388)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have shown excellent performance in question-answering of single-event videos. In this paper, we present question-answering dense video events, a novel task that requires answering and grounding the dense-event questions in long videos, thus challenging MLLMs to faithfully comprehend and reason about multiple events occurring over extended time periods. To facilitate the study, we construct DeVE-QA - a dataset featuring 78K questions about 26K events on 10.6K long videos. We then benchmark and show that existing MLLMs excelling at single-event QA struggle to perform well in DeVE-QA. For improvement, we propose DeVi, a novel training-free MLLM approach that highlights a hierarchical captioning module, a temporal event memory module, and a self-consistency checking module to respectively detect, contextualize and memorize, and ground dense-events in long videos for question answering. Extensive experiments show that DeVi is superior at answering dense-event questions and grounding relevant video moments. Compared with existing MLLMs, it achieves a remarkable increase of 4.1 percent and 3.7 percent for G(round)QA accuracy on DeVE-QA and NExT-GQA respectively.

Title: Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences

• Authors: Rui Yu, Runkai Zhao, Cong Nie, Heng Wang, HuaiCheng Yan, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04390
• Pdf URL: https://arxiv.org/pdf/2409.04390
• Copy Paste: [[2409.04390]] Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences(https://arxiv.org/abs/2409.04390)
• Keywords: robust
• Abstract: Accurate and robust LiDAR 3D object detection is essential for comprehensive scene understanding in autonomous driving. Despite its importance, LiDAR detection performance is limited by inherent constraints of point cloud data, particularly under conditions of extended distances and occlusions. Recently, temporal aggregation has been proven to significantly enhance detection accuracy by fusing multi-frame viewpoint information and enriching the spatial representation of objects. In this work, we introduce a novel LiDAR 3D object detection framework, namely LiSTM, to facilitate spatial-temporal feature learning with cross-frame motion forecasting information. We aim to improve the spatial-temporal interpretation capabilities of the LiDAR detector by incorporating a dynamic prior, generated from a non-learnable motion estimation model. Specifically, Motion-Guided Feature Aggregation (MGFA) is proposed to utilize the object trajectory from previous and future motion states to model spatial-temporal correlations into gaussian heatmap over a driving sequence. This motion-based heatmap then guides the temporal feature fusion, enriching the proposed object features. Moreover, we design a Dual Correlation Weighting Module (DCWM) that effectively facilitates the interaction between past and prospective frames through scene- and channel-wise feature abstraction. In the end, a cascade cross-attention-based decoder is employed to refine the 3D prediction. We have conducted experiments on the Waymo and nuScenes datasets to demonstrate that the proposed framework achieves superior 3D detection performance with effective spatial-temporal feature learning.

Title: Exploiting the Data Gap: Utilizing Non-ignorable Missingness to Manipulate Model Learning

• Authors: Deniz Koyuncu, Alex Gittens, Bülent Yener, Moti Yung
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04407
• Pdf URL: https://arxiv.org/pdf/2409.04407
• Copy Paste: [[2409.04407]] Exploiting the Data Gap: Utilizing Non-ignorable Missingness to Manipulate Model Learning(https://arxiv.org/abs/2409.04407)
• Keywords: defense, attack, robust
• Abstract: Missing data is commonly encountered in practice, and when the missingness is non-ignorable, effective remediation depends on knowledge of the missingness mechanism. Learning the underlying missingness mechanism from the data is not possible in general, so adversaries can exploit this fact by maliciously engineering non-ignorable missingness mechanisms. Such Adversarial Missingness (AM) attacks have only recently been motivated and introduced, and then successfully tailored to mislead causal structure learning algorithms into hiding specific cause-and-effect relationships. However, existing AM attacks assume the modeler (victim) uses full-information maximum likelihood methods to handle the missing data, and are of limited applicability when the modeler uses different remediation strategies. In this work we focus on associational learning in the context of AM attacks. We consider (i) complete case analysis, (ii) mean imputation, and (iii) regression-based imputation as alternative strategies used by the modeler. Instead of combinatorially searching for missing entries, we propose a novel probabilistic approximation by deriving the asymptotic forms of these methods used for handling the missing entries. We then formulate the learning of the adversarial missingness mechanism as a bi-level optimization problem. Experiments on generalized linear models show that AM attacks can be used to change the p-values of features from significant to insignificant in real datasets, such as the California-housing dataset, while using relatively moderate amounts of missingness (<20%). Additionally, we assess the robustness of our attacks against defense strategies based on data valuation.

Title: Train Till You Drop: Towards Stable and Robust Source-free Unsupervised 3D Domain Adaptation

• Authors: Björn Michele, Alexandre Boulch, Tuan-Hung Vu, Gilles Puy, Renaud Marlet, Nicolas Courty
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04409
• Pdf URL: https://arxiv.org/pdf/2409.04409
• Copy Paste: [[2409.04409]] Train Till You Drop: Towards Stable and Robust Source-free Unsupervised 3D Domain Adaptation(https://arxiv.org/abs/2409.04409)
• Keywords: robust, segmentation
• Abstract: We tackle the challenging problem of source-free unsupervised domain adaptation (SFUDA) for 3D semantic segmentation. It amounts to performing domain adaptation on an unlabeled target domain without any access to source data; the available information is a model trained to achieve good performance on the source domain. A common issue with existing SFUDA approaches is that performance degrades after some training time, which is a by product of an under-constrained and ill-posed problem. We discuss two strategies to alleviate this issue. First, we propose a sensible way to regularize the learning problem. Second, we introduce a novel criterion based on agreement with a reference model. It is used (1) to stop the training when appropriate and (2) as validator to select hyperparameters without any knowledge on the target domain. Our contributions are easy to implement and readily amenable for all SFUDA methods, ensuring stable improvements over all baselines. We validate our findings on various 3D lidar settings, achieving state-of-the-art performance. The project repository (with code) is: this http URL.

Title: RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs

• Authors: Jiaxing Wu, Lin Ning, Luyang Liu, Harrison Lee, Neo Wu, Chao Wang, Sushant Prakash, Shawn O'Banion, Bradley Green, Jun Xie
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04421
• Pdf URL: https://arxiv.org/pdf/2409.04421
• Copy Paste: [[2409.04421]] RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs(https://arxiv.org/abs/2409.04421)
• Keywords: large language model
• Abstract: LLM-powered personalization agent systems employ Large Language Models (LLMs) to predict users' behavior from their past activities. However, their effectiveness often hinges on the ability to effectively leverage extensive, long user historical data due to its inherent noise and length of such data. Existing pretrained LLMs may generate summaries that are concise but lack the necessary context for downstream tasks, hindering their utility in personalization systems. To address these challenges, we introduce Reinforcement Learning from Prediction Feedback (RLPF). RLPF fine-tunes LLMs to generate concise, human-readable user summaries that are optimized for downstream task performance. By maximizing the usefulness of the generated summaries, RLPF effectively distills extensive user history data while preserving essential information for downstream tasks. Our empirical evaluation demonstrates significant improvements in both extrinsic downstream task utility and intrinsic summary quality, surpassing baseline methods by up to 22% on downstream task performance and achieving an up to 84.59% win rate on Factuality, Abstractiveness, and Readability. RLPF also achieves a remarkable 74% reduction in context length while improving performance on 16 out of 19 unseen tasks and/or datasets, showcasing its generalizability. This approach offers a promising solution for enhancing LLM personalization by effectively transforming long, noisy user histories into informative and human-readable representations.

Title: VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

• Authors: Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, Yao Lu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04429
• Pdf URL: https://arxiv.org/pdf/2409.04429
• Copy Paste: [[2409.04429]] VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation(https://arxiv.org/abs/2409.04429)
• Keywords: diffusion
• Abstract: VILA-U is a Unified foundation model that integrates Video, Image, Language understanding and generation. Traditional visual language models (VLMs) use separate modules for understanding and generating visual content, which can lead to misalignment and increased complexity. In contrast, VILA-U employs a single autoregressive next-token prediction framework for both tasks, eliminating the need for additional components like diffusion models. This approach not only simplifies the model but also achieves near state-of-the-art performance in visual language understanding and generation. The success of VILA-U is attributed to two main factors: the unified vision tower that aligns discrete visual tokens with textual inputs during pretraining, which enhances visual perception, and autoregressive image generation can achieve similar quality as diffusion models with high-quality dataset. This allows VILA-U to perform comparably to more complex models using a fully token-based autoregressive framework.

Title: Theory, Analysis, and Best Practices for Sigmoid Self-Attention

• Authors: Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, Russ Webb
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04431
• Pdf URL: https://arxiv.org/pdf/2409.04431
• Copy Paste: [[2409.04431]] Theory, Analysis, and Best Practices for Sigmoid Self-Attention(https://arxiv.org/abs/2409.04431)
• Keywords: transformer
• Abstract: Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically obtained as the softmax of dot products between keys and queries. Recent work has explored alternatives to softmax attention in transformers, such as ReLU and sigmoid activations. In this work, we revisit sigmoid attention and conduct an in-depth theoretical and empirical analysis. Theoretically, we prove that transformers with sigmoid attention are universal function approximators and benefit from improved regularity compared to softmax attention. Through detailed empirical analysis, we identify stabilization of large initial attention norms during the early stages of training as a crucial factor for the successful training of models with sigmoid attention, outperforming prior attempts. We also introduce FLASHSIGMOID, a hardware-aware and memory-efficient implementation of sigmoid attention yielding a 17% inference kernel speed-up over FLASHATTENTION2 on H100 GPUs. Experiments across language, vision, and speech show that properly normalized sigmoid attention matches the strong performance of softmax attention on a wide range of domains and scales, which previous attempts at sigmoid attention were unable to fully achieve. Our work unifies prior art and establishes best practices for sigmoid attention as a drop-in softmax replacement in transformers.

Title: Accelerating Training with Neuron Interaction and Nowcasting Networks

• Authors: Boris Knyazev, Abhinav Moudgil, Guillaume Lajoie, Eugene Belilovsky, Simon Lacoste-Julien
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.04434
• Pdf URL: https://arxiv.org/pdf/2409.04434
• Copy Paste: [[2409.04434]] Accelerating Training with Neuron Interaction and Nowcasting Networks(https://arxiv.org/abs/2409.04434)
• Keywords: transformer
• Abstract: Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e.g. Adam). However, learnable update rules can be costly and unstable to train and use. A simpler recently proposed approach to accelerate training is to use Adam for most of the optimization steps and periodically, only every few steps, nowcast (predict future) parameters. We improve this approach by Neuron interaction and Nowcasting (NiNo) networks. NiNo leverages neuron connectivity and graph neural networks to more accurately nowcast parameters by learning in a supervised way from a set of training trajectories over multiple tasks. We show that in some networks, such as Transformers, neuron connectivity is non-trivial. By accurately modeling neuron connectivity, we allow NiNo to accelerate Adam training by up to 50\% in vision and language tasks.