2024-10-07

Title: Complex-valued convolutional neural network classification of hand gesture from radar images

• Authors: Shokooh Khandan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02771
• Pdf URL: https://arxiv.org/pdf/2410.02771
• Copy Paste: [[2410.02771]] Complex-valued convolutional neural network classification of hand gesture from radar images(https://arxiv.org/abs/2410.02771)
• Keywords: security
• Abstract: Hand gesture recognition systems have yielded many exciting advancements in the last decade and become more popular in HCI (human-computer interaction) with several application areas, which spans from safety and security applications to automotive field. Various deep neural network architectures have already been inspected for hand gesture recognition systems, including multi-layer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN) and a cascade of the last two architectures known as CNN-RNN. However, a major problem still exists, which is most of the existing ML algorithms are designed and developed the building blocks and techniques for real-valued (RV). Researchers applied various RV techniques on the complex-valued (CV) radar images, such as converting a CV optimisation problem into a RV one, by splitting the complex numbers into their real and imaginary parts. However, the major disadvantage of this method is that the resulting algorithm will double the network dimensions. Recent work on RNNs and other fundamental theoretical analysis suggest that CV numbers have a richer representational capacity, but due to the absence of the building blocks required to design such models, the performance of CV networks are marginalised. In this report, we propose a fully CV-CNN, including all building blocks, forward and backward operations, and derivatives all in complex domain. We explore our proposed classification model on two sets of CV hand gesture radar images in comparison with the equivalent RV model. In chapter five, we propose a CV-forward residual network, for the purpose of binary classification of the two sets of CV hand gesture radar datasets and compare its performance with our proposed CV-CNN and a baseline CV-forward CNN.

Title: Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models

• Authors: Eleonora Lopez, Luigi Sigillo, Federica Colonnese, Massimo Panella, Danilo Comminiello
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02780
• Pdf URL: https://arxiv.org/pdf/2410.02780
• Copy Paste: [[2410.02780]] Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models(https://arxiv.org/abs/2410.02780)
• Keywords: diffusion
• Abstract: Generating images from brain waves is gaining increasing attention due to its potential to advance brain-computer interface (BCI) systems by understanding how brain signals encode visual cues. Most of the literature has focused on fMRI-to-Image tasks as fMRI is characterized by high spatial resolution. However, fMRI is an expensive neuroimaging modality and does not allow for real-time BCI. On the other hand, electroencephalography (EEG) is a low-cost, non-invasive, and portable neuroimaging technique, making it an attractive option for future real-time applications. Nevertheless, EEG presents inherent challenges due to its low spatial resolution and susceptibility to noise and artifacts, which makes generating images from EEG more difficult. In this paper, we address these problems with a streamlined framework based on the ControlNet adapter for conditioning a latent diffusion model (LDM) through EEG signals. We conduct experiments and ablation studies on popular benchmarks to demonstrate that the proposed method beats other state-of-the-art models. Unlike these methods, which often require extensive preprocessing, pretraining, different losses, and captioning models, our approach is efficient and straightforward, requiring only minimal preprocessing and a few components. Code will be available after publication.

Title: Robust Symmetry Detection via Riemannian Langevin Dynamics

• Authors: Jihyeon Je, Jiayi Liu, Guandao Yang, Boyang Deng, Shengqu Cai, Gordon Wetzstein, Or Litany, Leonidas Guibas
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.02786
• Pdf URL: https://arxiv.org/pdf/2410.02786
• Copy Paste: [[2410.02786]] Robust Symmetry Detection via Riemannian Langevin Dynamics(https://arxiv.org/abs/2410.02786)
• Keywords: robust, generative
• Abstract: Symmetries are ubiquitous across all kinds of objects, whether in nature or in man-made creations. While these symmetries may seem intuitive to the human eye, detecting them with a machine is nontrivial due to the vast search space. Classical geometry-based methods work by aggregating "votes" for each symmetry but struggle with noise. In contrast, learning-based methods may be more robust to noise, but often overlook partial symmetries due to the scarcity of annotated data. In this work, we address this challenge by proposing a novel symmetry detection method that marries classical symmetry detection techniques with recent advances in generative modeling. Specifically, we apply Langevin dynamics to a redefined symmetry space to enhance robustness against noise. We provide empirical results on a variety of shapes that suggest our method is not only robust to noise, but can also identify both partial and global symmetries. Moreover, we demonstrate the utility of our detected symmetries in various downstream tasks, such as compression and symmetrization of noisy shapes.

Title: Navigation with VLM framework: Go to Any Language

• Authors: Zecheng Yin, Chonghao Cheng, Lizhen
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02787
• Pdf URL: https://arxiv.org/pdf/2410.02787
• Copy Paste: [[2410.02787]] Navigation with VLM framework: Go to Any Language(https://arxiv.org/abs/2410.02787)
• Keywords: large language model
• Abstract: Navigating towards fully open language goals and exploring open scenes in a manner akin to human exploration have always posed significant challenges. Recently, Vision Large Language Models (VLMs) have demonstrated remarkable capabilities in reasoning with both language and visual data. While many works have focused on leveraging VLMs for navigation in open scenes and with open vocabularies, these efforts often fall short of fully utilizing the potential of VLMs or require substantial computational resources. We introduce Navigation with VLM (NavVLM), a framework that harnesses equipment-level VLMs to enable agents to navigate towards any language goal specific or non-specific in open scenes, emulating human exploration behaviors without any prior training. The agent leverages the VLM as its cognitive core to perceive environmental information based on any language goal and constantly provides exploration guidance during navigation until it reaches the target location or area. Our framework not only achieves state-of-the-art performance in Success Rate (SR) and Success weighted by Path Length (SPL) in traditional specific goal settings but also extends the navigation capabilities to any open-set language goal. We evaluate NavVLM in richly detailed environments from the Matterport 3D (MP3D), Habitat Matterport 3D (HM3D), and Gibson datasets within the Habitat simulator. With the power of VLMs, navigation has entered a new era.

Title: RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture

• Authors: Xiaoyu Pan, Bowen Zheng, Xinwei Jiang, Zijiao Zeng, Qilong Kou, He Wang, Xiaogang Jin
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.02788
• Pdf URL: https://arxiv.org/pdf/2410.02788
• Copy Paste: [[2410.02788]] RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture(https://arxiv.org/abs/2410.02788)
• Keywords: robust, segmentation
• Abstract: Optical motion capture (MoCap) is the "gold standard" for accurately capturing full-body motions. To make use of raw MoCap point data, the system labels the points with corresponding body part locations and solves the full-body motions. However, MoCap data often contains mislabeling, occlusion and positional errors, requiring extensive manual correction. To alleviate this burden, we introduce RoMo, a learning-based framework for robustly labeling and solving raw optical motion capture data. In the labeling stage, RoMo employs a divide-and-conquer strategy to break down the complex full-body labeling challenge into manageable subtasks: alignment, full-body segmentation and part-specific labeling. To utilize the temporal continuity of markers, RoMo generates marker tracklets using a K-partite graph-based clustering algorithm, where markers serve as nodes, and edges are formed based on positional and feature similarities. For motion solving, to prevent error accumulation along the kinematic chain, we introduce a hybrid inverse kinematic solver that utilizes joint positions as intermediate representations and adjusts the template skeleton to match estimated joint positions. We demonstrate that RoMo achieves high labeling and solving accuracy across multiple metrics and various datasets. Extensive comparisons show that our method outperforms state-of-the-art research methods. On a real dataset, RoMo improves the F1 score of hand labeling from 0.94 to 0.98, and reduces joint position error of body motion solving by 25%. Furthermore, RoMo can be applied in scenarios where commercial systems are inadequate. The code and data for RoMo are available at this https URL.

Title: Logic-Free Building Automation: Learning the Control of Room Facilities with Wall Switches and Ceiling Camera

• Authors: Hideya Ochiai, Kohki Hashimoto, Takuya Sakamoto, Seiya Watanabe, Ryosuke Hara, Ryo Yagi, Yuji Aizono, Hiroshi Esaki
• Subjects: cs.CV, cs.AI, cs.HC, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.02789
• Pdf URL: https://arxiv.org/pdf/2410.02789
• Copy Paste: [[2410.02789]] Logic-Free Building Automation: Learning the Control of Room Facilities with Wall Switches and Ceiling Camera(https://arxiv.org/abs/2410.02789)
• Keywords: transformer
• Abstract: Artificial intelligence enables smarter control in building automation by its learning capability of users' preferences on facility control. Reinforcement learning (RL) was one of the approaches to this, but it has many challenges in real-world implementations. We propose a new architecture for logic-free building automation (LFBA) that leverages deep learning (DL) to control room facilities without predefined logic. Our approach differs from RL in that it uses wall switches as supervised signals and a ceiling camera to monitor the environment, allowing the DL model to learn users' preferred controls directly from the scenes and switch states. This LFBA system is tested by our testbed with various conditions and user activities. The results demonstrate the efficacy, achieving 93%-98% control accuracy with VGG, outperforming other DL models such as Vision Transformer and ResNet. This indicates that LFBA can achieve smarter and more user-friendly control by learning from the observable scenes and user interactions.

Title: Leveraging Retrieval Augment Approach for Multimodal Emotion Recognition Under Missing Modalities

• Authors: Qi Fan, Hongyu Yuan, Haolin Zuo, Rui Liu, Guanglai Gao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02804
• Pdf URL: https://arxiv.org/pdf/2410.02804
• Copy Paste: [[2410.02804]] Leveraging Retrieval Augment Approach for Multimodal Emotion Recognition Under Missing Modalities(https://arxiv.org/abs/2410.02804)
• Keywords: robust
• Abstract: Multimodal emotion recognition utilizes complete multimodal information and robust multimodal joint representation to gain high performance. However, the ideal condition of full modality integrity is often not applicable in reality and there always appears the situation that some modalities are missing. For example, video, audio, or text data is missing due to sensor failure or network bandwidth problems, which presents a great challenge to MER research. Traditional methods extract useful information from the complete modalities and reconstruct the missing modalities to learn robust multimodal joint representation. These methods have laid a solid foundation for research in this field, and to a certain extent, alleviated the difficulty of multimodal emotion recognition under missing modalities. However, relying solely on internal reconstruction and multimodal joint learning has its limitations, especially when the missing information is critical for emotion recognition. To address this challenge, we propose a novel framework of Retrieval Augment for Missing Modality Multimodal Emotion Recognition (RAMER), which introduces similar multimodal emotion data to enhance the performance of emotion recognition under missing modalities. By leveraging databases, that contain related multimodal emotion data, we can retrieve similar multimodal emotion information to fill in the gaps left by missing modalities. Various experimental results demonstrate that our framework is superior to existing state-of-the-art approaches in missing modality MER tasks. Our whole project is publicly available on this https URL.

Title: Ingest-And-Ground: Dispelling Hallucinations from Continually-Pretrained LLMs with RAG

• Authors: Chenhao Fang, Derek Larson, Shitong Zhu, Sophie Zeng, Wendy Summer, Yanqing Peng, Yuriy Hulovatyy, Rajeev Rao, Gabriel Forgues, Arya Pudota, Alex Goncalves, Hervé Robert
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02825
• Pdf URL: https://arxiv.org/pdf/2410.02825
• Copy Paste: [[2410.02825]] Ingest-And-Ground: Dispelling Hallucinations from Continually-Pretrained LLMs with RAG(https://arxiv.org/abs/2410.02825)
• Keywords: privacy
• Abstract: This paper presents new methods that have the potential to improve privacy process efficiency with LLM and RAG. To reduce hallucination, we continually pre-train the base LLM model with a privacy-specific knowledge base and then augment it with a semantic RAG layer. Our evaluations demonstrate that this approach enhances the model performance (as much as doubled metrics compared to out-of-box LLM) in handling privacy-related queries, by grounding responses with factual information which reduces inaccuracies.

Title: LinkThief: Combining Generalized Structure Knowledge with Node Similarity for Link Stealing Attack against GNN

• Authors: Yuxing Zhang, Siyuan Meng, Chunchun Chen, Mengyao Peng, Hongyan Gu, Xinli Huang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02826
• Pdf URL: https://arxiv.org/pdf/2410.02826
• Copy Paste: [[2410.02826]] LinkThief: Combining Generalized Structure Knowledge with Node Similarity for Link Stealing Attack against GNN(https://arxiv.org/abs/2410.02826)
• Keywords: privacy, attack, steal
• Abstract: Graph neural networks(GNNs) have a wide range of applications in this http URL studies have shown that Graph neural networks(GNNs) are vulnerable to link stealing attacks,which infers the existence of edges in the target GNN's training this http URL attacks are usually based on the assumption that links exist between two nodes that share similar posteriors;however,they fail to focus on links that do not hold under this this http URL this end,we propose LinkThief,an improved link stealing attack that combines generalized structure knowledge with node similarity,in a scenario where the attackers' background knowledge contains partially leaked target graph and shadow this http URL,to equip the attack model with insights into the link structure spanning both the shadow graph and the target graph,we introduce the idea of creating a Shadow-Target Bridge Graph and extracting edge subgraph structure features from this http URL theoretical analysis from the perspective of privacy theft,we first explore how to implement the aforementioned this http URL upon the findings,we design the Bridge Graph Generator to construct the Shadow-Target Bridge this http URL,the subgraph around the link is sampled by the Edge Subgraph Preparation this http URL,the Edge Structure Feature Extractor is designed to obtain generalized structure knowledge,which is combined with node similarity to form the features provided to the attack this http URL experiments validate the correctness of theoretical analysis and demonstrate that LinkThief still effectively steals links without extra assumptions.

Title: PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System

• Authors: Gary D. Lopez Munoz, Amanda J. Minnich, Roman Lutz, Richard Lundeen, Raja Sekhar Rao Dheekonda, Nina Chikanov, Bolor-Erdene Jagdagdorj, Martin Pouliot, Shiven Chawla, Whitney Maxwell, Blake Bullwinkel, Katherine Pratt, Joris de Gruyter, Charlotte Siska, Pete Bryan, Tori Westerhoff, Chang Kawaguchi, Christian Seifert, Ram Shankar Siva Kumar, Yonatan Zunger
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02828
• Pdf URL: https://arxiv.org/pdf/2410.02828
• Copy Paste: [[2410.02828]] PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System(https://arxiv.org/abs/2410.02828)
• Keywords: security, generative
• Abstract: Generative Artificial Intelligence (GenAI) is becoming ubiquitous in our daily lives. The increase in computational power and data availability has led to a proliferation of both single- and multi-modal models. As the GenAI ecosystem matures, the need for extensible and model-agnostic risk identification frameworks is growing. To meet this need, we introduce the Python Risk Identification Toolkit (PyRIT), an open-source framework designed to enhance red teaming efforts in GenAI systems. PyRIT is a model- and platform-agnostic tool that enables red teamers to probe for and identify novel harms, risks, and jailbreaks in multimodal generative AI models. Its composable architecture facilitates the reuse of core building blocks and allows for extensibility to future models and modalities. This paper details the challenges specific to red teaming generative AI systems, the development and features of PyRIT, and its practical applications in real-world scenarios.

Title: FlipAttack: Jailbreak LLMs via Flipping

• Authors: Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, Bryan Hooi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02832
• Pdf URL: https://arxiv.org/pdf/2410.02832
• Copy Paste: [[2410.02832]] FlipAttack: Jailbreak LLMs via Flipping(https://arxiv.org/abs/2410.02832)
• Keywords: attack, steal
• Abstract: This paper proposes a simple yet effective jailbreak attack named FlipAttack against black-box LLMs. First, from the autoregressive nature, we reveal that LLMs tend to understand the text from left to right and find that they struggle to comprehend the text when noise is added to the left side. Motivated by these insights, we propose to disguise the harmful prompt by constructing left-side noise merely based on the prompt itself, then generalize this idea to 4 flipping modes. Second, we verify the strong ability of LLMs to perform the text-flipping task, and then develop 4 variants to guide LLMs to denoise, understand, and execute harmful behaviors accurately. These designs keep FlipAttack universal, stealthy, and simple, allowing it to jailbreak black-box LLMs within only 1 query. Experiments on 8 LLMs demonstrate the superiority of FlipAttack. Remarkably, it achieves $\sim$98\% attack success rate on GPT-4o, and $\sim$98\% bypass rate against 5 guardrail models on average. The codes are available at GitHub\footnote{this https URL}.

Title: Overcoming Representation Bias in Fairness-Aware data Repair using Optimal Transport

• Authors: Abigail Langbridge, Anthony Quinn, Robert Shorten
• Subjects: cs.LG, cs.CY, math.ST
• Abstract URL: https://arxiv.org/abs/2410.02840
• Pdf URL: https://arxiv.org/pdf/2410.02840
• Copy Paste: [[2410.02840]] Overcoming Representation Bias in Fairness-Aware data Repair using Optimal Transport(https://arxiv.org/abs/2410.02840)
• Keywords: fair
• Abstract: Optimal transport (OT) has an important role in transforming data distributions in a manner which engenders fairness. Typically, the OT operators are learnt from the unfair attribute-labelled data, and then used for their repair. Two significant limitations of this approach are as follows: (i) the OT operators for underrepresented subgroups are poorly learnt (i.e. they are susceptible to representation bias); and (ii) these OT repairs cannot be effected on identically distributed but out-of-sample (i.e.\ archival) data. In this paper, we address both of these problems by adopting a Bayesian nonparametric stopping rule for learning each attribute-labelled component of the data distribution. The induced OT-optimal quantization operators can then be used to repair the archival data. We formulate a novel definition of the fair distributional target, along with quantifiers that allow us to trade fairness against damage in the transformed data. These are used to reveal excellent performance of our representation-bias-tolerant scheme in simulated and benchmark data sets.

Title: Demonstration Attack against In-Context Learning for Code Intelligence

• Authors: Yifei Ge, Weisong Sun, Yihang Lou, Chunrong Fang, Yiran Zhang, Yiming Li, Xiaofang Zhang, Yang Liu, Zhihong Zhao, Zhenyu Chen
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.02841
• Pdf URL: https://arxiv.org/pdf/2410.02841
• Copy Paste: [[2410.02841]] Demonstration Attack against In-Context Learning for Code Intelligence(https://arxiv.org/abs/2410.02841)
• Keywords: security, protect, attack, large language model
• Abstract: Recent advancements in large language models (LLMs) have revolutionized code intelligence by improving programming productivity and alleviating challenges faced by software developers. To further improve the performance of LLMs on specific code intelligence tasks and reduce training costs, researchers reveal a new capability of LLMs: in-context learning (ICL). ICL allows LLMs to learn from a few demonstrations within a specific context, achieving impressive results without parameter updating. However, the rise of ICL introduces new security vulnerabilities in the code intelligence field. In this paper, we explore a novel security scenario based on the ICL paradigm, where attackers act as third-party ICL agencies and provide users with bad ICL content to mislead LLMs outputs in code intelligence tasks. Our study demonstrates the feasibility and risks of such a scenario, revealing how attackers can leverage malicious demonstrations to construct bad ICL content and induce LLMs to produce incorrect outputs, posing significant threats to system security. We propose a novel method to construct bad ICL content called DICE, which is composed of two stages: Demonstration Selection and Bad ICL Construction, constructing targeted bad ICL content based on the user query and transferable across different query inputs. Ultimately, our findings emphasize the critical importance of securing ICL mechanisms to protect code intelligence systems from adversarial manipulation.

Title: Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradients

• Authors: Minh Duong Nguyen, Khanh Le, Khoi Do, Nguyen H.Tran, Duc Nguyen, Chien Trinh, Zhaohui Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02845
• Pdf URL: https://arxiv.org/pdf/2410.02845
• Copy Paste: [[2410.02845]] Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradients(https://arxiv.org/abs/2410.02845)
• Keywords: federate
• Abstract: In personalized Federated Learning (pFL), high data heterogeneity can cause significant gradient divergence across devices, adversely affecting the learning process. This divergence, especially when gradients from different users form an obtuse angle during aggregation, can negate progress, leading to severe weight and gradient update degradation. To address this issue, we introduce a new approach to pFL design, namely Federated Learning with Layer-wise Aggregation via Gradient Analysis (FedLAG), utilizing the concept of gradient conflict at the layer level. Specifically, when layer-wise gradients of different clients form acute angles, those gradients align in the same direction, enabling updates across different clients toward identifying client-invariant features. Conversely, when layer-wise gradient pairs make create obtuse angles, the layers tend to focus on client-specific tasks. In hindsights, FedLAG assigns layers for personalization based on the extent of layer-wise gradient conflicts. Specifically, layers with gradient conflicts are excluded from the global aggregation process. The theoretical evaluation demonstrates that when integrated into other pFL baselines, FedLAG enhances pFL performance by a certain margin. Therefore, our proposed method achieves superior convergence behavior compared with other baselines. Extensive experiments show that our FedLAG outperforms several state-of-the-art methods and can be easily incorporated with many existing methods to further enhance performance.

Title: Position: LLM Unlearning Benchmarks are Weak Measures of Progress

• Authors: Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, Virginia Smith
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02879
• Pdf URL: https://arxiv.org/pdf/2410.02879
• Copy Paste: [[2410.02879]] Position: LLM Unlearning Benchmarks are Weak Measures of Progress(https://arxiv.org/abs/2410.02879)
• Keywords: privacy, large language model
• Abstract: Unlearning methods have the potential to improve the privacy and safety of large language models (LLMs) by removing sensitive or harmful information post hoc. The LLM unlearning research community has increasingly turned toward empirical benchmarks to assess the effectiveness of such methods. In this paper, we find that existing benchmarks provide an overly optimistic and potentially misleading view on the effectiveness of candidate unlearning methods. By introducing simple, benign modifications to a number of popular benchmarks, we expose instances where supposedly unlearned information remains accessible, or where the unlearning process has degraded the model's performance on retained information to a much greater extent than indicated by the original benchmark. We identify that existing benchmarks are particularly vulnerable to modifications that introduce even loose dependencies between the forget and retain information. Further, we show that ambiguity in unlearning targets in existing benchmarks can easily lead to the design of methods that overfit to the given test queries. Based on our findings, we urge the community to be cautious when interpreting benchmark results as reliable measures of progress, and we provide several recommendations to guide future LLM unlearning research.

Title: Universally Optimal Watermarking Schemes for LLMs: from Theory to Practice

• Authors: Haiyun He, Yepeng Liu, Ziqiao Wang, Yongyi Mao, Yuheng Bu
• Subjects: cs.CR, cs.IT, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02890
• Pdf URL: https://arxiv.org/pdf/2410.02890
• Copy Paste: [[2410.02890]] Universally Optimal Watermarking Schemes for LLMs: from Theory to Practice(https://arxiv.org/abs/2410.02890)
• Keywords: attack, robust, watermark, large language model
• Abstract: Large Language Models (LLMs) boosts human efficiency but also poses misuse risks, with watermarking serving as a reliable method to differentiate AI-generated content from human-created text. In this work, we propose a novel theoretical framework for watermarking LLMs. Particularly, we jointly optimize both the watermarking scheme and detector to maximize detection performance, while controlling the worst-case Type-I error and distortion in the watermarked text. Within our framework, we characterize the universally minimum Type-II error, showing a fundamental trade-off between detection performance and distortion. More importantly, we identify the optimal type of detectors and watermarking schemes. Building upon our theoretical analysis, we introduce a practical, model-agnostic and computationally efficient token-level watermarking algorithm that invokes a surrogate model and the Gumbel-max trick. Empirical results on Llama-13B and Mistral-8$\times$7B demonstrate the effectiveness of our method. Furthermore, we also explore how robustness can be integrated into our theoretical framework, which provides a foundation for designing future watermarking systems with improved resilience to adversarial attacks.

Title: Safeguard is a Double-edged Sword: Denial-of-service Attack on Large Language Models

• Authors: Qingzhao Zhang, Ziyang Xiong, Z. Morley Mao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02916
• Pdf URL: https://arxiv.org/pdf/2410.02916
• Copy Paste: [[2410.02916]] Safeguard is a Double-edged Sword: Denial-of-service Attack on Large Language Models(https://arxiv.org/abs/2410.02916)
• Keywords: attack, large language model
• Abstract: Safety is a paramount concern of large language models (LLMs) in their open deployment. To this end, safeguard methods aim to enforce the ethical and responsible use of LLMs through safety alignment or guardrail mechanisms. However, we found that the malicious attackers could exploit false positives of safeguards, i.e., fooling the safeguard model to block safe content mistakenly, leading to a new denial-of-service (DoS) attack on LLMs. Specifically, by software or phishing attacks on user client software, attackers insert a short, seemingly innocuous adversarial prompt into to user prompt templates in configuration files; thus, this prompt appears in final user requests without visibility in the user interface and is not trivial to identify. By designing an optimization process that utilizes gradient and attention information, our attack can automatically generate seemingly safe adversarial prompts, approximately only 30 characters long, that universally block over 97\% of user requests on Llama Guard 3. The attack presents a new dimension of evaluating LLM safeguards focusing on false positives, fundamentally different from the classic jailbreak.

Title: Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification

• Authors: Sudipta Singha Roy, Xindi Wang, Robert E. Mercer, Frank Rudzicz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02930
• Pdf URL: https://arxiv.org/pdf/2410.02930
• Copy Paste: [[2410.02930]] Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification(https://arxiv.org/abs/2410.02930)
• Keywords: transformer
• Abstract: Long document classification presents challenges in capturing both local and global dependencies due to their extensive content and complex structure. Existing methods often struggle with token limits and fail to adequately model hierarchical relationships within documents. To address these constraints, we propose a novel model leveraging a graph-tree structure. Our approach integrates syntax trees for sentence encodings and document graphs for document encodings, which capture fine-grained syntactic relationships and broader document contexts, respectively. We use Tree Transformers to generate sentence encodings, while a graph attention network models inter- and intra-sentence dependencies. During training, we implement bidirectional information propagation from word-to-sentence-to-document and vice versa, which enriches the contextual representation. Our proposed method enables a comprehensive understanding of content at all hierarchical levels and effectively handles arbitrarily long contexts without token limit constraints. Experimental results demonstrate the effectiveness of our approach in all types of long document classification tasks.

Title: SymmetricDiffusers: Learning Discrete Diffusion on Finite Symmetric Groups

• Authors: Yongxing Zhang, Donglin Yang, Renjie Liao
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02942
• Pdf URL: https://arxiv.org/pdf/2410.02942
• Copy Paste: [[2410.02942]] SymmetricDiffusers: Learning Discrete Diffusion on Finite Symmetric Groups(https://arxiv.org/abs/2410.02942)
• Keywords: diffusion
• Abstract: Finite symmetric groups $S_n$ are essential in fields such as combinatorics, physics, and chemistry. However, learning a probability distribution over $S_n$ poses significant challenges due to its intractable size and discrete nature. In this paper, we introduce SymmetricDiffusers, a novel discrete diffusion model that simplifies the task of learning a complicated distribution over $S_n$ by decomposing it into learning simpler transitions of the reverse diffusion using deep neural networks. We identify the riffle shuffle as an effective forward transition and provide empirical guidelines for selecting the diffusion length based on the theory of random walks on finite groups. Additionally, we propose a generalized Plackett-Luce (PL) distribution for the reverse transition, which is provably more expressive than the PL distribution. We further introduce a theoretically grounded "denoising schedule" to improve sampling and learning efficiency. Extensive experiments show that our model achieves state-of-the-art or comparable performances on solving tasks including sorting 4-digit MNIST images, jigsaw puzzles, and traveling salesman problems. Our code is released at this https URL.

Title: LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences

• Authors: Zhenxiao Fu, Fan Chen, Shan Zhou, Haitong Li, Lei Jiang
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.02950
• Pdf URL: https://arxiv.org/pdf/2410.02950
• Copy Paste: [[2410.02950]] LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences(https://arxiv.org/abs/2410.02950)
• Keywords: large language model
• Abstract: Throughout its lifecycle, a large language model (LLM) generates a substantially larger carbon footprint during inference than training. LLM inference requests vary in batch size, prompt length, and token generation number, while cloud providers employ different GPU types and quantities to meet diverse service-level objectives for accuracy and latency. It is crucial for both users and cloud providers to have a tool that quickly and accurately estimates the carbon impact of LLM inferences based on a combination of inference request and hardware configurations before execution. Estimating the carbon footprint of LLM inferences is more complex than training due to lower and highly variable model FLOPS utilization, rendering previous equation-based models inaccurate. Additionally, existing machine learning (ML) prediction methods either lack accuracy or demand extensive training data, as they inadequately handle the distinct prefill and decode phases, overlook hardware-specific features, and inefficiently sample uncommon inference configurations. We introduce \coo, a graph neural network (GNN)-based model that greatly improves the accuracy of LLM inference carbon footprint predictions compared to previous methods.

Title: Unlocking Structured Thinking in Language Models with Cognitive prompting

• Authors: Oliver Kramer, Jill Baumann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02953
• Pdf URL: https://arxiv.org/pdf/2410.02953
• Copy Paste: [[2410.02953]] Unlocking Structured Thinking in Language Models with Cognitive prompting(https://arxiv.org/abs/2410.02953)
• Keywords: interpretability, large language model
• Abstract: We propose cognitive prompting as a novel approach to guide problem-solving in large language models (LLMs) through structured, human-like cognitive operations such as goal clarification, decomposition, filtering, abstraction, and pattern recognition. By employing systematic, step-by-step reasoning, cognitive prompting enables LLMs to efficiently tackle complex, multi-step tasks. We evaluate the effectiveness of cognitive prompting on Meta's LLaMA models, comparing performance on arithmetic reasoning tasks using the GSM8K dataset and on commonsense reasoning benchmarks. Our analysis includes comparisons between models without cognitive prompting, models with a static sequence of cognitive operations, and models using reflective cognitive prompting, where the LLM dynamically self-selects the sequence of cognitive operations. The results show that cognitive prompting, particularly when dynamically adapted, significantly improves the performance of larger models, such as LLaMA3.1 70B, and enhances their ability to handle multi-step reasoning tasks. This approach also improves interpretability and flexibility, highlighting cognitive prompting as a promising strategy for general-purpose AI reasoning.

Title: AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML

• Authors: Patara Trirat, Wonyong Jeong, Sung Ju Hwang
• Subjects: cs.LG, cs.AI, cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2410.02958
• Pdf URL: https://arxiv.org/pdf/2410.02958
• Copy Paste: [[2410.02958]] AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML(https://arxiv.org/abs/2410.02958)
• Keywords: large language model
• Abstract: Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline, such as optimal model search and hyperparameter tuning. Existing AutoML systems often require technical expertise to set up complex tools, which is in general time-consuming and requires a large amount of human effort. Therefore, recent works have started exploiting large language models (LLM) to lessen such burden and increase the usability of AutoML frameworks via a natural language interface, allowing non-expert users to build their data-driven solutions. These methods, however, are usually designed only for a particular process in the AI development pipeline and do not efficiently use the inherent capacity of the LLMs. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML, i.e., from data retrieval to model deployment. AutoML-Agent takes user's task descriptions, facilitates collaboration between specialized LLM agents, and delivers deployment-ready models. Unlike existing work, instead of devising a single plan, we introduce a retrieval-augmented planning strategy to enhance exploration to search for more optimal plans. We also decompose each plan into sub-tasks (e.g., data preprocessing and neural network design) each of which is solved by a specialized agent we build via prompting executing in parallel, making the search process more efficient. Moreover, we propose a multi-stage verification to verify executed results and guide the code generation LLM in implementing successful solutions. Extensive experiments on seven downstream tasks using fourteen datasets show that AutoML-Agent achieves a higher success rate in automating the full AutoML process, yielding systems with good performance throughout the diverse domains.

Title: Coal Mining Question Answering with LLMs

• Authors: Antonio Carlos Rivera, Anthony Moore, Steven Robinson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02959
• Pdf URL: https://arxiv.org/pdf/2410.02959
• Copy Paste: [[2410.02959]] Coal Mining Question Answering with LLMs(https://arxiv.org/abs/2410.02959)
• Keywords: robust, large language model
• Abstract: In this paper, we present a novel approach to coal mining question answering (QA) using large language models (LLMs) combined with tailored prompt engineering techniques. Coal mining is a complex, high-risk industry where accurate, context-aware information is critical for safe and efficient operations. Current QA systems struggle to handle the technical and dynamic nature of mining-related queries. To address these challenges, we propose a multi-turn prompt engineering framework designed to guide LLMs, such as GPT-4, in answering coal mining questions with higher precision and relevance. By breaking down complex queries into structured components, our approach allows LLMs to process nuanced technical information more effectively. We manually curated a dataset of 500 questions from real-world mining scenarios and evaluated the system's performance using both accuracy (ACC) and GPT-4-based scoring metrics. Experiments comparing ChatGPT, Claude2, and GPT-4 across baseline, chain-of-thought (CoT), and multi-turn prompting methods demonstrate that our method significantly improves both accuracy and contextual relevance, with an average accuracy improvement of 15-18\% and a notable increase in GPT-4 scores. The results show that our prompt-engineering approach provides a robust, adaptable solution for domain-specific question answering in high-stakes environments like coal mining.

Title: A Simple Method for Secret-Key Generation Between Mobile Users Across Networks

• Authors: Yingbo Hua
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.02964
• Pdf URL: https://arxiv.org/pdf/2410.02964
• Copy Paste: [[2410.02964]] A Simple Method for Secret-Key Generation Between Mobile Users Across Networks(https://arxiv.org/abs/2410.02964)
• Keywords: privacy, robust
• Abstract: Two or more mobiles users can continuously superimpose sequences of bits chosen from different packets or files already exchanged and authenticated between themselves to continuously renew a secret key for continuous strengthening of their privacy and authentication. This accumulative, adaptable and additive (AAA) method is discussed in this paper. The equivocation to Eve of any bit in the generated key by the AAA method equals to the probability that not all corresponding independent bits exchanged between the users are intercepted by Eve. This performance, achieved without using any knowledge of non-stationary probabilities of bits being intercepted by Eve, is compared to an established capacity achievable using that knowledge. A secrecy robustness of the AAA method against some correlations known to Eve is also discussed.

Title: F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI

• Authors: Xu Zheng, Farhad Shirani, Zhuomin Chen, Chaohao Lin, Wei Cheng, Wenbo Guo, Dongsheng Luo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02970
• Pdf URL: https://arxiv.org/pdf/2410.02970
• Copy Paste: [[2410.02970]] F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI(https://arxiv.org/abs/2410.02970)
• Keywords: robust, fair
• Abstract: Recent research has developed a number of eXplainable AI (XAI) techniques. Although extracting meaningful insights from deep learning models, how to properly evaluate these XAI methods remains an open problem. The most widely used approach is to perturb or even remove what the XAI method considers to be the most important features in an input and observe the changes in the output prediction. This approach although efficient suffers the Out-of-Distribution (OOD) problem as the perturbed samples may no longer follow the original data distribution. A recent method RemOve And Retrain (ROAR) solves the OOD issue by retraining the model with perturbed samples guided by explanations. However, the training may not always converge given the distribution difference. Furthermore, using the model retrained based on XAI methods to evaluate these explainers may cause information leakage and thus lead to unfair comparisons. We propose Fine-tuned Fidelity F-Fidelity, a robust evaluation framework for XAI, which utilizes i) an explanation-agnostic fine-tuning strategy, thus mitigating the information leakage issue and ii) a random masking operation that ensures that the removal step does not generate an OOD input. We designed controlled experiments with state-of-the-art (SOTA) explainers and their degraded version to verify the correctness of our framework. We conducted experiments on multiple data structures, such as images, time series, and natural language. The results demonstrate that F-Fidelity significantly improves upon prior evaluation metrics in recovering the ground-truth ranking of the explainers. Furthermore, we show both theoretically and empirically that, given a faithful explainer, F-Fidelity metric can be used to compute the sparsity of influential input components, i.e., to extract the true explanation size.

Title: Learning Optimal Control and Dynamical Structure of Global Trajectory Search Problems with Diffusion Models

• Authors: Jannik Graebner, Anjian Li, Amlan Sinha, Ryne Beeson
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2410.02976
• Pdf URL: https://arxiv.org/pdf/2410.02976
• Copy Paste: [[2410.02976]] Learning Optimal Control and Dynamical Structure of Global Trajectory Search Problems with Diffusion Models(https://arxiv.org/abs/2410.02976)
• Keywords: diffusion, generative
• Abstract: Spacecraft trajectory design is a global search problem, where previous work has revealed specific solution structures that can be captured with data-driven methods. This paper explores two global search problems in the circular restricted three-body problem: hybrid cost function of minimum fuel/time-of-flight and transfers to energy-dependent invariant manifolds. These problems display a fundamental structure either in the optimal control profile or the use of dynamical structures. We build on our prior generative machine learning framework to apply diffusion models to learn the conditional probability distribution of the search problem and analyze the model's capability to capture these structures.

Title: Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient

• Authors: George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, Daniel Murfet
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02984
• Pdf URL: https://arxiv.org/pdf/2410.02984
• Copy Paste: [[2410.02984]] Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient(https://arxiv.org/abs/2410.02984)
• Keywords: interpretability, transformer
• Abstract: We introduce refined variants of the Local Learning Coefficient (LLC), a measure of model complexity grounded in singular learning theory, to study the development of internal structure in transformer language models during training. By applying these \textit{refined LLCs} (rLLCs) to individual components of a two-layer attention-only transformer, we gain novel insights into the progressive differentiation and specialization of attention heads. Our methodology reveals how attention heads differentiate into distinct functional roles over the course of training, analyzes the types of data these heads specialize to process, and discovers a previously unidentified multigram circuit. These findings demonstrate that rLLCs provide a principled, quantitative toolkit for \textit{developmental interpretability}, which aims to understand models through their evolution across the learning process. More broadly, this work takes a step towards establishing the correspondence between data distributional structure, geometric properties of the loss landscape, learning dynamics, and emergent computational structures in neural networks.

Title: Fully Automated CTC Detection, Segmentation and Classification for Multi-Channel IF Imaging

• Authors: Evan Schwab, Bharat Annaldas, Nisha Ramesh, Anna Lundberg, Vishal Shelke, Xinran Xu, Cole Gilbertson, Jiyun Byun, Ernest T. Lam
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.02988
• Pdf URL: https://arxiv.org/pdf/2410.02988
• Copy Paste: [[2410.02988]] Fully Automated CTC Detection, Segmentation and Classification for Multi-Channel IF Imaging(https://arxiv.org/abs/2410.02988)
• Keywords: extraction, segmentation
• Abstract: Liquid biopsies (eg., blood draws) offer a less invasive and non-localized alternative to tissue biopsies for monitoring the progression of metastatic breast cancer (mBCa). Immunofluoresence (IF) microscopy is a tool to image and analyze millions of blood cells in a patient sample. By detecting and genetically sequencing circulating tumor cells (CTCs) in the blood, personalized treatment plans are achievable for various cancer subtypes. However, CTCs are rare (about 1 in 2M), making manual CTC detection very difficult. In addition, clinicians rely on quantitative cellular biomarkers to manually classify CTCs. This requires prior tasks of cell detection, segmentation and feature extraction. To assist clinicians, we have developed a fully automated machine learning-based production-level pipeline to efficiently detect, segment and classify CTCs in multi-channel IF images. We achieve over 99% sensitivity and 97% specificity on 9,533 cells from 15 mBCa patients. Our pipeline has been successfully deployed on real mBCa patients, reducing a patient average of 14M detected cells to only 335 CTC candidates for manual review.

Title: Towards Universal Certified Robustness with Multi-Norm Training

• Authors: Enyi Jiang, Gagandeep Singh
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.03000
• Pdf URL: https://arxiv.org/pdf/2410.03000
• Copy Paste: [[2410.03000]] Towards Universal Certified Robustness with Multi-Norm Training(https://arxiv.org/abs/2410.03000)
• Keywords: defense, robust
• Abstract: Existing certified training methods can only train models to be robust against a certain perturbation type (e.g. $l_\infty$ or $l_2$). However, an $l_\infty$ certifiably robust model may not be certifiably robust against $l_2$ perturbation (and vice versa) and also has low robustness against other perturbations (e.g. geometric transformation). To this end, we propose the first multi-norm certified training framework \textbf{CURE}, consisting of a new $l_2$ deterministic certified training defense and several multi-norm certified training methods, to attain better \emph{union robustness} when training from scratch or fine-tuning a pre-trained certified model. Further, we devise bound alignment and connect natural training with certified training for better union robustness. Compared with SOTA certified training, \textbf{CURE} improves union robustness up to $22.8\%$ on MNIST, $23.9\%$ on CIFAR-10, and $8.0\%$ on TinyImagenet. Further, it leads to better generalization on a diverse set of challenging unseen geometric perturbations, up to $6.8\%$ on CIFAR-10. Overall, our contributions pave a path towards \textit{universal certified robustness}.

Title: Can Transformers Learn $n$-gram Language Models?

• Authors: Anej Svete, Nadav Borenstein, Mike Zhou, Isabelle Augenstein, Ryan Cotterell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03001
• Pdf URL: https://arxiv.org/pdf/2410.03001
• Copy Paste: [[2410.03001]] Can Transformers Learn $n$-gram Language Models?(https://arxiv.org/abs/2410.03001)
• Keywords: transformer
• Abstract: Much theoretical work has described the ability of transformers to represent formal languages. However, linking theoretical results to empirical performance is not straightforward due to the complex interplay between the architecture, the learning algorithm, and training data. To test whether theoretical lower bounds imply \emph{learnability} of formal languages, we turn to recent work relating transformers to $n$-gram language models (LMs). We study transformers' ability to learn random $n$-gram LMs of two kinds: ones with arbitrary next-symbol probabilities and ones where those are defined with shared parameters. We find that classic estimation techniques for $n$-gram LMs such as add-$\lambda$ smoothing outperform transformers on the former, while transformers perform better on the latter, outperforming methods specifically designed to learn $n$-gram LMs.

Title: MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection

• Authors: Niki Nezakati, Md Kaykobad Reza, Ameya Patil, Mashhour Solh, M. Salman Asif
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03010
• Pdf URL: https://arxiv.org/pdf/2410.03010
• Copy Paste: [[2410.03010]] MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection(https://arxiv.org/abs/2410.03010)
• Keywords: robust
• Abstract: Multimodal learning seeks to combine data from multiple input sources to enhance the performance of different downstream tasks. In real-world scenarios, performance can degrade substantially if some input modalities are missing. Existing methods that can handle missing modalities involve custom training or adaptation steps for each input modality combination. These approaches are either tied to specific modalities or become computationally expensive as the number of input modalities increases. In this paper, we propose Masked Modality Projection (MMP), a method designed to train a single model that is robust to any missing modality scenario. We achieve this by randomly masking a subset of modalities during training and learning to project available input modalities to estimate the tokens for the masked modalities. This approach enables the model to effectively learn to leverage the information from the available modalities to compensate for the missing ones, enhancing missing modality robustness. We conduct a series of experiments with various baseline models and datasets to assess the effectiveness of this strategy. Experiments demonstrate that our approach improves robustness to different missing modality scenarios, outperforming existing methods designed for missing modalities or specific modality combinations.

Title: Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

• Authors: Rose E. Wang, Ana T. Ribeiro, Carly D. Robinson, Susanna Loeb, Dora Demszky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03017
• Pdf URL: https://arxiv.org/pdf/2410.03017
• Copy Paste: [[2410.03017]] Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise(https://arxiv.org/abs/2410.03017)
• Keywords: generative
• Abstract: Generative AI, particularly Language Models (LMs), has the potential to transform real-world domains with societal impact, particularly where access to experts is limited. For example, in education, training novice educators with expert guidance is important for effectiveness but expensive, creating significant barriers to improving education quality at scale. This challenge disproportionately harms students from under-served communities, who stand to gain the most from high-quality education. We introduce Tutor CoPilot, a novel Human-AI approach that leverages a model of expert thinking to provide expert-like guidance to tutors as they tutor. This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities. Following a preregistered analysis plan, we find that students working with tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master topics (p<0.01). Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by 9 p.p. We find that Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages using classifiers to identify pedagogical strategies, and find that tutors with access to Tutor CoPilot are more likely to use high-quality strategies to foster student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student. Tutor interviews highlight how Tutor CoPilot's guidance helps tutors to respond to student needs, though they flag issues in Tutor CoPilot, such as generating suggestions that are not grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students.

Title: Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review

• Authors: Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, Phillip Howard
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03019
• Pdf URL: https://arxiv.org/pdf/2410.03019
• Copy Paste: [[2410.03019]] Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review(https://arxiv.org/abs/2410.03019)
• Keywords: generative, large language model
• Abstract: Peer review is a critical process for ensuring the integrity of published scientific research. Confidence in this process is predicated on the assumption that experts in the relevant domain give careful consideration to the merits of manuscripts which are submitted for publication. With the recent rapid advancements in the linguistic capabilities of large language models (LLMs), a new potential risk to the peer review process is that negligent reviewers will rely on LLMs to perform the often time consuming process of reviewing a paper. In this study, we investigate the ability of existing AI text detection algorithms to distinguish between peer reviews written by humans and different state-of-the-art LLMs. Our analysis shows that existing approaches fail to identify many GPT-4o written reviews without also producing a high number of false positive classifications. To address this deficiency, we propose a new detection approach which surpasses existing methods in the identification of GPT-4o written peer reviews at low levels of false positive classifications. Our work reveals the difficulty of accurately identifying AI-generated text at the individual review level, highlighting the urgent need for new tools and methods to detect this type of unethical application of generative AI.

Title: On Logical Extrapolation for Mazes with Recurrent and Implicit Networks

• Authors: Brandon Knutson, Amandin Chyba Rabeendran, Michael Ivanitskiy, Jordan Pettyjohn, Cecilia Diniz-Behn, Samy Wu Fung, Daniel McKenzie
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.03020
• Pdf URL: https://arxiv.org/pdf/2410.03020
• Copy Paste: [[2410.03020]] On Logical Extrapolation for Mazes with Recurrent and Implicit Networks(https://arxiv.org/abs/2410.03020)
• Keywords: robust
• Abstract: Recent work has suggested that certain neural network architectures-particularly recurrent neural networks (RNNs) and implicit neural networks (INNs) are capable of logical extrapolation. That is, one may train such a network on easy instances of a specific task and then apply it successfully to more difficult instances of the same task. In this paper, we revisit this idea and show that (i) The capacity for extrapolation is less robust than previously suggested. Specifically, in the context of a maze-solving task, we show that while INNs (and some RNNs) are capable of generalizing to larger maze instances, they fail to generalize along axes of difficulty other than maze size. (ii) Models that are explicitly trained to converge to a fixed point (e.g. the INN we test) are likely to do so when extrapolating, while models that are not (e.g. the RNN we test) may exhibit more exotic limiting behaviour such as limit cycles, even when they correctly solve the problem. Our results suggest that (i) further study into why such networks extrapolate easily along certain axes of difficulty yet struggle with others is necessary, and (ii) analyzing the dynamics of extrapolation may yield insights into designing more efficient and interpretable logical extrapolators.

Title: PixelShuffler: A Simple Image Translation Through Pixel Rearrangement

• Authors: Omar Zamzam
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.03021
• Pdf URL: https://arxiv.org/pdf/2410.03021
• Copy Paste: [[2410.03021]] PixelShuffler: A Simple Image Translation Through Pixel Rearrangement(https://arxiv.org/abs/2410.03021)
• Keywords: diffusion
• Abstract: Image-to-image translation is a topic in computer vision that has a vast range of use cases ranging from medical image translation, such as converting MRI scans to CT scans or to other MRI contrasts, to image colorization, super-resolution, domain adaptation, and generating photorealistic images from sketches or semantic maps. Image style transfer is also a widely researched application of image-to-image translation, where the goal is to synthesize an image that combines the content of one image with the style of another. Existing state-of-the-art methods often rely on complex neural networks, including diffusion models and language models, to achieve high-quality style transfer, but these methods can be computationally expensive and intricate to implement. In this paper, we propose a novel pixel shuffle method that addresses the image-to-image translation problem generally with a specific demonstrative application in style transfer. The proposed method approaches style transfer by shuffling the pixels of the style image such that the mutual information between the shuffled image and the content image is maximized. This approach inherently preserves the colors of the style image while ensuring that the structural details of the content image are retained in the stylized output. We demonstrate that this simple and straightforward method produces results that are comparable to state-of-the-art techniques, as measured by the Learned Perceptual Image Patch Similarity (LPIPS) loss for content preservation and the Fréchet Inception Distance (FID) score for style similarity. Our experiments validate that the proposed pixel shuffle method achieves competitive performance with significantly reduced complexity, offering a promising alternative for efficient image style transfer, as well as a promise in usability of the method in general image-to-image translation tasks.

Title: Flow Matching with Gaussian Process Priors for Probabilistic Time Series Forecasting

• Authors: Marcel Kollovieh, Marten Lienen, David Lüdke, Leo Schwinn, Stephan Günnemann
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.03024
• Pdf URL: https://arxiv.org/pdf/2410.03024
• Copy Paste: [[2410.03024]] Flow Matching with Gaussian Process Priors for Probabilistic Time Series Forecasting(https://arxiv.org/abs/2410.03024)
• Keywords: diffusion, generative
• Abstract: Recent advancements in generative modeling, particularly diffusion models, have opened new directions for time series modeling, achieving state-of-the-art performance in forecasting and synthesis. However, the reliance of diffusion-based models on a simple, fixed prior complicates the generative process since the data and prior distributions differ significantly. We introduce TSFlow, a conditional flow matching (CFM) model for time series that simplifies the generative problem by combining Gaussian processes, optimal transport paths, and data-dependent prior distributions. By incorporating (conditional) Gaussian processes, TSFlow aligns the prior distribution more closely with the temporal structure of the data, enhancing both unconditional and conditional generation. Furthermore, we propose conditional prior sampling to enable probabilistic forecasting with an unconditionally trained model. In our experimental evaluation on eight real-world datasets, we demonstrate the generative capabilities of TSFlow, producing high-quality unconditional samples. Finally, we show that both conditionally and unconditionally trained models achieve competitive results in forecasting benchmarks, surpassing other methods on 6 out of 8 datasets.

Title: Characterizing Context Influence and Hallucination in Summarization

• Authors: James Flemings, Wanrong Zhang, Bo Jiang, Zafar Takhirov, Murali Annavaram
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03026
• Pdf URL: https://arxiv.org/pdf/2410.03026
• Copy Paste: [[2410.03026]] Characterizing Context Influence and Hallucination in Summarization(https://arxiv.org/abs/2410.03026)
• Keywords: privacy, large language model
• Abstract: Although Large Language Models (LLMs) have achieved remarkable performance in numerous downstream tasks, their ubiquity has raised two significant concerns. One is that LLMs can hallucinate by generating content that contradicts relevant contextual information; the other is that LLMs can inadvertently leak private information due to input regurgitation. Many prior works have extensively studied each concern independently, but none have investigated them simultaneously. Furthermore, auditing the influence of provided context during open-ended generation with a privacy emphasis is understudied. To this end, we comprehensively characterize the influence and hallucination of contextual information during summarization. We introduce a definition for context influence and Context-Influence Decoding (CID), and then we show that amplifying the context (by factoring out prior knowledge) and the context being out of distribution with respect to prior knowledge increases the context's influence on an LLM. Moreover, we show that context influence gives a lower bound of the private information leakage of CID. We corroborate our analytical findings with experimental evaluations that show improving the F1 ROGUE-L score on CNN-DM for LLaMA 3 by $\textbf{10}$% over regular decoding also leads to $\textbf{1.5x}$ more influence by the context. Moreover, we empirically evaluate how context influence and hallucination are affected by (1) model capacity, (2) context size, (3) the length of the current response, and (4) different token $n$-grams of the context. Our code can be accessed here: this https URL.

Title: MLP-KAN: Unifying Deep Representation and Function Learning

• Authors: Yunhong He, Yifeng Xie, Zhengqing Yuan, Lichao Sun
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03027
• Pdf URL: https://arxiv.org/pdf/2410.03027
• Copy Paste: [[2410.03027]] MLP-KAN: Unifying Deep Representation and Function Learning(https://arxiv.org/abs/2410.03027)
• Keywords: transformer
• Abstract: Recent advancements in both representation learning and function learning have demonstrated substantial promise across diverse domains of artificial intelligence. However, the effective integration of these paradigms poses a significant challenge, particularly in cases where users must manually decide whether to apply a representation learning or function learning model based on dataset characteristics. To address this issue, we introduce MLP-KAN, a unified method designed to eliminate the need for manual model selection. By integrating Multi-Layer Perceptrons (MLPs) for representation learning and Kolmogorov-Arnold Networks (KANs) for function learning within a Mixture-of-Experts (MoE) architecture, MLP-KAN dynamically adapts to the specific characteristics of the task at hand, ensuring optimal performance. Embedded within a transformer-based framework, our work achieves remarkable results on four widely-used datasets across diverse domains. Extensive experimental evaluation demonstrates its superior versatility, delivering competitive performance across both deep representation and function learning tasks. These findings highlight the potential of MLP-KAN to simplify the model selection process, offering a comprehensive, adaptable solution across various domains. Our code and weights are available at \url{this https URL}.

Title: Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness

• Authors: Boqian Wu, Qiao Xiao, Shunxin Wang, Nicola Strisciuglio, Mykola Pechenizkiy, Maurice van Keulen, Decebal Constantin Mocanu, Elena Mocanu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03030
• Pdf URL: https://arxiv.org/pdf/2410.03030
• Copy Paste: [[2410.03030]] Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness(https://arxiv.org/abs/2410.03030)
• Keywords: robust
• Abstract: It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, we claim that, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10% and up to 50%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.

Title: Disentangling Textual and Acoustic Features of Neural Speech Representations

• Authors: Hosein Mohebbi, Grzegorz Chrupała, Willem Zuidema, Afra Alishahi, Ivan Titov
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.03037
• Pdf URL: https://arxiv.org/pdf/2410.03037
• Copy Paste: [[2410.03037]] Disentangling Textual and Acoustic Features of Neural Speech Representations(https://arxiv.org/abs/2410.03037)
• Keywords: privacy
• Abstract: Neural speech models build deeply entangled internal representations, which capture a variety of features (e.g., fundamental frequency, loudness, syntactic category, or semantic content of a word) in a distributed encoding. This complexity makes it difficult to track the extent to which such representations rely on textual and acoustic information, or to suppress the encoding of acoustic features that may pose privacy risks (e.g., gender or speaker identity) in critical, real-world applications. In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates complex speech representations into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a given downstream task. We apply and evaluate our framework to emotion recognition and speaker identification downstream tasks, quantifying the contribution of textual and acoustic features at each model layer. Additionally, we explore the application of our disentanglement framework as an attribution method to identify the most salient speech frame representations from both the textual and acoustic perspectives.

Title: Revealing the Unseen: Guiding Personalized Diffusion Models to Expose Training Data

• Authors: Xiaoyu Wu, Jiaru Zhang, Steven Wu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03039
• Pdf URL: https://arxiv.org/pdf/2410.03039
• Copy Paste: [[2410.03039]] Revealing the Unseen: Guiding Personalized Diffusion Models to Expose Training Data(https://arxiv.org/abs/2410.03039)
• Keywords: extraction, diffusion
• Abstract: Diffusion Models (DMs) have evolved into advanced image generation tools, especially for few-shot fine-tuning where a pretrained DM is fine-tuned on a small set of images to capture specific styles or objects. Many people upload these personalized checkpoints online, fostering communities such as Civitai and HuggingFace. However, model owners may overlook the potential risks of data leakage by releasing their fine-tuned checkpoints. Moreover, concerns regarding copyright violations arise when unauthorized data is used during fine-tuning. In this paper, we ask: "Can training data be extracted from these fine-tuned DMs shared online?" A successful extraction would present not only data leakage threats but also offer tangible evidence of copyright infringement. To answer this, we propose FineXtract, a framework for extracting fine-tuning data. Our method approximates fine-tuning as a gradual shift in the model's learned distribution -- from the original pretrained DM toward the fine-tuning data. By extrapolating the models before and after fine-tuning, we guide the generation toward high-probability regions within the fine-tuned data distribution. We then apply a clustering algorithm to extract the most probable images from those generated using this extrapolated guidance. Experiments on DMs fine-tuned with datasets such as WikiArt, DreamBooth, and real-world checkpoints posted online validate the effectiveness of our method, extracting approximately 20% of fine-tuning data in most cases, significantly surpassing baseline performance.

Title: Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models

• Authors: Mingxue Xu, Sadia Sharmin, Danilo P. Mandic
• Subjects: cs.CL, cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2410.03040
• Pdf URL: https://arxiv.org/pdf/2410.03040
• Copy Paste: [[2410.03040]] Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models(https://arxiv.org/abs/2410.03040)
• Keywords: generative
• Abstract: Matrix and tensor-guided parametrization for Natural Language Processing (NLP) models is fundamentally useful for the improvement of the model's systematic efficiency. However, the internal links between these two algebra structures and language model parametrization are poorly understood. Also, the existing matrix and tensor research is math-heavy and far away from machine learning (ML) and NLP research concepts. These two issues result in the recent progress on matrices and tensors for model parametrization being more like a loose collection of separate components from matrix/tensor and NLP studies, rather than a well-structured unified approach, further hindering algorithm design. To this end, we propose a unified taxonomy, which bridges the matrix/tensor compression approaches and model compression concepts in ML and NLP research. Namely, we adopt an elementary concept in linear algebra, that of a subspace, which is also the core concept in geometric algebra, to reformulate the matrix/tensor and ML/NLP concepts (e.g. attention mechanism) under one umbrella. In this way, based on our subspace formalization, typical matrix and tensor decomposition algorithms can be interpreted as geometric transformations. Finally, we revisit recent literature on matrix- or tensor-guided language model compression, rephrase and compare their core ideas, and then point out the current research gap and potential solutions.

Title: FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning

• Authors: Nurbek Tastan, Samuel Horvath, Martin Takac, Karthik Nandakumar
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.03042
• Pdf URL: https://arxiv.org/pdf/2410.03042
• Copy Paste: [[2410.03042]] FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learning(https://arxiv.org/abs/2410.03042)
• Keywords: federate
• Abstract: Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is extreme data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This personalized warmup allows the participants to focus initially on learning specific subnetworks tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (FedPeWS) approach improves accuracy and convergence speed over standard federated optimization methods.

Title: Towards Understanding the Feasibility of Machine Unlearning

• Authors: Mahtab Sarvmaili, Hassan Sajjad, Ga Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03043
• Pdf URL: https://arxiv.org/pdf/2410.03043
• Copy Paste: [[2410.03043]] Towards Understanding the Feasibility of Machine Unlearning(https://arxiv.org/abs/2410.03043)
• Keywords: privacy
• Abstract: In light of recent privacy regulations, machine unlearning has attracted significant attention in the research community. However, current studies predominantly assess the overall success of unlearning approaches, overlooking the varying difficulty of unlearning individual training samples. As a result, the broader feasibility of machine unlearning remains under-explored. This paper presents a set of novel metrics for quantifying the difficulty of unlearning by jointly considering the properties of target model and data distribution. Specifically, we propose several heuristics to assess the conditions necessary for a successful unlearning operation, examine the variations in unlearning difficulty across different training samples, and present a ranking mechanism to identify the most challenging samples to unlearn. We highlight the effectiveness of the Kernelized Stein Discrepancy (KSD), a parameterized kernel function tailored to each model and dataset, as a heuristic for evaluating unlearning difficulty. Our approach is validated through multiple classification tasks and established machine unlearning algorithms, demonstrating the practical feasibility of unlearning operations across diverse scenarios.

Title: Scalable Frame-based Construction of Sociocultural NormBases for Socially-Aware Dialogues

• Authors: Shilin Qu, Weiqing Wang, Xin Zhou, Haolan Zhan, Zhuang Li, Lizhen Qu, Linhao Luo, Yuan-Fang Li, Gholamreza Haffari
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03049
• Pdf URL: https://arxiv.org/pdf/2410.03049
• Copy Paste: [[2410.03049]] Scalable Frame-based Construction of Sociocultural NormBases for Socially-Aware Dialogues(https://arxiv.org/abs/2410.03049)
• Keywords: large language model
• Abstract: Sociocultural norms serve as guiding principles for personal conduct in social interactions, emphasizing respect, cooperation, and appropriate behavior, which is able to benefit tasks including conversational information retrieval, contextual information retrieval and retrieval-enhanced machine learning. We propose a scalable approach for constructing a Sociocultural Norm (SCN) Base using Large Language Models (LLMs) for socially aware dialogues. We construct a comprehensive and publicly accessible Chinese Sociocultural NormBase. Our approach utilizes socially aware dialogues, enriched with contextual frames, as the primary data source to constrain the generating process and reduce the hallucinations. This enables extracting of high-quality and nuanced natural-language norm statements, leveraging the pragmatic implications of utterances with respect to the situation. As real dialogue annotated with gold frames are not readily available, we propose using synthetic data. Our empirical results show: (i) the quality of the SCNs derived from synthetic data is comparable to that from real dialogues annotated with gold frames, and (ii) the quality of the SCNs extracted from real data, annotated with either silver (predicted) or gold frames, surpasses that without the frame annotations. We further show the effectiveness of the extracted SCNs in a RAG-based (Retrieval-Augmented Generation) model to reason about multiple downstream dialogue tasks.

Title: CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization

• Authors: Shigemichi Matsuzaki, Kazuhito Tanaka, Kazuhiro Shintani
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2410.03054
• Pdf URL: https://arxiv.org/pdf/2410.03054
• Copy Paste: [[2410.03054]] CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization(https://arxiv.org/abs/2410.03054)
• Keywords: robust, extraction
• Abstract: This letter proposes a method of global localization on a map with semantic object landmarks. One of the most promising approaches for localization on object maps is to use semantic graph matching using landmark descriptors calculated from the distribution of surrounding objects. These descriptors are vulnerable to misclassification and partial observations. Moreover, many existing methods rely on inlier extraction using RANSAC, which is stochastic and sensitive to a high outlier rate. To address the former issue, we augment the correspondence matching using Vision Language Models (VLMs). Landmark discriminability is improved by VLM embeddings, which are independent of surrounding objects. In addition, inliers are estimated deterministically using a graph-theoretic approach. We also incorporate pose calculation using the weighted least squares considering correspondence similarity and observation completeness to improve the robustness. We confirmed improvements in matching and pose estimation accuracy through experiments on ScanNet and TUM datasets.

Title: Permissive Information-Flow Analysis for Large Language Models

• Authors: Shoaib Ahmed Siddiqui, Radhika Gaonkar, Boris Köpf, David Krueger, Andrew Paverd, Ahmed Salem, Shruti Tople, Lukas Wutschitz, Menglin Xia, Santiago Zanella-Béguelin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03055
• Pdf URL: https://arxiv.org/pdf/2410.03055
• Copy Paste: [[2410.03055]] Permissive Information-Flow Analysis for Large Language Models(https://arxiv.org/abs/2410.03055)
• Keywords: security, privacy, large language model
• Abstract: Large Language Models (LLMs) are rapidly becoming commodity components of larger software systems. This poses natural security and privacy problems: poisoned data retrieved from one component can change the model's behavior and compromise the entire system, including coercing the model to spread confidential data to untrusted components. One promising approach is to tackle this problem at the system level via dynamic information flow (aka taint) tracking. Unfortunately, the traditional approach of propagating the most restrictive input label to the output is too conservative for applications where LLMs operate on inputs retrieved from diverse sources. In this paper, we propose a novel, more permissive approach to propagate information flow labels through LLM queries. The key idea behind our approach is to propagate only the labels of the samples that were influential in generating the model output and to eliminate the labels of unnecessary input. We implement and investigate the effectiveness of two variations of this approach, based on (i) prompt-based retrieval augmentation, and (ii) a $k$-nearest-neighbors language model. We compare these with the baseline of an introspection-based influence estimator that directly asks the language model to predict the output label. The results obtained highlight the superiority of our prompt-based label propagator, which improves the label in more than 85% of the cases in an LLM agent setting. These findings underscore the practicality of permissive label propagation for retrieval augmentation.

Title: DiffKillR: Killing and Recreating Diffeomorphisms for Cell Annotation in Dense Microscopy Images

• Authors: Chen Liu, Danqi Liao, Alejandro Parada-Mayorga, Alejandro Ribeiro, Marcello DiStasio, Smita Krishnaswamy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03058
• Pdf URL: https://arxiv.org/pdf/2410.03058
• Copy Paste: [[2410.03058]] DiffKillR: Killing and Recreating Diffeomorphisms for Cell Annotation in Dense Microscopy Images(https://arxiv.org/abs/2410.03058)
• Keywords: robust
• Abstract: The proliferation of digital microscopy images, driven by advances in automated whole slide scanning, presents significant opportunities for biomedical research and clinical diagnostics. However, accurately annotating densely packed information in these images remains a major challenge. To address this, we introduce DiffKillR, a novel framework that reframes cell annotation as the combination of archetype matching and image registration tasks. DiffKillR employs two complementary neural networks: one that learns a diffeomorphism-invariant feature space for robust cell matching and another that computes the precise warping field between cells for annotation mapping. Using a small set of annotated archetypes, DiffKillR efficiently propagates annotations across large microscopy images, reducing the need for extensive manual labeling. More importantly, it is suitable for any type of pixel-level annotation. We will discuss the theoretical properties of DiffKillR and validate it on three microscopy tasks, demonstrating its advantages over existing supervised, semi-supervised, and unsupervised methods.

Title: Compute Or Load KV Cache? Why Not Both?

• Authors: Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03065
• Pdf URL: https://arxiv.org/pdf/2410.03065
• Copy Paste: [[2410.03065]] Compute Or Load KV Cache? Why Not Both?(https://arxiv.org/abs/2410.03065)
• Keywords: large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have significantly increased context window sizes, enabling sophisticated applications but also introducing substantial computational overheads, particularly computing key-value (KV) cache in the prefill stage. Prefix caching has emerged to save GPU power in this scenario, which saves KV cache at disks and reuse them across multiple queries. However, traditional prefix caching mechanisms often suffer from substantial latency because the speed of loading KV cache from disks to GPU memory is bottlenecked by the throughput of I/O devices. To optimize the latency of long-context prefill, we propose Cake, a novel KV cache loader, which employs a bidirectional parallelized KV cache generation strategy. Upon receiving a prefill task, Cake simultaneously and dynamically loads saved KV cache from prefix cache locations and computes KV cache on local GPUs, maximizing the utilization of available computation and I/O bandwidth resources. Additionally, Cake automatically adapts to diverse system statuses without manual parameter. tuning. In experiments on various prompt datasets, GPUs, and I/O devices, Cake offers up to 68.1% Time To First Token (TTFT) reduction compare with compute-only method and 94.6% TTFT reduction compare with I/O-only method.

Title: FedCert: Federated Accuracy Certification

• Authors: Minh Hieu Nguyen, Huu Tien Nguyen, Trung Thanh Nguyen, Manh Duong Nguyen, Trong Nghia Hoang, Truong Thao Nguyen, Phi Le Nguyen
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.03067
• Pdf URL: https://arxiv.org/pdf/2410.03067
• Copy Paste: [[2410.03067]] FedCert: Federated Accuracy Certification(https://arxiv.org/abs/2410.03067)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models in a decentralized manner, preserving data privacy by keeping local data on clients. However, evaluating the robustness of these models against data perturbations on clients remains a significant challenge. Previous studies have assessed the effectiveness of models in centralized training based on certified accuracy, which guarantees that a certain percentage of the model's predictions will remain correct even if the input data is perturbed. However, the challenge of extending these evaluations to FL remains unresolved due to the unknown client's local data. To tackle this challenge, this study proposed a method named FedCert to take the first step toward evaluating the robustness of FL systems. The proposed method is designed to approximate the certified accuracy of a global model based on the certified accuracy and class distribution of each client. Additionally, considering the Non-Independent and Identically Distributed (Non-IID) nature of data in real-world scenarios, we introduce the client grouping algorithm to ensure reliable certified accuracy during the aggregation step of the approximation algorithm. Through theoretical analysis, we demonstrate the effectiveness of FedCert in assessing the robustness and reliability of FL systems. Moreover, experimental results on the CIFAR-10 and CIFAR-100 datasets under various scenarios show that FedCert consistently reduces the estimation error compared to baseline methods. This study offers a solution for evaluating the robustness of FL systems and lays the groundwork for future research to enhance the dependability of decentralized learning. The source code is available at this https URL.

Title: FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization

• Authors: Manh Duong Nguyen, Trung Thanh Nguyen, Huy Hieu Pham, Trong Nghia Hoang, Phi Le Nguyen, Thanh Trung Huynh
• Subjects: cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.03070
• Pdf URL: https://arxiv.org/pdf/2410.03070
• Copy Paste: [[2410.03070]] FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization(https://arxiv.org/abs/2410.03070)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is a method for training machine learning models using distributed data sources. It ensures privacy by allowing clients to collaboratively learn a shared global model while storing their data locally. However, a significant challenge arises when dealing with missing modalities in clients' datasets, where certain features or modalities are unavailable or incomplete, leading to heterogeneous data distribution. While previous studies have addressed the issue of complete-modality missing, they fail to tackle partial-modality missing on account of severe heterogeneity among clients at an instance level, where the pattern of missing data can vary significantly from one sample to another. To tackle this challenge, this study proposes a novel framework named FedMAC, designed to address multi-modality missing under conditions of partial-modality missing in FL. Additionally, to avoid trivial aggregation of multi-modal features, we introduce contrastive-based regularization to impose additional constraints on the latent representation space. The experimental results demonstrate the effectiveness of FedMAC across various client configurations with statistical heterogeneity, outperforming baseline methods by up to 26% in severe missing scenarios, highlighting its potential as a solution for the challenge of partially missing modalities in federated systems.

Title: Enhancing Short-Text Topic Modeling with LLM-Driven Context Expansion and Prefix-Tuned VAEs

• Authors: Pritom Saha Akash, Kevin Chen-Chuan Chang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.03071
• Pdf URL: https://arxiv.org/pdf/2410.03071
• Copy Paste: [[2410.03071]] Enhancing Short-Text Topic Modeling with LLM-Driven Context Expansion and Prefix-Tuned VAEs(https://arxiv.org/abs/2410.03071)
• Keywords: large language model
• Abstract: Topic modeling is a powerful technique for uncovering hidden themes within a collection of documents. However, the effectiveness of traditional topic models often relies on sufficient word co-occurrence, which is lacking in short texts. Therefore, existing approaches, whether probabilistic or neural, frequently struggle to extract meaningful patterns from such data, resulting in incoherent topics. To address this challenge, we propose a novel approach that leverages large language models (LLMs) to extend short texts into more detailed sequences before applying topic modeling. To further improve the efficiency and solve the problem of semantic inconsistency from LLM-generated texts, we propose to use prefix tuning to train a smaller language model coupled with a variational autoencoder for short-text topic modeling. Our method significantly improves short-text topic modeling performance, as demonstrated by extensive experiments on real-world datasets with extreme data sparsity, outperforming current state-of-the-art topic models.

Title: Multilingual Topic Classification in X: Dataset and Analysis

• Authors: Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Jose Camacho-Collados
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03075
• Pdf URL: https://arxiv.org/pdf/2410.03075
• Copy Paste: [[2410.03075]] Multilingual Topic Classification in X: Dataset and Analysis(https://arxiv.org/abs/2410.03075)
• Keywords: robust
• Abstract: In the dynamic realm of social media, diverse topics are discussed daily, transcending linguistic boundaries. However, the complexities of understanding and categorising this content across various languages remain an important challenge with traditional techniques like topic modelling often struggling to accommodate this multilingual diversity. In this paper, we introduce X-Topic, a multilingual dataset featuring content in four distinct languages (English, Spanish, Japanese, and Greek), crafted for the purpose of tweet topic classification. Our dataset includes a wide range of topics, tailored for social media content, making it a valuable resource for scientists and professionals working on cross-linguistic analysis, the development of robust multilingual models, and computational scientists studying online dialogue. Finally, we leverage X-Topic to perform a comprehensive cross-linguistic and multilingual analysis, and compare the capabilities of current general- and domain-specific language models.

Title: CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions

• Authors: Jun Rao, Xuebo Liu, Lian Lian, Shengjun Cheng, Yunjie Liao, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03077
• Pdf URL: https://arxiv.org/pdf/2410.03077
• Copy Paste: [[2410.03077]] CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions(https://arxiv.org/abs/2410.03077)
• Keywords: large language model
• Abstract: With instruction tuning, Large Language Models (LLMs) can enhance their ability to adhere to commands. Diverging from most works focusing on data mixing, our study concentrates on enhancing the model's capabilities from the perspective of data sampling during training. Drawing inspiration from the human learning process, where it is generally easier to master solutions to similar topics through focused practice on a single type of topic, we introduce a novel instruction tuning strategy termed CommonIT: Commonality-aware Instruction Tuning. Specifically, we cluster instruction datasets into distinct groups with three proposed metrics (Task, Embedding and Length). We ensure each training mini-batch, or "partition", consists solely of data from a single group, which brings about both data randomness across mini-batches and intra-batch data similarity. Rigorous testing on LLaMa models demonstrates CommonIT's effectiveness in enhancing the instruction-following capabilities of LLMs through IT datasets (FLAN, CoT, and Alpaca) and models (LLaMa2-7B, Qwen2-7B, LLaMa 13B, and BLOOM 7B). CommonIT consistently boosts an average improvement of 2.1\% on the general domain (i.e., the average score of Knowledge, Reasoning, Multilinguality and Coding) with the Length metric, and 5.2\% on the special domain (i.e., GSM, Openfunctions and Code) with the Task metric, and 3.8\% on the specific tasks (i.e., MMLU) with the Embedding metric. Code is available at \url{this https URL}.

Title: Generative Edge Detection with Stable Diffusion

• Authors: Caixia Zhou, Yaping Huang, Mochu Xiang, Jiahui Ren, Haibin Ling, Jing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03080
• Pdf URL: https://arxiv.org/pdf/2410.03080
• Copy Paste: [[2410.03080]] Generative Edge Detection with Stable Diffusion(https://arxiv.org/abs/2410.03080)
• Keywords: diffusion, generative
• Abstract: Edge detection is typically viewed as a pixel-level classification problem mainly addressed by discriminative methods. Recently, generative edge detection methods, especially diffusion model based solutions, are initialized in the edge detection task. Despite great potential, the retraining of task-specific designed modules and multi-step denoising inference limits their broader applications. Upon closer investigation, we speculate that part of the reason is the under-exploration of the rich discriminative information encoded in extensively pre-trained large models (\eg, stable diffusion models). Thus motivated, we propose a novel approach, named Generative Edge Detector (GED), by fully utilizing the potential of the pre-trained stable diffusion model. Our model can be trained and inferred efficiently without specific network design due to the rich high-level and low-level prior knowledge empowered by the pre-trained stable diffusion. Specifically, we propose to finetune the denoising U-Net and predict latent edge maps directly, by taking the latent image feature maps as input. Additionally, due to the subjectivity and ambiguity of the edges, we also incorporate the granularity of the edges into the denoising U-Net model as one of the conditions to achieve controllable and diverse predictions. Furthermore, we devise a granularity regularization to ensure the relative granularity relationship of the multiple predictions. We conduct extensive experiments on multiple datasets and achieve competitive performance (\eg, 0.870 and 0.880 in terms of ODS and OIS on the BSDS test dataset).

Title: UNComp: Uncertainty-Aware Long-Context Compressor for Efficient Large Language Model Inference

• Authors: Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Lingpeng Kong, Ngai Wong
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03090
• Pdf URL: https://arxiv.org/pdf/2410.03090
• Copy Paste: [[2410.03090]] UNComp: Uncertainty-Aware Long-Context Compressor for Efficient Large Language Model Inference(https://arxiv.org/abs/2410.03090)
• Keywords: large language model
• Abstract: Deploying large language models (LLMs) is challenging due to their high memory and computational demands, especially during long-context inference. While key-value (KV) caching accelerates inference by reusing previously computed keys and values, it also introduces significant memory overhead. Existing KV cache compression methods such as eviction and merging typically compress the KV cache after it is generated and overlook the eviction of hidden states, failing to improve the speed of the prefilling stage. Additionally, applying a uniform compression rate across different attention heads can harm crucial retrieval heads in needle-in-a-haystack tasks due to excessive compression. In this paper, we propose UNComp, an uncertainty-aware compression scheme that leverages matrix entropy to estimate model uncertainty across layers and heads at the token sequence level. By grouping layers and heads based on their uncertainty, UNComp adaptively compresses both the hidden states and the KV cache. Our method achieves a 1.6x speedup in the prefilling stage and reduces the KV cache to 4.74% of its original size, resulting in a 6.4x increase in throughput and a 1.4x speedup in inference with only a 1.41% performance loss. Remarkably, in needle-in-a-haystack tasks, UNComp outperforms the full-size KV cache even when compressed to 9.38% of its original size. Our approach offers an efficient, training-free Grouped-Query Attention paradigm that can be seamlessly integrated into existing KV cache schemes.

Title: Combing Text-based and Drag-based Editing for Precise and Flexible Image Editing

• Authors: Ziqi Jiang, Zhen Wang, Long Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03097
• Pdf URL: https://arxiv.org/pdf/2410.03097
• Copy Paste: [[2410.03097]] Combing Text-based and Drag-based Editing for Precise and Flexible Image Editing(https://arxiv.org/abs/2410.03097)
• Keywords: diffusion
• Abstract: Precise and flexible image editing remains a fundamental challenge in computer vision. Based on the modified areas, most editing methods can be divided into two main types: global editing and local editing. In this paper, we choose the two most common editing approaches (ie text-based editing and drag-based editing) and analyze their drawbacks. Specifically, text-based methods often fail to describe the desired modifications precisely, while drag-based methods suffer from ambiguity. To address these issues, we proposed \textbf{CLIPDrag}, a novel image editing method that is the first to combine text and drag signals for precise and ambiguity-free manipulations on diffusion models. To fully leverage these two signals, we treat text signals as global guidance and drag points as local information. Then we introduce a novel global-local motion supervision method to integrate text signals into existing drag-based methods by adapting a pre-trained language-vision model like CLIP. Furthermore, we also address the problem of slow convergence in CLIPDrag by presenting a fast point-tracking method that enforces drag points moving toward correct directions. Extensive experiments demonstrate that CLIPDrag outperforms existing single drag-based methods or text-based methods.

Title: Mamba in Vision: A Comprehensive Survey of Techniques and Applications

• Authors: Md Maklachur Rahman, Abdullah Aman Tutul, Ankur Nath, Lamyanba Laishram, Soon Ki Jung, Tracy Hammond
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03105
• Pdf URL: https://arxiv.org/pdf/2410.03105
• Copy Paste: [[2410.03105]] Mamba in Vision: A Comprehensive Survey of Techniques and Applications(https://arxiv.org/abs/2410.03105)
• Keywords: transformer
• Abstract: Mamba is emerging as a novel approach to overcome the challenges faced by Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in computer vision. While CNNs excel at extracting local features, they often struggle to capture long-range dependencies without complex architectural modifications. In contrast, ViTs effectively model global relationships but suffer from high computational costs due to the quadratic complexity of their self-attention mechanisms. Mamba addresses these limitations by leveraging Selective Structured State Space Models to effectively capture long-range dependencies with linear computational complexity. This survey analyzes the unique contributions, computational benefits, and applications of Mamba models while also identifying challenges and potential future research directions. We provide a foundational resource for advancing the understanding and growth of Mamba models in computer vision. An overview of this work is available at this https URL.

Title: A Training-Free Conditional Diffusion Model for Learning Stochastic Dynamical Systems

• Authors: Yanfang Liu, Yuan Chen, Dongbin Xiu, Guannan Zhang
• Subjects: cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2410.03108
• Pdf URL: https://arxiv.org/pdf/2410.03108
• Copy Paste: [[2410.03108]] A Training-Free Conditional Diffusion Model for Learning Stochastic Dynamical Systems(https://arxiv.org/abs/2410.03108)
• Keywords: diffusion
• Abstract: This study introduces a training-free conditional diffusion model for learning unknown stochastic differential equations (SDEs) using data. The proposed approach addresses key challenges in computational efficiency and accuracy for modeling SDEs by utilizing a score-based diffusion model to approximate their stochastic flow map. Unlike the existing methods, this technique is based on an analytically derived closed-form exact score function, which can be efficiently estimated by Monte Carlo method using the trajectory data, and eliminates the need for neural network training to learn the score function. By generating labeled data through solving the corresponding reverse ordinary differential equation, the approach enables supervised learning of the flow map. Extensive numerical experiments across various SDE types, including linear, nonlinear, and multi-dimensional systems, demonstrate the versatility and effectiveness of the method. The learned models exhibit significant improvements in predicting both short-term and long-term behaviors of unknown stochastic systems, often surpassing baseline methods like GANs in estimating drift and diffusion coefficients.

Title: LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

• Authors: Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, Yelong Shen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03111
• Pdf URL: https://arxiv.org/pdf/2410.03111
• Copy Paste: [[2410.03111]] LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy(https://arxiv.org/abs/2410.03111)
• Keywords: transformer, large language model
• Abstract: The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive large language models (LLMs), enabling faster inference by storing previously computed KV vectors. However, its memory consumption scales linearly with sequence length and batch size, posing a significant bottleneck in LLM deployment. Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific. This paper introduces an orthogonal approach to KV cache compression. We propose a low-rank approximation of KV weight matrices, allowing for plug-in integration with existing transformer-based LLMs without model retraining. To effectively compress KV cache at the weight level, we adjust for layerwise sensitivity and introduce a progressive compression strategy, which is supported by our theoretical analysis on how compression errors accumulate in deep networks. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages. Extensive experiments with LLaMA models ranging from 8B to 70B parameters across various tasks show that our approach significantly reduces the GPU memory footprint while maintaining performance.

Title: X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale

• Authors: Haoran Xu, Kenton Murray, Philipp Koehn, Hieu Hoang, Akiko Eriguchi, Huda Khayrallah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03115
• Pdf URL: https://arxiv.org/pdf/2410.03115
• Copy Paste: [[2410.03115]] X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale(https://arxiv.org/abs/2410.03115)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved remarkable success across various NLP tasks, yet their focus has predominantly been on English due to English-centric pre-training and limited multilingual data. While some multilingual LLMs claim to support for hundreds of languages, models often fail to provide high-quality response for mid- and low-resource languages, leading to imbalanced performance heavily skewed in favor of high-resource languages like English and Chinese. In this paper, we prioritize quality over scaling number of languages, with a focus on multilingual machine translation task, and introduce X-ALMA, a model designed with a commitment to ensuring top-tier performance across 50 diverse languages, regardless of their resource levels. X-ALMA surpasses state-of-the-art open-source multilingual LLMs, such as Aya-101 and Aya-23, in every single translation direction on the FLORES and WMT'23 test datasets according to COMET-22. This is achieved by plug-and-play language-specific module architecture to prevent language conflicts during training and a carefully designed training regimen with novel optimization methods to maximize the translation performance. At the final stage of training regimen, our proposed Adaptive Rejection Preference Optimization (ARPO) surpasses existing preference optimization methods in translation tasks.

Title: Precision, Stability, and Generalization: A Comprehensive Assessment of RNNs learnability capability for Classifying Counter and Dyck Languages

• Authors: Neisarg Dave, Daniel Kifer, Lee Giles, Ankur Mali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03118
• Pdf URL: https://arxiv.org/pdf/2410.03118
• Copy Paste: [[2410.03118]] Precision, Stability, and Generalization: A Comprehensive Assessment of RNNs learnability capability for Classifying Counter and Dyck Languages(https://arxiv.org/abs/2410.03118)
• Keywords: robust
• Abstract: This study investigates the learnability of Recurrent Neural Networks (RNNs) in classifying structured formal languages, focusing on counter and Dyck languages. Traditionally, both first-order (LSTM) and second-order (O2RNN) RNNs have been considered effective for such tasks, primarily based on their theoretical expressiveness within the Chomsky hierarchy. However, our research challenges this notion by demonstrating that RNNs primarily operate as state machines, where their linguistic capabilities are heavily influenced by the precision of their embeddings and the strategies used for sampling negative examples. Our experiments revealed that performance declines significantly as the structural similarity between positive and negative examples increases. Remarkably, even a basic single-layer classifier using RNN embeddings performed better than chance. To evaluate generalization, we trained models on strings up to a length of 40 and tested them on strings from lengths 41 to 500, using 10 unique seeds to ensure statistical robustness. Stability comparisons between LSTM and O2RNN models showed that O2RNNs generally offer greater stability across various scenarios. We further explore the impact of different initialization strategies revealing that our hypothesis is consistent with various RNNs. Overall, this research questions established beliefs about RNNs' computational capabilities, highlighting the importance of data structure and sampling techniques in assessing neural networks' potential for language classification tasks. It emphasizes that stronger constraints on expressivity are crucial for understanding true learnability, as mere expressivity does not capture the essence of learning.

Title: RIPPLECOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning

• Authors: Zihao Zhao, Yuchen Yang, Yijiang Li, Yinzhi Cao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03122
• Pdf URL: https://arxiv.org/pdf/2410.03122
• Copy Paste: [[2410.03122]] RIPPLECOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning(https://arxiv.org/abs/2410.03122)
• Keywords: large language model
• Abstract: The ripple effect poses a significant challenge in knowledge editing for large language models. Namely, when a single fact is edited, the model struggles to accurately update the related facts in a sequence, which is evaluated by multi-hop questions linked to a chain of related facts. Recent strategies have moved away from traditional parameter updates to more flexible, less computation-intensive methods, proven to be more effective in addressing the ripple effect. In-context learning (ICL) editing uses a simple demonstration `Imagine that + new fact` to guide LLMs, but struggles with complex multi-hop questions as the new fact alone fails to specify the chain of facts involved in such scenarios. Besides, memory-based editing maintains additional storage for all edits and related facts, requiring continuous updates to stay effective. As a result of these design limitations, the challenge remains, with the highest accuracy being only 33.8% on the MQuAKE-cf benchmarks for Vicuna-7B. To address this, we propose RippleCOT, a novel ICL editing approach integrating Chain-of-Thought (COT) reasoning. RippleCOT structures demonstrations as `newfact, question, thought, answer`, incorporating a thought component to identify and decompose the multi-hop logic within questions. This approach effectively guides the model through complex multi-hop questions with chains of related facts. Comprehensive experiments demonstrate that RippleCOT significantly outperforms the state-of-the-art on the ripple effect, achieving accuracy gains ranging from 7.8% to 87.1%.

Title: On Unsupervised Prompt Learning for Classification with Black-box Language Models

• Authors: Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao, Gang Niu, Masashi Sugiyama
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03124
• Pdf URL: https://arxiv.org/pdf/2410.03124
• Copy Paste: [[2410.03124]] On Unsupervised Prompt Learning for Classification with Black-box Language Models(https://arxiv.org/abs/2410.03124)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved impressive success in text-formatted learning problems, and most popular LLMs have been deployed in a black-box fashion. Meanwhile, fine-tuning is usually necessary for a specific downstream task to obtain better performance, and this functionality is provided by the owners of the black-box LLMs. To fine-tune a black-box LLM, labeled data are always required to adjust the model parameters. However, in many real-world applications, LLMs can label textual datasets with even better quality than skilled human annotators, motivating us to explore the possibility of fine-tuning black-box LLMs with unlabeled data. In this paper, we propose unsupervised prompt learning for classification with black-box LLMs, where the learning parameters are the prompt itself and the pseudo labels of unlabeled data. Specifically, the prompt is modeled as a sequence of discrete tokens, and every token has its own to-be-learned categorical distribution. On the other hand, for learning the pseudo labels, we are the first to consider the in-context learning (ICL) capabilities of LLMs: we first identify reliable pseudo-labeled data using the LLM, and then assign pseudo labels to other unlabeled data based on the prompt, allowing the pseudo-labeled data to serve as in-context demonstrations alongside the prompt. Those in-context demonstrations matter: previously, they are involved when the prompt is used for prediction while they are not involved when the prompt is trained; thus, taking them into account during training makes the prompt-learning and prompt-using stages more consistent. Experiments on benchmark datasets show the effectiveness of our proposed algorithm. After unsupervised prompt learning, we can use the pseudo-labeled dataset for further fine-tuning by the owners of the black-box LLMs.

Title: ARB-LLM: Alternating Refined Binarizations for Large Language Models

• Authors: Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, zhongchao shi, Linghe Kong, Yulun Zhang, Xiaokang Yang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03129
• Pdf URL: https://arxiv.org/pdf/2410.03129
• Copy Paste: [[2410.03129]] ARB-LLM: Alternating Refined Binarizations for Large Language Models(https://arxiv.org/abs/2410.03129)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_\text{X}$ and ARB-LLM$_\text{RC}$ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$_\text{RC}$ is the first to surpass FP16 models of the same size. The code and models will be available at this https URL.

Title: Remaining Useful Life Prediction: A Study on Multidimensional Industrial Signal Processing and Efficient Transfer Learning Based on Large Language Models

• Authors: Yan Chen, Cheng Liu
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.03134
• Pdf URL: https://arxiv.org/pdf/2410.03134
• Copy Paste: [[2410.03134]] Remaining Useful Life Prediction: A Study on Multidimensional Industrial Signal Processing and Efficient Transfer Learning Based on Large Language Models(https://arxiv.org/abs/2410.03134)
• Keywords: large language model
• Abstract: Remaining useful life (RUL) prediction is crucial for maintaining modern industrial systems, where equipment reliability and operational safety are paramount. Traditional methods, based on small-scale deep learning or physical/statistical models, often struggle with complex, multidimensional sensor data and varying operating conditions, limiting their generalization capabilities. To address these challenges, this paper introduces an innovative regression framework utilizing large language models (LLMs) for RUL prediction. By leveraging the modeling power of LLMs pre-trained on corpus data, the proposed model can effectively capture complex temporal dependencies and improve prediction accuracy. Extensive experiments on the Turbofan engine's RUL prediction task show that the proposed model surpasses state-of-the-art (SOTA) methods on the challenging FD002 and FD004 subsets and achieves near-SOTA results on the other subsets. Notably, different from previous research, our framework uses the same sliding window length and all sensor signals for all subsets, demonstrating strong consistency and generalization. Moreover, transfer learning experiments reveal that with minimal target domain data for fine-tuning, the model outperforms SOTA methods trained on full target domain data. This research highlights the significant potential of LLMs in industrial signal processing and RUL prediction, offering a forward-looking solution for health management in future intelligent industrial systems.

Title: Deliberate Reasoning for LLMs as Structure-aware Planning with Accurate World Model

• Authors: Siheng Xiong, Ali Payani, Yuan Yang, Faramarz Fekri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03136
• Pdf URL: https://arxiv.org/pdf/2410.03136
• Copy Paste: [[2410.03136]] Deliberate Reasoning for LLMs as Structure-aware Planning with Accurate World Model(https://arxiv.org/abs/2410.03136)
• Keywords: large language model
• Abstract: Enhancing the reasoning capabilities of large language models (LLMs) remains a key challenge, especially for tasks that require complex, multi-step decision-making. Humans excel at these tasks by leveraging deliberate planning with an internal world model to simulate the potential outcomes of various actions. Inspired by this, we propose a novel multi-step reasoning framework for LLMs, referred to as Structure-aware Planning with Accurate World Model (SWAP). Unlike previous approaches that rely solely on Chain-of-Thought (CoT) reasoning in natural language, SWAP incorporates structural information to guide the reasoning process via a world model and provides a soft verification mechanism over the steps. Moreover, SWAP overcomes the challenge of accurate world state predictions in complex reasoning tasks by introducing a Generator-Discriminator architecture, which enables more reliable world modeling. Specifically, the generator predicts the next state, and the discriminator ensures alignment with the logical consistency required by the problem context. SWAP also encourages the policy model to explore a broad range of potential actions to prevent premature convergence. By resolving the bottlenecks of generation diversity for both actions and states using diversity-based modeling (DBM) and improving discrimination accuracy through contrastive ranking (CR), SWAP significantly enhances the reasoning performance of LLMs. We evaluate SWAP across diverse reasoning-intensive benchmarks including math reasoning, logical reasoning, and coding tasks. Extensive experiments demonstrate that SWAP achieves substantial improvements over the baselines and consistently outperforms existing LLMs of similar sizes.

Title: SAG: Style-Aligned Article Generation via Model Collaboration

• Authors: Chenning Xu, Fangxun Shu, Dian Jin, Jinghao Wei, Hao Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03137
• Pdf URL: https://arxiv.org/pdf/2410.03137
• Copy Paste: [[2410.03137]] SAG: Style-Aligned Article Generation via Model Collaboration(https://arxiv.org/abs/2410.03137)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have increased the demand for personalized and stylish content generation. However, closed-source models like GPT-4 present limitations in optimization opportunities, while the substantial training costs and inflexibility of open-source alternatives, such as Qwen-72B, pose considerable challenges. Conversely, small language models (SLMs) struggle with understanding complex instructions and transferring learned capabilities to new contexts, often exhibiting more pronounced limitations. In this paper, we present a novel collaborative training framework that leverages the strengths of both LLMs and SLMs for style article generation, surpassing the performance of either model alone. We freeze the LLMs to harness their robust instruction-following capabilities and subsequently apply supervised fine-tuning on the SLM using style-specific data. Additionally, we introduce a self-improvement method to enhance style consistency. Our new benchmark, NoteBench, thoroughly evaluates style-aligned generation. Extensive experiments show that our approach achieves state-of-the-art performance, with improvements of 0.78 in ROUGE-L and 0.55 in BLEU-4 scores compared to GPT-4, while maintaining a low hallucination rate regarding factual and faithfulness.

Title: Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity

• Authors: Hyosoon Jang, Yunhui Jang, Jaehyung Kim, Sungsoo Ahn
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.03138
• Pdf URL: https://arxiv.org/pdf/2410.03138
• Copy Paste: [[2410.03138]] Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity(https://arxiv.org/abs/2410.03138)
• Keywords: generative, large language model
• Abstract: Recent advancements in large language models (LLMs) have demonstrated impressive performance in generating molecular structures as drug candidates, which offers significant potential to accelerate drug discovery. However, the current LLMs overlook a critical requirement for drug discovery: proposing a diverse set of molecules. This diversity is essential for improving the chances of finding a viable drug, as it provides alternative molecules that may succeed where others fail in wet-lab or clinical validations. Despite such a need for diversity, the LLMs often output structurally similar molecules from a given prompt. While decoding schemes like beam search may enhance textual diversity, this often does not align with molecular structural diversity. In response, we propose a new method for fine-tuning molecular generative LLMs to autoregressively generate a set of structurally diverse molecules, where each molecule is generated by conditioning on the previously generated molecules. Our approach consists of two stages: (1) supervised fine-tuning to adapt LLMs to autoregressively generate molecules in a sequence and (2) reinforcement learning to maximize structural diversity within the generated molecules. Our experiments show that (1) our fine-tuning approach enables the LLMs to better discover diverse molecules compared to existing decoding schemes and (2) our fine-tuned model outperforms other representative LLMs in generating diverse molecules, including the ones fine-tuned on chemical domains.

Title: In-context Learning in Presence of Spurious Correlations

• Authors: Hrayr Harutyunyan, Rafayel Darbinyan, Samvel Karapetyan, Hrant Khachatrian
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03140
• Pdf URL: https://arxiv.org/pdf/2410.03140
• Copy Paste: [[2410.03140]] In-context Learning in Presence of Spurious Correlations(https://arxiv.org/abs/2410.03140)
• Keywords: transformer, large language model
• Abstract: Large language models exhibit a remarkable capacity for in-context learning, where they learn to solve tasks given a few examples. Recent work has shown that transformers can be trained to perform simple regression tasks in-context. This work explores the possibility of training an in-context learner for classification tasks involving spurious features. We find that the conventional approach of training in-context learners is susceptible to spurious features. Moreover, when the meta-training dataset includes instances of only one task, the conventional approach leads to task memorization and fails to produce a model that leverages context for predictions. Based on these observations, we propose a novel technique to train such a learner for a given classification task. Remarkably, this in-context learner matches and sometimes outperforms strong methods like ERM and GroupDRO. However, unlike these algorithms, it does not generalize well to other tasks. We show that it is possible to obtain an in-context learner that generalizes to unseen tasks by training on a diverse dataset of synthetic in-context learning instances.

Title: Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

• Authors: Kyuyoung Kim, Ah Jeong Seo, Hao Liu, Jinwoo Shin, Kimin Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03145
• Pdf URL: https://arxiv.org/pdf/2410.03145
• Copy Paste: [[2410.03145]] Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback(https://arxiv.org/abs/2410.03145)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) fine-tuned with alignment techniques, such as reinforcement learning from human feedback, have been instrumental in developing some of the most capable AI systems to date. Despite their success, existing methods typically rely on simple binary labels, such as those indicating preferred outputs in pairwise preferences, which fail to capture the subtle differences in relative quality between pairs. To address this limitation, we introduce an approach called Margin Matching Preference Optimization (MMPO), which incorporates relative quality margins into optimization, leading to improved LLM policies and reward models. Specifically, given quality margins in pairwise preferences, we design soft target probabilities based on the Bradley-Terry model, which are then used to train models with the standard cross-entropy objective. Experiments with both human and AI feedback data demonstrate that MMPO consistently outperforms baseline methods, often by a substantial margin, on popular benchmarks including MT-bench and RewardBench. Notably, the 7B model trained with MMPO achieves state-of-the-art performance on RewardBench as of June 2024, outperforming other models of the same scale. Our analysis also shows that MMPO is more robust to overfitting, leading to better-calibrated models.

Title: Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insights

• Authors: Shrabon Das, Ankur Mali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03154
• Pdf URL: https://arxiv.org/pdf/2410.03154
• Copy Paste: [[2410.03154]] Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insights(https://arxiv.org/abs/2410.03154)
• Keywords: robust
• Abstract: This study explores the learnability of memory-less and memory-augmented RNNs, which are theoretically equivalent to Pushdown Automata. Empirical results show that these models often fail to generalize on longer sequences, relying more on precision than mastering symbolic grammar. Experiments on fully trained and component-frozen models reveal that freezing the memory component significantly improves performance, achieving state-of-the-art results on the Penn Treebank dataset (test perplexity reduced from 123.5 to 120.5). Models with frozen memory retained up to 90% of initial performance on longer sequences, compared to a 60% drop in standard models. Theoretical analysis suggests that freezing memory stabilizes temporal dependencies, leading to robust convergence. These findings stress the need for stable memory designs and long-sequence evaluations to understand RNNs true learnability limits.

Title: MELODI: Exploring Memory Compression for Long Contexts

• Authors: Yinpeng Chen, DeLesley Hutchins, Aren Jansen, Andrey Zhmoginov, David Racz, Jesper Andersen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03156
• Pdf URL: https://arxiv.org/pdf/2410.03156
• Copy Paste: [[2410.03156]] MELODI: Exploring Memory Compression for Long Contexts(https://arxiv.org/abs/2410.03156)
• Keywords: transformer
• Abstract: We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8.

Title: Autoregressive Moving-average Attention Mechanism for Time Series Forecasting

• Authors: Jiecheng Lu, Xu Han, Yan Sun, Shihao Yang
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.03159
• Pdf URL: https://arxiv.org/pdf/2410.03159
• Copy Paste: [[2410.03159]] Autoregressive Moving-average Attention Mechanism for Time Series Forecasting(https://arxiv.org/abs/2410.03159)
• Keywords: transformer
• Abstract: We propose an Autoregressive (AR) Moving-average (MA) attention structure that can adapt to various linear attention mechanisms, enhancing their ability to capture long-range and local temporal patterns in time series. In this paper, we first demonstrate that, for the time series forecasting (TSF) task, the previously overlooked decoder-only autoregressive Transformer model can achieve results comparable to the best baselines when appropriate tokenization and training methods are applied. Moreover, inspired by the ARMA model from statistics and recent advances in linear attention, we introduce the full ARMA structure into existing autoregressive attention mechanisms. By using an indirect MA weight generation method, we incorporate the MA term while maintaining the time complexity and parameter size of the underlying efficient attention models. We further explore how indirect parameter generation can produce implicit MA weights that align with the modeling requirements for local temporal impacts. Experimental results show that incorporating the ARMA structure consistently improves the performance of various AR attentions on TSF tasks, achieving state-of-the-art results.

Title: Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach

• Authors: Yaofang Liu, Yumeng Ren, Xiaodong Cun, Aitor Artola, Yang Liu, Tieyong Zeng, Raymond H. Chan, Jean-michel Morel
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03160
• Pdf URL: https://arxiv.org/pdf/2410.03160
• Copy Paste: [[2410.03160]] Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach(https://arxiv.org/abs/2410.03160)
• Keywords: robust, diffusion, generative
• Abstract: Diffusion models have revolutionized image generation, and their extension to video generation has shown promise. However, current video diffusion models~(VDMs) rely on a scalar timestep variable applied at the clip level, which limits their ability to model complex temporal dependencies needed for various tasks like image-to-video generation. To address this limitation, we propose a frame-aware video diffusion model~(FVDM), which introduces a novel vectorized timestep variable~(VTV). Unlike conventional VDMs, our approach allows each frame to follow an independent noise schedule, enhancing the model's capacity to capture fine-grained temporal dependencies. FVDM's flexibility is demonstrated across multiple tasks, including standard video generation, image-to-video generation, video interpolation, and long video synthesis. Through a diverse set of VTV configurations, we achieve superior quality in generated videos, overcoming challenges such as catastrophic forgetting during fine-tuning and limited generalizability in zero-shot this http URL empirical evaluations show that FVDM outperforms state-of-the-art methods in video generation quality, while also excelling in extended tasks. By addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm in video synthesis, offering a robust framework with significant implications for generative modeling and multimedia applications.

Title: Can Watermarked LLMs be Identified by Users via Crafted Prompts?

• Authors: Aiwei Liu, Sheng Guan, Yiming Liu, Leyi Pan, Yifei Zhang, Liancheng Fang, Lijie Wen, Philip S. Yu, Xuming Hu
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03168
• Pdf URL: https://arxiv.org/pdf/2410.03168
• Copy Paste: [[2410.03168]] Can Watermarked LLMs be Identified by Users via Crafted Prompts?(https://arxiv.org/abs/2410.03168)
• Keywords: attack, robust, watermark, large language model
• Abstract: Text watermarking for Large Language Models (LLMs) has made significant progress in detecting LLM outputs and preventing misuse. Current watermarking techniques offer high detectability, minimal impact on text quality, and robustness to text editing. However, current researches lack investigation into the imperceptibility of watermarking techniques in LLM services. This is crucial as LLM providers may not want to disclose the presence of watermarks in real-world scenarios, as it could reduce user willingness to use the service and make watermarks more vulnerable to attacks. This work is the first to investigate the imperceptibility of watermarked LLMs. We design an identification algorithm called Water-Probe that detects watermarks through well-designed prompts to the LLM. Our key motivation is that current watermarked LLMs expose consistent biases under the same watermark key, resulting in similar differences across prompts under different watermark keys. Experiments show that almost all mainstream watermarking algorithms are easily identified with our well-designed prompts, while Water-Probe demonstrates a minimal false positive rate for non-watermarked LLMs. Finally, we propose that the key to enhancing the imperceptibility of watermarked LLMs is to increase the randomness of watermark key selection. Based on this, we introduce the Water-Bag strategy, which significantly improves watermark imperceptibility by merging multiple watermark keys.

Title: Autoregressive Large Language Models are Computationally Universal

• Authors: Dale Schuurmans, Hanjun Dai, Francesco Zanini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03170
• Pdf URL: https://arxiv.org/pdf/2410.03170
• Copy Paste: [[2410.03170]] Autoregressive Large Language Models are Computationally Universal(https://arxiv.org/abs/2410.03170)
• Keywords: transformer, large language model
• Abstract: We show that autoregressive decoding of a transformer-based language model can realize universal computation, without external intervention or modification of the model's weights. Establishing this result requires understanding how a language model can process arbitrarily long inputs using a bounded context. For this purpose, we consider a generalization of autoregressive decoding where, given a long input, emitted tokens are appended to the end of the sequence as the context window advances. We first show that the resulting system corresponds to a classical model of computation, a Lag system, that has long been known to be computationally universal. By leveraging a new proof, we show that a universal Turing machine can be simulated by a Lag system with 2027 production rules. We then investigate whether an existing large language model can simulate the behaviour of such a universal Lag system. We give an affirmative answer by showing that a single system-prompt can be developed for gemini-1.5-pro-001 that drives the model, under deterministic (greedy) decoding, to correctly apply each of the 2027 production rules. We conclude that, by the Church-Turing thesis, prompted gemini-1.5-pro-001 with extended autoregressive (greedy) decoding is a general purpose computer.

Title: Selective Transformer for Hyperspectral Image Classification

• Authors: Yichu Xu, Di Wang, Lefei Zhang, Liangpei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03171
• Pdf URL: https://arxiv.org/pdf/2410.03171
• Copy Paste: [[2410.03171]] Selective Transformer for Hyperspectral Image Classification(https://arxiv.org/abs/2410.03171)
• Keywords: transformer
• Abstract: Transformer has achieved satisfactory results in the field of hyperspectral image (HSI) classification. However, existing Transformer models face two key challenges when dealing with HSI scenes characterized by diverse land cover types and rich spectral information: (1) fixed receptive field representation overlooks effective contextual information; (2) redundant self-attention feature representation. To address these limitations, we propose a novel Selective Transformer (SFormer) for HSI classification. The SFormer is designed to dynamically select receptive fields for capturing both spatial and spectral contextual information, while mitigating the impact of redundant data by prioritizing the most relevant features. This enables a highly accurate classification of the land covers of the HSI. Specifically, a Kernel Selective Transformer Block (KSTB) is first utilized to dynamically select an appropriate receptive field range to effectively extract spatial-spectral features. Furthermore, to capture the most crucial tokens, a Token Selective Transformer Block (TSTB) is introduced, which selects the most relevant tokens based on the ranking of attention scores for each query. Extensive experiments on four benchmark HSI datasets demonstrate that the proposed SFormer outperforms the state-of-the-art HSI classification models. The codes will be released.

Title: Rapid optimization in high dimensional space by deep kernel learning augmented genetic algorithms

• Authors: Mani Valleti, Aditya Raghavan, Sergei V. Kalinin
• Subjects: cs.LG, cond-mat.mtrl-sci, physics.comp-ph, physics.data-an
• Abstract URL: https://arxiv.org/abs/2410.03173
• Pdf URL: https://arxiv.org/pdf/2410.03173
• Copy Paste: [[2410.03173]] Rapid optimization in high dimensional space by deep kernel learning augmented genetic algorithms(https://arxiv.org/abs/2410.03173)
• Keywords: generative
• Abstract: Exploration of complex high-dimensional spaces presents significant challenges in fields such as molecular discovery, process optimization, and supply chain management. Genetic Algorithms (GAs), while offering significant power for creating new candidate spaces, often entail high computational demands due to the need for evaluation of each new proposed solution. On the other hand, Deep Kernel Learning (DKL) efficiently navigates the spaces of preselected candidate structures but lacks generative capabilities. This study introduces an approach that amalgamates the generative power of GAs to create new candidates with the efficiency of DKL-based surrogate models to rapidly ascertain the behavior of new candidate spaces. This DKL-GA framework can be further used to build Bayesian Optimization (BO) workflows. We demonstrate the effectiveness of this approach through the optimization of the FerroSIM model, showcasing its broad applicability to diverse challenges, including molecular discovery and battery charging optimization.

Title: HRVMamba: High-Resolution Visual State Space Model for Dense Prediction

• Authors: Hao Zhang, Yongqiang Ma, Wenqi Shao, Ping Luo, Nanning Zheng, Kaipeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03174
• Pdf URL: https://arxiv.org/pdf/2410.03174
• Copy Paste: [[2410.03174]] HRVMamba: High-Resolution Visual State Space Model for Dense Prediction(https://arxiv.org/abs/2410.03174)
• Keywords: segmentation
• Abstract: Recently, State Space Models (SSMs) with efficient hardware-aware designs, i.e., Mamba, have demonstrated significant potential in computer vision tasks due to their linear computational complexity with respect to token length and their global receptive field. However, Mamba's performance on dense prediction tasks, including human pose estimation and semantic segmentation, has been constrained by three key challenges: insufficient inductive bias, long-range forgetting, and low-resolution output representation. To address these challenges, we introduce the Dynamic Visual State Space (DVSS) block, which utilizes multi-scale convolutional kernels to extract local features across different scales and enhance inductive bias, and employs deformable convolution to mitigate the long-range forgetting problem while enabling adaptive spatial aggregation based on input and task-specific information. By leveraging the multi-resolution parallel design proposed in HRNet, we introduce High-Resolution Visual State Space Model (HRVMamba) based on the DVSS block, which preserves high-resolution representations throughout the entire process while promoting effective multi-scale feature learning. Extensive experiments highlight HRVMamba's impressive performance on dense prediction tasks, achieving competitive results against existing benchmark models without bells and whistles. Code is available at this https URL.

Title: Kiss up, Kick down: Exploring Behavioral Changes in Multi-modal Large Language Models with Assigned Visual Personas

• Authors: Seungjong Sun, Eungu Lee, Seo Yeon Baek, Seunghyun Hwang, Wonbyung Lee, Dongyan Nan, Bernard J. Jansen, Jang Hyun Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03181
• Pdf URL: https://arxiv.org/pdf/2410.03181
• Copy Paste: [[2410.03181]] Kiss up, Kick down: Exploring Behavioral Changes in Multi-modal Large Language Models with Assigned Visual Personas(https://arxiv.org/abs/2410.03181)
• Keywords: large language model
• Abstract: This study is the first to explore whether multi-modal large language models (LLMs) can align their behaviors with visual personas, addressing a significant gap in the literature that predominantly focuses on text-based personas. We developed a novel dataset of 5K fictional avatar images for assignment as visual personas to LLMs, and analyzed their negotiation behaviors based on the visual traits depicted in these images, with a particular focus on aggressiveness. The results indicate that LLMs assess the aggressiveness of images in a manner similar to humans and output more aggressive negotiation behaviors when prompted with an aggressive visual persona. Interestingly, the LLM exhibited more aggressive negotiation behaviors when the opponent's image appeared less aggressive than their own, and less aggressive behaviors when the opponents image appeared more aggressive.

Title: Generating bilingual example sentences with large language models as lexicography assistants

• Authors: Raphael Merx, Ekaterina Vylomova, Kemal Kurniawan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03182
• Pdf URL: https://arxiv.org/pdf/2410.03182
• Copy Paste: [[2410.03182]] Generating bilingual example sentences with large language models as lexicography assistants(https://arxiv.org/abs/2410.03182)
• Keywords: large language model
• Abstract: We present a study of LLMs' performance in generating and rating example sentences for bilingual dictionaries across languages with varying resource levels: French (high-resource), Indonesian (mid-resource), and Tetun (low-resource), with English as the target language. We evaluate the quality of LLM-generated examples against the GDEX (Good Dictionary EXample) criteria: typicality, informativeness, and intelligibility. Our findings reveal that while LLMs can generate reasonably good dictionary examples, their performance degrades significantly for lower-resourced languages. We also observe high variability in human preferences for example quality, reflected in low inter-annotator agreement rates. To address this, we demonstrate that in-context learning can successfully align LLMs with individual annotator preferences. Additionally, we explore the use of pre-trained language models for automated rating of examples, finding that sentence perplexity serves as a good proxy for typicality and intelligibility in higher-resourced languages. Our study also contributes a novel dataset of 600 ratings for LLM-generated sentence pairs, and provides insights into the potential of LLMs in reducing the cost of lexicographic work, particularly for low-resource languages.

Title: Research Directions for Verifiable Crypto-Physically Secure TEEs

• Authors: Sylvain Bellemare
• Subjects: cs.CR, cs.AR, cs.ET
• Abstract URL: https://arxiv.org/abs/2410.03183
• Pdf URL: https://arxiv.org/pdf/2410.03183
• Copy Paste: [[2410.03183]] Research Directions for Verifiable Crypto-Physically Secure TEEs(https://arxiv.org/abs/2410.03183)
• Keywords: secure, attack
• Abstract: A niche corner of the Web3 world is increasingly making use of hardware-based Trusted Execution Environments (TEEs) to build decentralized infrastructure. One of the motivations to use TEEs is to go beyond the current performance limitations of cryptography-based alternatives such as zero-knowledge proofs (ZKP), fully homomorphic encryption (FHE), and multi-party computation (MPC). Despite their appealing advantages, current TEEs suffer from serious limitations as they are not secure against physical attacks, and their attestation mechanism is rooted in the chip manufacturer's trust. As a result, Web3 applications have to rely on cloud infrastruture to act as trusted guardians of hardware-based TEEs and have to accept to trust chip manufacturers. This work aims at exploring how we could potentially architect and implement chips that would be secure against physical attacks and would not require putting trust in chip manufacturers. One goal of this work is to motivate the Web3 movement to acknowledge and leverage the substantial amount of relevant hardware research that already exists. In brief, a combination of: (1) physical unclonable functions (PUFs) to secure the root-of-trust; (2) masking and redundancy techniques to secure computations; (3) open source hardware and imaging techniques to verify that a chip matches its expected design; can help move towards attesting that a given TEE can be trusted without the need to trust a cloud provider and a chip manufacturer.

Title: EXAQ: Exponent Aware Quantization For LLMs Acceleration

• Authors: Moran Shkolnik, Maxim Fishman, Brian Chmiel, Hilla Ben-Yaacov, Ron Banner, Kfir Yehuda Levy
• Subjects: cs.LG, cs.AI, cs.PF
• Abstract URL: https://arxiv.org/abs/2410.03185
• Pdf URL: https://arxiv.org/pdf/2410.03185
• Copy Paste: [[2410.03185]] EXAQ: Exponent Aware Quantization For LLMs Acceleration(https://arxiv.org/abs/2410.03185)
• Keywords: large language model
• Abstract: Quantization has established itself as the primary approach for decreasing the computational and storage expenses associated with Large Language Models (LLMs) inference. The majority of current research emphasizes quantizing weights and activations to enable low-bit general-matrix-multiply (GEMM) operations, with the remaining non-linear operations executed at higher precision. In our study, we discovered that following the application of these techniques, the primary bottleneck in LLMs inference lies in the softmax layer. The softmax operation comprises three phases: exponent calculation, accumulation, and normalization, Our work focuses on optimizing the first two phases. We propose an analytical approach to determine the optimal clipping value for the input to the softmax function, enabling sub-4-bit quantization for LLMs inference. This method accelerates the calculations of both $e^x$ and $\sum(e^x)$ with minimal to no accuracy degradation. For example, in LLaMA1-30B, we achieve baseline performance with 2-bit quantization on the well-known "Physical Interaction: Question Answering" (PIQA) dataset evaluation. This ultra-low bit quantization allows, for the first time, an acceleration of approximately 4x in the accumulation phase. The combination of accelerating both $e^x$ and $\sum(e^x)$ results in a 36.9% acceleration in the softmax operation.

Title: Autonomous Character-Scene Interaction Synthesis from Text Instruction

• Authors: Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, Yixin Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03187
• Pdf URL: https://arxiv.org/pdf/2410.03187
• Copy Paste: [[2410.03187]] Autonomous Character-Scene Interaction Synthesis from Text Instruction(https://arxiv.org/abs/2410.03187)
• Keywords: diffusion
• Abstract: Synthesizing human motions in 3D environments, particularly those with complex activities such as locomotion, hand-reaching, and human-object interaction, presents substantial demands for user-defined waypoints and stage transitions. These requirements pose challenges for current models, leading to a notable gap in automating the animation of characters from simple human inputs. This paper addresses this challenge by introducing a comprehensive framework for synthesizing multi-stage scene-aware interaction motions directly from a single text instruction and goal location. Our approach employs an auto-regressive diffusion model to synthesize the next motion segment, along with an autonomous scheduler predicting the transition for each action stage. To ensure that the synthesized motions are seamlessly integrated within the environment, we propose a scene representation that considers the local perception both at the start and the goal location. We further enhance the coherence of the generated motion by integrating frame embeddings with language input. Additionally, to support model training, we present a comprehensive motion-captured dataset comprising 16 hours of motion sequences in 120 indoor scenes covering 40 types of motions, each annotated with precise language descriptions. Experimental results demonstrate the efficacy of our method in generating high-quality, multi-stage motions closely aligned with environmental and textual conditions.

Title: Looking into Concept Explanation Methods for Diabetic Retinopathy Classification

• Authors: Andrea M. Storås, Josefine V. Sundgaard
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03188
• Pdf URL: https://arxiv.org/pdf/2410.03188
• Copy Paste: [[2410.03188]] Looking into Concept Explanation Methods for Diabetic Retinopathy Classification(https://arxiv.org/abs/2410.03188)
• Keywords: interpretability
• Abstract: Diabetic retinopathy is a common complication of diabetes, and monitoring the progression of retinal abnormalities using fundus imaging is crucial. Because the images must be interpreted by a medical expert, it is infeasible to screen all individuals with diabetes for diabetic retinopathy. Deep learning has shown impressive results for automatic analysis and grading of fundus images. One drawback is, however, the lack of interpretability, which hampers the implementation of such systems in the clinic. Explainable artificial intelligence methods can be applied to explain the deep neural networks. Explanations based on concepts have shown to be intuitive for humans to understand, but have not yet been explored in detail for diabetic retinopathy grading. This work investigates and compares two concept-based explanation techniques for explaining deep neural networks developed for automatic diagnosis of diabetic retinopathy: Quantitative Testing with Concept Activation Vectors and Concept Bottleneck Models. We found that both methods have strengths and weaknesses, and choice of method should take the available data and the end user's preferences into account.

Title: Generalizable Prompt Tuning for Vision-Language Models

• Authors: Qian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03189
• Pdf URL: https://arxiv.org/pdf/2410.03189
• Copy Paste: [[2410.03189]] Generalizable Prompt Tuning for Vision-Language Models(https://arxiv.org/abs/2410.03189)
• Keywords: robust
• Abstract: Prompt tuning for vision-language models such as CLIP involves optimizing the text prompts used to generate image-text pairs for specific downstream tasks. While hand-crafted or template-based prompts are generally applicable to a wider range of unseen classes, they tend to perform poorly in downstream tasks (i.e., seen classes). Learnable soft prompts, on the other hand, often perform well in downstream tasks but lack generalizability. Additionally, prior research has predominantly concentrated on the textual modality, with very few studies attempting to explore the prompt's generalization potential from the visual modality. Keeping these limitations in mind, we investigate how to prompt tuning to obtain both a competitive downstream performance and generalization. The study shows that by treating soft and hand-crafted prompts as dual views of the textual modality, and maximizing their mutual information, we can better ensemble task-specific and general semantic information. Moreover, to generate more expressive prompts, the study introduces a class-wise augmentation from the visual modality, resulting in significant robustness to a wider range of unseen classes. Extensive evaluations on several benchmarks report that the proposed approach achieves competitive results in terms of both task-specific performance and general abilities.

Title: Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization

• Authors: Zichen Miao, Zhengyuan Yang, Kevin Lin, Ze Wang, Zicheng Liu, Lijuan Wang, Qiang Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03190
• Pdf URL: https://arxiv.org/pdf/2410.03190
• Copy Paste: [[2410.03190]] Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization(https://arxiv.org/abs/2410.03190)
• Keywords: diffusion
• Abstract: Recent advancements in timestep-distilled diffusion models have enabled high-quality image generation that rivals non-distilled multi-step models, but with significantly fewer inference steps. While such models are attractive for applications due to the low inference cost and latency, fine-tuning them with a naive diffusion objective would result in degraded and blurry outputs. An intuitive alternative is to repeat the diffusion distillation process with a fine-tuned teacher model, which produces good results but is cumbersome and computationally intensive; the distillation training usually requires magnitude higher of training compute compared to fine-tuning for specific image styles. In this paper, we present an algorithm named pairwise sample optimization (PSO), which enables the direct fine-tuning of an arbitrary timestep-distilled diffusion model. PSO introduces additional reference images sampled from the current time-step distilled model, and increases the relative likelihood margin between the training images and reference images. This enables the model to retain its few-step generation ability, while allowing for fine-tuning of its output distribution. We also demonstrate that PSO is a generalized formulation which can be flexibly extended to both offline-sampled and online-sampled pairwise data, covering various popular objectives for diffusion model preference optimization. We evaluate PSO in both preference optimization and other fine-tuning tasks, including style transfer and concept customization. We show that PSO can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data. PSO also demonstrates effectiveness in style transfer and concept customization by directly tuning timestep-distilled diffusion models.

Title: Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languages

• Authors: Seonjeong Hwang, Yunsu Kim, Gary Geunbae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03197
• Pdf URL: https://arxiv.org/pdf/2410.03197
• Copy Paste: [[2410.03197]] Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languages(https://arxiv.org/abs/2410.03197)
• Keywords: large language model
• Abstract: Automatic question generation (QG) serves a wide range of purposes, such as augmenting question-answering (QA) corpora, enhancing chatbot systems, and developing educational materials. Despite its importance, most existing datasets predominantly focus on English, resulting in a considerable gap in data availability for other languages. Cross-lingual transfer for QG (XLT-QG) addresses this limitation by allowing models trained on high-resource language datasets to generate questions in low-resource languages. In this paper, we propose a simple and efficient XLT-QG method that operates without the need for monolingual, parallel, or labeled data in the target language, utilizing a small language model. Our model, trained solely on English QA datasets, learns interrogative structures from a limited set of question exemplars, which are then applied to generate questions in the target language. Experimental results show that our method outperforms several XLT-QG baselines and achieves performance comparable to GPT-3.5-turbo across different languages. Additionally, the synthetic data generated by our model proves beneficial for training multilingual QA models. With significantly fewer parameters than large language models and without requiring additional training for target languages, our approach offers an effective solution for QG and QA tasks across various languages.

Title: PersoBench: Benchmarking Personalized Response Generation in Large Language Models

• Authors: Saleh Afzoon, Usman Naseem, Amin Beheshti, Zahra Jamali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03198
• Pdf URL: https://arxiv.org/pdf/2410.03198
• Copy Paste: [[2410.03198]] PersoBench: Benchmarking Personalized Response Generation in Large Language Models(https://arxiv.org/abs/2410.03198)
• Keywords: large language model
• Abstract: While large language models (LLMs) have exhibited impressive conversational capabilities, their proficiency in delivering personalized responses remains unclear. Although recent benchmarks automatically evaluate persona consistency in role-playing contexts using LLM-based judgment, the evaluation of personalization in response generation remains underexplored. To address this gap, we present a new benchmark, PersoBench, to evaluate the personalization ability of LLMs in persona-aware dialogue generation within a zero-shot setting. We assess the performance of three open-source and three closed-source LLMs using well-known datasets and a range of metrics. Our analysis, conducted on three well-known persona-aware datasets, evaluates multiple dimensions of response quality, including fluency, diversity, coherence, and personalization, across both standard and chain-of-thought prompting methods. Our findings reveal that while LLMs excel at generating fluent and diverse responses, they are far from satisfactory in delivering personalized and coherent responses considering both the conversation context and the provided personas. Our benchmark implementation is available at this https URL.

Title: Learning test generators for cyber-physical systems

• Authors: Jarkko Peltomäki, Ivan Porres
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.03202
• Pdf URL: https://arxiv.org/pdf/2410.03202
• Copy Paste: [[2410.03202]] Learning test generators for cyber-physical systems(https://arxiv.org/abs/2410.03202)
• Keywords: generative
• Abstract: Black-box runtime verification methods for cyber-physical systems can be used to discover errors in systems whose inputs and outputs are expressed as signals over time and their correctness requirements are specified in a temporal logic. Existing methods, such as requirement falsification, often focus on finding a single input that is a counterexample to system correctness. In this paper, we study how to create test generators that can produce multiple and diverse counterexamples for a single requirement. Several counterexamples expose system failures in varying input conditions and support the root cause analysis of the faults. We present the WOGAN algorithm to create such test generators automatically. The algorithm works by training iteratively a Wasserstein generative adversarial network that models the target distribution of the uniform distribution on the set of counterexamples. WOGAN is an algorithm that trains generative models that act as test generators for runtime verification. The training is performed online without the need for a previous model or dataset. We also propose criteria to evaluate such test generators. We evaluate the trained generators on several well-known problems including the ARCH-COMP falsification benchmarks. Our experimental results indicate that generators trained by the WOGAN algorithm are as effective as state-of-the-art requirement falsification algorithms while producing tests that are as diverse as a sample from uniform random sampling. We conclude that WOGAN is a viable method to produce test generators automatically and that these test generators can generate multiple and diverse counterexamples for the runtime verification of cyber-physical systems.

Title: Learning Semantic Structure through First-Order-Logic Translation

• Authors: Akshay Chaturvedi, Nicholas Asher
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03203
• Pdf URL: https://arxiv.org/pdf/2410.03203
• Copy Paste: [[2410.03203]] Learning Semantic Structure through First-Order-Logic Translation(https://arxiv.org/abs/2410.03203)
• Keywords: transformer, large language model
• Abstract: In this paper, we study whether transformer-based language models can extract predicate argument structure from simple sentences. We firstly show that language models sometimes confuse which predicates apply to which objects. To mitigate this, we explore two tasks: question answering (Q/A), and first order logic (FOL) translation, and two regimes, prompting and finetuning. In FOL translation, we finetune several large language models on synthetic datasets designed to gauge their generalization abilities. For Q/A, we finetune encoder models like BERT and RoBERTa and use prompting for LLMs. The results show that FOL translation for LLMs is better suited to learn predicate argument structure.

Title: CUDLE: Learning Under Label Scarcity to Detect Cannabis Use in Uncontrolled Environments

• Authors: Reza Rahimi Azghan, Nicholas C. Glodosky, Ramesh Kumar Sah, Carrie Cuttler, Ryan McLaughlin, Michael J. Cleveland, Hassan Ghasemzadeh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03211
• Pdf URL: https://arxiv.org/pdf/2410.03211
• Copy Paste: [[2410.03211]] CUDLE: Learning Under Label Scarcity to Detect Cannabis Use in Uncontrolled Environments(https://arxiv.org/abs/2410.03211)
• Keywords: robust
• Abstract: Wearable sensor systems have demonstrated a great potential for real-time, objective monitoring of physiological health to support behavioral interventions. However, obtaining accurate labels in free-living environments remains difficult due to limited human supervision and the reliance on self-labeling by patients, making data collection and supervised learning particularly challenging. To address this issue, we introduce CUDLE (Cannabis Use Detection with Label Efficiency), a novel framework that leverages self-supervised learning with real-world wearable sensor data to tackle a pressing healthcare challenge: the automatic detection of cannabis consumption in free-living environments. CUDLE identifies cannabis consumption moments using sensor-derived data through a contrastive learning framework. It first learns robust representations via a self-supervised pretext task with data augmentation. These representations are then fine-tuned in a downstream task with a shallow classifier, enabling CUDLE to outperform traditional supervised methods, especially with limited labeled data. To evaluate our approach, we conducted a clinical study with 20 cannabis users, collecting over 500 hours of wearable sensor data alongside user-reported cannabis use moments through EMA (Ecological Momentary Assessment) methods. Our extensive analysis using the collected data shows that CUDLE achieves a higher accuracy of 73.4%, compared to 71.1% for the supervised approach, with the performance gap widening as the number of labels decreases. Notably, CUDLE not only surpasses the supervised model while using 75% less labels, but also reaches peak performance with far fewer subjects.

Title: An Intelligent Quantum Cyber-Security Framework for Healthcare Data Management

• Authors: Kishu Gupta, Deepika Saxena, Pooja Rani, Jitendra Kumar, Aaisha Makkar, Ashutosh Kumar Singh, Chung-Nan Lee
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.03217
• Pdf URL: https://arxiv.org/pdf/2410.03217
• Copy Paste: [[2410.03217]] An Intelligent Quantum Cyber-Security Framework for Healthcare Data Management(https://arxiv.org/abs/2410.03217)
• Keywords: secure, security, privacy
• Abstract: Digital healthcare is essential to facilitate consumers to access and disseminate their medical data easily for enhanced medical care services. However, the significant concern with digitalization across healthcare systems necessitates for a prompt, productive, and secure storage facility along with a vigorous communication strategy, to stimulate sensitive digital healthcare data sharing and proactive estimation of malicious entities. In this context, this paper introduces a comprehensive quantum-based framework to overwhelm the potential security and privacy issues for secure healthcare data management. It equips quantum encryption for the secured storage and dispersal of healthcare data over the shared cloud platform by employing quantum encryption. Also, the framework furnishes a quantum feed-forward neural network unit to examine the intention behind the data request before granting access, for proactive estimation of potential data breach. In this way, the proposed framework delivers overall healthcare data management by coupling the advanced and more competent quantum approach with machine learning to safeguard the data storage, access, and prediction of malicious entities in an automated manner. Thus, the proposed IQ-HDM leads to more cooperative and effective healthcare delivery and empowers individuals with adequate custody of their health data. The experimental evaluation and comparison of the proposed IQ-HDM framework with state-of-the-art methods outline a considerable improvement up to 67.6%, in tackling cyber threats related to healthcare data security.

Title: Consultation on Industrial Machine Faults with Large language Models

• Authors: Apiradee Boonmee, Kritsada Wongsuwan, Pimchanok Sukjai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03223
• Pdf URL: https://arxiv.org/pdf/2410.03223
• Copy Paste: [[2410.03223]] Consultation on Industrial Machine Faults with Large language Models(https://arxiv.org/abs/2410.03223)
• Keywords: large language model
• Abstract: Industrial machine fault diagnosis is a critical component of operational efficiency and safety in manufacturing environments. Traditional methods rely heavily on expert knowledge and specific machine learning models, which can be limited in their adaptability and require extensive labeled data. This paper introduces a novel approach leveraging Large Language Models (LLMs), specifically through a structured multi-round prompting technique, to improve fault diagnosis accuracy. By dynamically crafting prompts, our method enhances the model's ability to synthesize information from diverse data sources, leading to improved contextual understanding and actionable recommendations. Experimental results demonstrate that our approach outperforms baseline models, achieving an accuracy of 91% in diagnosing various fault types. The findings underscore the potential of LLMs in revolutionizing industrial fault consultation practices, paving the way for more effective maintenance strategies in complex environments.

Title: AutoPenBench: Benchmarking Generative Agents for Penetration Testing

• Authors: Luca Gioacchini, Marco Mellia, Idilio Drago, Alexander Delsanto, Giuseppe Siracusano, Roberto Bifulco
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03225
• Pdf URL: https://arxiv.org/pdf/2410.03225
• Copy Paste: [[2410.03225]] AutoPenBench: Benchmarking Generative Agents for Penetration Testing(https://arxiv.org/abs/2410.03225)
• Keywords: security, attack, generative, large language model
• Abstract: Generative AI agents, software systems powered by Large Language Models (LLMs), are emerging as a promising approach to automate cybersecurity tasks. Among the others, penetration testing is a challenging field due to the task complexity and the diverse strategies to simulate cyber-attacks. Despite growing interest and initial studies in automating penetration testing with generative agents, there remains a significant gap in the form of a comprehensive and standard framework for their evaluation and development. This paper introduces AutoPenBench, an open benchmark for evaluating generative agents in automated penetration testing. We present a comprehensive framework that includes 33 tasks, each representing a vulnerable system that the agent has to attack. Tasks are of increasing difficulty levels, including in-vitro and real-world scenarios. We assess the agent performance with generic and specific milestones that allow us to compare results in a standardised manner and understand the limits of the agent under test. We show the benefits of AutoPenBench by testing two agent architectures: a fully autonomous and a semi-autonomous supporting human interaction. We compare their performance and limitations. For example, the fully autonomous agent performs unsatisfactorily achieving a 21% Success Rate (SR) across the benchmark, solving 27% of the simple tasks and only one real-world task. In contrast, the assisted agent demonstrates substantial improvements, with 64% of SR. AutoPenBench allows us also to observe how different LLMs like GPT-4o or OpenAI o1 impact the ability of the agents to complete the tasks. We believe that our benchmark fills the gap with a standard and flexible framework to compare penetration testing agents on a common ground. We hope to extend AutoPenBench along with the research community by making it available under this https URL.

Title: Frame-Voyager: Learning to Query Frames for Video Large Language Models

• Authors: Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xioalei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, Qianru Sun
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03226
• Pdf URL: https://arxiv.org/pdf/2410.03226
• Copy Paste: [[2410.03226]] Frame-Voyager: Learning to Query Frames for Video Large Language Models(https://arxiv.org/abs/2410.03226)
• Keywords: large language model
• Abstract: Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

Title: ALR$^2$: A Retrieve-then-Reason Framework for Long-context Question Answering

• Authors: Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, Yixuan Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03227
• Pdf URL: https://arxiv.org/pdf/2410.03227
• Copy Paste: [[2410.03227]] ALR$^2$: A Retrieve-then-Reason Framework for Long-context Question Answering(https://arxiv.org/abs/2410.03227)
• Keywords: large language model
• Abstract: The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capability of the model to accurately reason over that context degrades noticeably. This occurs because modern LLMs often become overwhelmed by the vast amount of information in the context; when answering questions, the model must identify and reason over relevant evidence sparsely distributed throughout the text. To alleviate the challenge of long-context reasoning, we develop a retrieve-then-reason framework, enabling LLMs to reason over relevant evidence collected during an intermediate retrieval step. We find that modern LLMs struggle to accurately retrieve relevant facts and instead, often hallucinate "retrieved facts", resulting in flawed reasoning and the production of incorrect answers. To address these issues, we introduce ALR$^2$, a method that augments the long-context reasoning capability of LLMs via an explicit two-stage procedure, i.e., aligning LLMs with the objectives of both retrieval and reasoning. We demonstrate the efficacy of ALR$^2$ for mitigating performance degradation in long-context reasoning tasks. Through extensive experiments on long-context QA benchmarks, we find our method to outperform competitive baselines by large margins, achieving at least 8.4 and 7.9 EM gains on the long-context versions of HotpotQA and SQuAD datasets, respectively.

Title: Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?

• Authors: Adam Nohejl, Frederikus Hudi, Eunike Andriani Kardinata, Shintaro Ozaki, Maria Angelica Riera Machin, Hongyu Sun, Justin Vasselli, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03240
• Pdf URL: https://arxiv.org/pdf/2410.03240
• Copy Paste: [[2410.03240]] Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?(https://arxiv.org/abs/2410.03240)
• Keywords: large language model
• Abstract: Word frequency is a key variable in psycholinguistics, useful for modeling human familiarity with words even in the era of large language models (LLMs). Frequency in film subtitles has proved to be a particularly good approximation of everyday language exposure. For many languages, however, film subtitles are not easily available, or are overwhelmingly translated from English. We demonstrate that frequencies extracted from carefully processed YouTube subtitles provide an approximation comparable to, and often better than, the best currently available resources. Moreover, they are available for languages for which a high-quality subtitle or speech corpus does not exist. We use YouTube subtitles to construct frequency norms for five diverse languages, Chinese, English, Indonesian, Japanese, and Spanish, and evaluate their correlation with lexical decision time, word familiarity, and lexical complexity. In addition to being strongly correlated with two psycholinguistic variables, a simple linear regression on the new frequencies achieves a new high score on a lexical complexity prediction task in English and Japanese, surpassing both models trained on film subtitle frequencies and the LLM GPT-4. Our code, the frequency lists, fastText word embeddings, and statistical language models are freely available at this https URL.

Title: How much can we forget about Data Contamination?

• Authors: Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko, Ulrike von Luxburg
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03249
• Pdf URL: https://arxiv.org/pdf/2410.03249
• Copy Paste: [[2410.03249]] How much can we forget about Data Contamination?(https://arxiv.org/abs/2410.03249)
• Keywords: large language model
• Abstract: The leakage of benchmark data into the training data has emerged as a significant challenge for evaluating the capabilities of large language models (LLMs). In this work, we use experimental evidence and theoretical estimates to challenge the common assumption that small-scale contamination renders benchmark evaluations invalid. First, we experimentally quantify the magnitude of benchmark overfitting based on scaling along three dimensions: The number of model parameters (up to 1.6B), the number of times an example is seen (up to 144), and the number of training tokens (up to 40B). We find that if model and data follow the Chinchilla scaling laws, minor contamination indeed leads to overfitting. At the same time, even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla, a regime characteristic of many modern LLMs. We then derive a simple theory of example forgetting via cumulative weight decay. It allows us to bound the number of gradient steps required to forget past data for any training run where we know the hyperparameters of AdamW. This indicates that many LLMs, including Llama 3, have forgotten the data seen at the beginning of training. Experimentally, we demonstrate that forgetting occurs faster than what is predicted by our bounds. Taken together, our results suggest that moderate amounts of contamination can be forgotten at the end of realistically scaled training runs.

Title: Sm: enhanced localization in Multiple Instance Learning for medical imaging classification

• Authors: Francisco M. Castro-Macías, Pablo Morales-Álvarez, Yunan Wu, Rafael Molina, Aggelos K. Katsaggelos
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03276
• Pdf URL: https://arxiv.org/pdf/2410.03276
• Copy Paste: [[2410.03276]] Sm: enhanced localization in Multiple Instance Learning for medical imaging classification(https://arxiv.org/abs/2410.03276)
• Keywords: transformer
• Abstract: Multiple Instance Learning (MIL) is widely used in medical imaging classification to reduce the labeling effort. While only bag labels are available for training, one typically seeks predictions at both bag and instance levels (classification and localization tasks, respectively). Early MIL methods treated the instances in a bag independently. Recent methods account for global and local dependencies among instances. Although they have yielded excellent results in classification, their performance in terms of localization is comparatively limited. We argue that these models have been designed to target the classification task, while implications at the instance level have not been deeply investigated. Motivated by a simple observation -- that neighboring instances are likely to have the same label -- we propose a novel, principled, and flexible mechanism to model local dependencies. It can be used alone or combined with any mechanism to model global dependencies (e.g., transformers). A thorough empirical validation shows that our module leads to state-of-the-art performance in localization while being competitive or superior in classification. Our code is at this https URL.

Title: What do Large Language Models Need for Machine Translation Evaluation?

• Authors: Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orăsan, Tharindu Ranasinghe, Frédéric Blain
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03278
• Pdf URL: https://arxiv.org/pdf/2410.03278
• Copy Paste: [[2410.03278]] What do Large Language Models Need for Machine Translation Evaluation?(https://arxiv.org/abs/2410.03278)
• Keywords: large language model
• Abstract: Leveraging large language models (LLMs) for various natural language processing tasks has led to superlative claims about their performance. For the evaluation of machine translation (MT), existing research shows that LLMs are able to achieve results comparable to fine-tuned multilingual pre-trained language models. In this paper, we explore what translation information, such as the source, reference, translation errors and annotation guidelines, is needed for LLMs to evaluate MT quality. In addition, we investigate prompting techniques such as zero-shot, Chain of Thought (CoT) and few-shot prompting for eight language pairs covering high-, medium- and low-resource languages, leveraging varying LLM variants. Our findings indicate the importance of reference translations for an LLM-based evaluation. While larger models do not necessarily fare better, they tend to benefit more from CoT prompting, than smaller models. We also observe that LLMs do not always provide a numerical score when generating evaluations, which poses a question on their reliability for the task. Our work presents a comprehensive analysis for resource-constrained and training-less LLM-based evaluation of machine translation. We release the accrued prompt templates, code and data publicly for reproducibility.

Title: BN-SCAFFOLD: controlling the drift of Batch Normalization statistics in Federated Learning

• Authors: Gonzalo Iñaki Quintana, Laurence Vancamberg, Vincent Jugnon, Mathilde Mougeot, Agnès Desolneux
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03281
• Pdf URL: https://arxiv.org/pdf/2410.03281
• Copy Paste: [[2410.03281]] BN-SCAFFOLD: controlling the drift of Batch Normalization statistics in Federated Learning(https://arxiv.org/abs/2410.03281)
• Keywords: federate
• Abstract: Federated Learning (FL) is gaining traction as a learning paradigm for training Machine Learning (ML) models in a decentralized way. Batch Normalization (BN) is ubiquitous in Deep Neural Networks (DNN), as it improves convergence and generalization. However, BN has been reported to hinder performance of DNNs in heterogeneous FL. Recently, the FedTAN algorithm has been proposed to mitigate the effect of heterogeneity on BN, by aggregating BN statistics and gradients from all the clients. However, it has a high communication cost, that increases linearly with the depth of the DNN. SCAFFOLD is a variance reduction algorithm, that estimates and corrects the client drift in a communication-efficient manner. Despite its promising results in heterogeneous FL settings, it has been reported to underperform for models with BN. In this work, we seek to revive SCAFFOLD, and more generally variance reduction, as an efficient way of training DNN with BN in heterogeneous FL. We introduce a unified theoretical framework for analyzing the convergence of variance reduction algorithms in the BN-DNN setting, inspired of by the work of Wang et al. 2023, and show that SCAFFOLD is unable to remove the bias introduced by BN. We thus propose the BN-SCAFFOLD algorithm, which extends the client drift correction of SCAFFOLD to BN statistics. We prove convergence using the aforementioned framework and validate the theoretical results with experiments on MNIST and CIFAR-10. BN-SCAFFOLD equals the performance of FedTAN, without its high communication cost, outperforming Federated Averaging (FedAvg), SCAFFOLD, and other FL algorithms designed to mitigate BN heterogeneity.

Title: uniINF: Best-of-Both-Worlds Algorithm for Parameter-Free Heavy-Tailed MABs

• Authors: Yu Chen, Jiatai Huang, Yan Dai, Longbo Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03284
• Pdf URL: https://arxiv.org/pdf/2410.03284
• Copy Paste: [[2410.03284]] uniINF: Best-of-Both-Worlds Algorithm for Parameter-Free Heavy-Tailed MABs(https://arxiv.org/abs/2410.03284)
• Keywords: robust
• Abstract: In this paper, we present a novel algorithm, uniINF, for the Heavy-Tailed Multi-Armed Bandits (HTMAB) problem, demonstrating robustness and adaptability in both stochastic and adversarial environments. Unlike the stochastic MAB setting where loss distributions are stationary with time, our study extends to the adversarial setup, where losses are generated from heavy-tailed distributions that depend on both arms and time. Our novel algorithm `uniINF` enjoys the so-called Best-of-Both-Worlds (BoBW) property, performing optimally in both stochastic and adversarial environments without knowing the exact environment type. Moreover, our algorithm also possesses a Parameter-Free feature, i.e., it operates without the need of knowing the heavy-tail parameters $(\sigma, \alpha)$ a-priori. To be precise, uniINF ensures nearly-optimal regret in both stochastic and adversarial environments, matching the corresponding lower bounds when $(\sigma, \alpha)$ is known (up to logarithmic factors). To our knowledge, uniINF is the first parameter-free algorithm to achieve the BoBW property for the heavy-tailed MAB problem. Technically, we develop innovative techniques to achieve BoBW guarantees for Parameter-Free HTMABs, including a refined analysis for the dynamics of log-barrier, an auto-balancing learning rate scheduling scheme, an adaptive skipping-clipping loss tuning technique, and a stopping-time analysis for logarithmic regret.

Title: Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

• Authors: Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, Lifu Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03290
• Pdf URL: https://arxiv.org/pdf/2410.03290
• Copy Paste: [[2410.03290]] Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models(https://arxiv.org/abs/2410.03290)
• Keywords: large language model
• Abstract: Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.

Title: Enhanced Transformer architecture for in-context learning of dynamical systems

• Authors: Matteo Rufolo, Dario Piga, Gabriele Maroni, Marco Forgione
• Subjects: cs.LG, cs.AI, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.03291
• Pdf URL: https://arxiv.org/pdf/2410.03291
• Copy Paste: [[2410.03291]] Enhanced Transformer architecture for in-context learning of dynamical systems(https://arxiv.org/abs/2410.03291)
• Keywords: transformer
• Abstract: Recently introduced by some of the authors, the in-context identification paradigm aims at estimating, offline and based on synthetic data, a meta-model that describes the behavior of a whole class of systems. Once trained, this meta-model is fed with an observed input/output sequence (context) generated by a real system to predict its behavior in a zero-shot learning fashion. In this paper, we enhance the original meta-modeling framework through three key innovations: by formulating the learning task within a probabilistic framework; by managing non-contiguous context and query windows; and by adopting recurrent patching to effectively handle long context sequences. The efficacy of these modifications is demonstrated through a numerical example focusing on the Wiener-Hammerstein system class, highlighting the model's enhanced performance and scalability.

Title: Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs

• Authors: Tianheng Ling, Chao Qian, Gregor Schiele
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03294
• Pdf URL: https://arxiv.org/pdf/2410.03294
• Copy Paste: [[2410.03294]] Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs(https://arxiv.org/abs/2410.03294)
• Keywords: transformer
• Abstract: This study addresses the deployment challenges of integer-only quantized Transformers on resource-constrained embedded FPGAs (Xilinx Spartan-7 XC7S15). We enhanced the flexibility of our VHDL template by introducing a selectable resource type for storing intermediate results across model layers, thereby breaking the deployment bottleneck by utilizing BRAM efficiently. Moreover, we developed a resource-aware mixed-precision quantization approach that enables researchers to explore hardware-level quantization strategies without requiring extensive expertise in Neural Architecture Search. This method provides accurate resource utilization estimates with a precision discrepancy as low as 3%, compared to actual deployment metrics. Compared to previous work, our approach has successfully facilitated the deployment of model configurations utilizing mixed-precision quantization, thus overcoming the limitations inherent in five previously non-deployable configurations with uniform quantization bitwidths. Consequently, this research enhances the applicability of Transformers in embedded systems, facilitating a broader range of Transformer-powered applications on edge devices.

Title: Action Selection Learning for Multi-label Multi-view Action Recognition

• Authors: Trung Thanh Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03302
• Pdf URL: https://arxiv.org/pdf/2410.03302
• Copy Paste: [[2410.03302]] Action Selection Learning for Multi-label Multi-view Action Recognition(https://arxiv.org/abs/2410.03302)
• Keywords: transformer
• Abstract: Multi-label multi-view action recognition aims to recognize multiple concurrent or sequential actions from untrimmed videos captured by multiple cameras. Existing work has focused on multi-view action recognition in a narrow area with strong labels available, where the onset and offset of each action are labeled at the frame-level. This study focuses on real-world scenarios where cameras are distributed to capture a wide-range area with only weak labels available at the video-level. We propose the method named MultiASL (Multi-view Action Selection Learning), which leverages action selection learning to enhance view fusion by selecting the most useful information from different viewpoints. The proposed method includes a Multi-view Spatial-Temporal Transformer video encoder to extract spatial and temporal features from multi-viewpoint videos. Action Selection Learning is employed at the frame-level, using pseudo ground-truth obtained from weak labels at the video-level, to identify the most relevant frames for action recognition. Experiments in a real-world office environment using the MM-Office dataset demonstrate the superior performance of the proposed method compared to existing methods.

Title: SELU: Self-Learning Embodied MLLMs in Unknown Environments

• Authors: Boyu Li, Haobin Jiang, Ziluo Ding, Xinrun Xu, Haoran Li, Dongbin Zhao, Zongqing Lu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03303
• Pdf URL: https://arxiv.org/pdf/2410.03303
• Copy Paste: [[2410.03303]] SELU: Self-Learning Embodied MLLMs in Unknown Environments(https://arxiv.org/abs/2410.03303)
• Keywords: large language model
• Abstract: Recently, multimodal large language models (MLLMs) have demonstrated strong visual understanding and decision-making capabilities, enabling the exploration of autonomously improving MLLMs in unknown environments. However, external feedback like human or environmental feedback is not always available. To address this challenge, existing methods primarily focus on enhancing the decision-making capabilities of MLLMs through voting and scoring mechanisms, while little effort has been paid to improving the environmental comprehension of MLLMs in unknown environments. To fully unleash the self-learning potential of MLLMs, we propose a novel actor-critic self-learning paradigm, dubbed SELU, inspired by the actor-critic paradigm in reinforcement learning. The critic employs self-asking and hindsight relabeling to extract knowledge from interaction trajectories collected by the actor, thereby augmenting its environmental comprehension. Simultaneously, the actor is improved by the self-feedback provided by the critic, enhancing its decision-making. We evaluate our method in the AI2-THOR and VirtualHome environments, and SELU achieves critic improvements of approximately 28% and 30%, and actor improvements of about 20% and 24% via self-learning.

Title: Quo Vadis, Motion Generation? From Large Language Models to Large Motion Models

• Authors: Ye Wang, Sipeng Zheng, Bin Cao, Qianshan Wei, Qin Jin, Zongqing Lu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03311
• Pdf URL: https://arxiv.org/pdf/2410.03311
• Copy Paste: [[2410.03311]] Quo Vadis, Motion Generation? From Large Language Models to Large Motion Models(https://arxiv.org/abs/2410.03311)
• Keywords: large language model
• Abstract: Inspired by the recent success of LLMs, the field of human motion understanding has increasingly shifted towards the development of large motion models. Despite some progress, current state-of-the-art works remain far from achieving truly generalist models, largely due to the lack of large-scale, high-quality motion data. To address this, we present MotionBase, the first million-level motion generation benchmark, offering 15 times the data volume of the previous largest dataset, and featuring multimodal data with hierarchically detailed text descriptions. By leveraging this vast dataset, our large motion model demonstrates strong performance across a broad range of motions, including unseen ones. Through systematic investigation, we underscore the importance of scaling both data and model size, with synthetic data and pseudo labels playing a crucial role in mitigating data acquisition costs. Moreover, our research reveals the limitations of existing evaluation metrics, particularly in handling out-of-domain text instructions -- an issue that has long been overlooked. In addition to these, we introduce a novel 2D lookup-free approach for motion tokenization, which preserves motion information and expands codebook capacity, further enhancing the representative ability of large motion models. The release of MotionBase and the insights gained from this study are expected to pave the way for the development of more powerful and versatile motion generation models.

Title: Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models

• Authors: Pavel Stepachev, Pinzhen Chen, Barry Haddow
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.03312
• Pdf URL: https://arxiv.org/pdf/2410.03312
• Copy Paste: [[2410.03312]] Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models(https://arxiv.org/abs/2410.03312)
• Keywords: large language model
• Abstract: Large language models (LLMs) have started to play a vital role in modelling speech and text. To explore the best use of context and multiple systems' outputs for post-ASR speech emotion prediction, we study LLM prompting on a recent task named GenSEC. Our techniques include ASR transcript ranking, variable conversation context, and system output fusion. We show that the conversation context has diminishing returns and the metric used to select the transcript for prediction is crucial. Finally, our best submission surpasses the provided baseline by 20% in absolute accuracy.

Title: Influence-oriented Personalized Federated Learning

• Authors: Yue Tan, Guodong Long, Jing Jiang, Chengqi Zhang
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.03315
• Pdf URL: https://arxiv.org/pdf/2410.03315
• Copy Paste: [[2410.03315]] Influence-oriented Personalized Federated Learning(https://arxiv.org/abs/2410.03315)
• Keywords: federate
• Abstract: Traditional federated learning (FL) methods often rely on fixed weighting for parameter aggregation, neglecting the mutual influence by others. Hence, their effectiveness in heterogeneous data contexts is limited. To address this problem, we propose an influence-oriented federated learning framework, namely FedC^2I, which quantitatively measures Client-level and Class-level Influence to realize adaptive parameter aggregation for each client. Our core idea is to explicitly model the inter-client influence within an FL system via the well-crafted influence vector and influence matrix. The influence vector quantifies client-level influence, enables clients to selectively acquire knowledge from others, and guides the aggregation of feature representation layers. Meanwhile, the influence matrix captures class-level influence in a more fine-grained manner to achieve personalized classifier aggregation. We evaluate the performance of FedC^2I against existing federated learning methods under non-IID settings and the results demonstrate the superiority of our method.

Title: An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

• Authors: Ahmed Abdulaal, Hugo Fry, Nina Montaña-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C. Alexander, Daniel C. Castro
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03334
• Pdf URL: https://arxiv.org/pdf/2410.03334
• Copy Paste: [[2410.03334]] An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation(https://arxiv.org/abs/2410.03334)
• Keywords: interpretability, transformer
• Abstract: Radiological services are experiencing unprecedented demand, leading to increased interest in automating radiology report generation. Existing Vision-Language Models (VLMs) suffer from hallucinations, lack interpretability, and require expensive fine-tuning. We introduce SAE-Rad, which uses sparse autoencoders (SAEs) to decompose latent representations from a pre-trained vision transformer into human-interpretable features. Our hybrid architecture combines state-of-the-art SAE advancements, achieving accurate latent reconstructions while maintaining sparsity. Using an off-the-shelf language model, we distil ground-truth reports into radiological descriptions for each SAE feature, which we then compile into a full report for each image, eliminating the need for fine-tuning large models for this task. To the best of our knowledge, SAE-Rad represents the first instance of using mechanistic interpretability techniques explicitly for a downstream multi-modal reasoning task. On the MIMIC-CXR dataset, SAE-Rad achieves competitive radiology-specific metrics compared to state-of-the-art models while using significantly fewer computational resources for training. Qualitative analysis reveals that SAE-Rad learns meaningful visual concepts and generates reports aligning closely with expert interpretations. Our results suggest that SAEs can enhance multimodal reasoning in healthcare, providing a more interpretable alternative to existing VLMs.

Title: Zero-Shot Fact Verification via Natural Logic and Large Language Models

• Authors: Marek Strong, Rami Aly, Andreas Vlachos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03341
• Pdf URL: https://arxiv.org/pdf/2410.03341
• Copy Paste: [[2410.03341]] Zero-Shot Fact Verification via Natural Logic and Large Language Models(https://arxiv.org/abs/2410.03341)
• Keywords: explainability, large language model
• Abstract: The recent development of fact verification systems with natural logic has enhanced their explainability by aligning claims with evidence through set-theoretic operators, providing faithful justifications. Despite these advancements, such systems often rely on a large amount of training data annotated with natural logic. To address this issue, we propose a zero-shot method that utilizes the generalization capabilities of instruction-tuned large language models. To comprehensively assess the zero-shot capabilities of our method and other fact verification systems, we evaluate all models on both artificial and real-world claims, including multilingual datasets. We also compare our method against other fact verification systems in two setups. First, in the zero-shot generalization setup, we demonstrate that our approach outperforms other systems that were not specifically trained on natural logic data, achieving an average accuracy improvement of 8.96 points over the best-performing baseline. Second, in the zero-shot transfer setup, we show that current systems trained on natural logic data do not generalize well to other domains, and our method outperforms these systems across all datasets with real-world claims.

Title: Generating Equivalent Representations of Code By A Self-Reflection Approach

• Authors: Jia Li, Ge Li, Lecheng Wang, Hao Zhu, Zhi Jin
• Subjects: cs.CL, cs.PL, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.03351
• Pdf URL: https://arxiv.org/pdf/2410.03351
• Copy Paste: [[2410.03351]] Generating Equivalent Representations of Code By A Self-Reflection Approach(https://arxiv.org/abs/2410.03351)
• Keywords: large language model
• Abstract: Equivalent Representations (ERs) of code are textual representations that preserve the same semantics as the code itself, e.g., natural language comments and pseudocode. ERs play a critical role in software development and maintenance. However, how to automatically generate ERs of code remains an open challenge. In this paper, we propose a self-reflection approach to generating ERs of code. It enables two Large Language Models (LLMs) to work mutually and produce an ER through a reflection process. Depending on whether constraints on ERs are applied, our approach generates ERs in both open and constrained settings. We conduct a empirical study to generate ERs in two settings and obtain eight findings. (1) Generating ERs in the open setting. In the open setting, we allow LLMs to represent code without any constraints, analyzing the resulting ERs and uncovering five key findings. These findings shed light on how LLMs comprehend syntactic structures, APIs, and numerical computations in code. (2) Generating ERs in the constrained setting. In the constrained setting, we impose constraints on ERs, such as natural language comments, pseudocode, and flowcharts. This allows our approach to address a range of software engineering tasks. Based on our experiments, we have three findings demonstrating that our approach can effectively generate ERs that adhere to specific constraints, thus supporting various software engineering tasks. (3) Future directions. We also discuss potential future research directions, such as deriving intermediate languages for code generation, exploring LLM-friendly requirement descriptions, and further supporting software engineering tasks. We believe that this paper will spark discussions in research communities and inspire many follow-up studies.

Title: LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding

• Authors: Doohyuk Jang, Sihwan Park, June Yong Yang, Yeonsung Jung, Jihun Yun, Souvik Kundu, Sung-Yub Kim, Eunho Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03355
• Pdf URL: https://arxiv.org/pdf/2410.03355
• Copy Paste: [[2410.03355]] LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding(https://arxiv.org/abs/2410.03355)
• Keywords: diffusion
• Abstract: Auto-Regressive (AR) models have recently gained prominence in image generation, often matching or even surpassing the performance of diffusion models. However, one major limitation of AR models is their sequential nature, which processes tokens one at a time, slowing down generation compared to models like GANs or diffusion-based methods that operate more efficiently. While speculative decoding has proven effective for accelerating LLMs by generating multiple tokens in a single forward, its application in visual AR models remains largely unexplored. In this work, we identify a challenge in this setting, which we term \textit{token selection ambiguity}, wherein visual AR models frequently assign uniformly low probabilities to tokens, hampering the performance of speculative decoding. To overcome this challenge, we propose a relaxed acceptance condition referred to as LANTERN that leverages the interchangeability of tokens in latent space. This relaxation restores the effectiveness of speculative decoding in visual AR models by enabling more flexible use of candidate tokens that would otherwise be prematurely rejected. Furthermore, by incorporating a total variation distance bound, we ensure that these speed gains are achieved without significantly compromising image quality or semantic coherence. Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. In specific, compared to a naïve application of the state-of-the-art speculative decoding, LANTERN increases speed-ups by $\mathbf{1.75}\times$ and $\mathbf{1.76}\times$, as compared to greedy decoding and random sampling, respectively, when applied to LlamaGen, a contemporary visual AR model.

Title: Latent Abstractions in Generative Diffusion Models

• Authors: Giulio Franzese, Mattia Martini, Giulio Corallo, Paolo Papotti, Pietro Michiardi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.03368
• Pdf URL: https://arxiv.org/pdf/2410.03368
• Copy Paste: [[2410.03368]] Latent Abstractions in Generative Diffusion Models(https://arxiv.org/abs/2410.03368)
• Keywords: diffusion, generative
• Abstract: In this work we study how diffusion-based generative models produce high-dimensional data, such as an image, by implicitly relying on a manifestation of a low-dimensional set of latent abstractions, that guide the generative process. We present a novel theoretical framework that extends NLF, and that offers a unique perspective on SDE-based generative models. The development of our theory relies on a novel formulation of the joint (state and measurement) dynamics, and an information-theoretic measure of the influence of the system state on the measurement process. According to our theory, diffusion models can be cast as a system of SDE, describing a non-linear filter in which the evolution of unobservable latent abstractions steers the dynamics of an observable measurement process (corresponding to the generative pathways). In addition, we present an empirical study to validate our theory and previous empirical results on the emergence of latent abstractions at different stages of the generative process.

Title: Make Interval Bound Propagation great again

• Authors: Patryk Krukowski, Daniel Wilczak, Jacek Tabor, Anna Bielawska, Przemysław Spurek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03373
• Pdf URL: https://arxiv.org/pdf/2410.03373
• Copy Paste: [[2410.03373]] Make Interval Bound Propagation great again(https://arxiv.org/abs/2410.03373)
• Keywords: robust
• Abstract: In various scenarios motivated by real life, such as medical data analysis, autonomous driving, and adversarial training, we are interested in robust deep networks. A network is robust when a relatively small perturbation of the input cannot lead to drastic changes in output (like change of class, etc.). This falls under the broader scope field of Neural Network Certification (NNC). Two crucial problems in NNC are of profound interest to the scientific community: how to calculate the robustness of a given pre-trained network and how to construct robust networks. The common approach to constructing robust networks is Interval Bound Propagation (IBP). This paper demonstrates that IBP is sub-optimal in the first case due to its susceptibility to the wrapping effect. Even for linear activation, IBP gives strongly sub-optimal bounds. Consequently, one should use strategies immune to the wrapping effect to obtain bounds close to optimal ones. We adapt two classical approaches dedicated to strict computations -- Dubleton Arithmetic and Affine Arithmetic -- to mitigate the wrapping effect in neural networks. These techniques yield precise results for networks with linear activation functions, thus resisting the wrapping effect. As a result, we achieve bounds significantly closer to the optimal level than IBPs.

Title: Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantization

• Authors: Tung M. Luu, Thanh Nguyen, Tee Joshua Tian Jin, Sungwoon Kim, Chang D. Yoo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03376
• Pdf URL: https://arxiv.org/pdf/2410.03376
• Copy Paste: [[2410.03376]] Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantization(https://arxiv.org/abs/2410.03376)
• Keywords: defense, attack, robust
• Abstract: Recent studies reveal that well-performing reinforcement learning (RL) agents in training often lack resilience against adversarial perturbations during deployment. This highlights the importance of building a robust agent before deploying it in the real world. Most prior works focus on developing robust training-based procedures to tackle this problem, including enhancing the robustness of the deep neural network component itself or adversarially training the agent on strong attacks. In this work, we instead study an input transformation-based defense for RL. Specifically, we propose using a variant of vector quantization (VQ) as a transformation for input observations, which is then used to reduce the space of adversarial attacks during testing, resulting in the transformed observations being less affected by attacks. Our method is computationally efficient and seamlessly integrates with adversarial training, further enhancing the robustness of RL agents against adversarial attacks. Through extensive experiments in multiple environments, we demonstrate that using VQ as the input transformation effectively defends against adversarial attacks on the agent's observations.

Title: From Epilepsy Seizures Classification to Detection: A Deep Learning-based Approach for Raw EEG Signals

• Authors: Davy Darankoum, Manon Villalba, Clelia Allioux, Baptiste Caraballo, Carine Dumont, Eloise Gronlier, Corinne Roucard, Yann Roche, Chloe Habermacher, Sergei Grudinin, Julien Volle
• Subjects: cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2410.03385
• Pdf URL: https://arxiv.org/pdf/2410.03385
• Copy Paste: [[2410.03385]] From Epilepsy Seizures Classification to Detection: A Deep Learning-based Approach for Raw EEG Signals(https://arxiv.org/abs/2410.03385)
• Keywords: transformer
• Abstract: Epilepsy represents the most prevalent neurological disease in the world. One-third of people suffering from mesial temporal lobe epilepsy (MTLE) exhibit drug resistance, urging the need to develop new treatments. A key part in anti-seizure medication (ASM) development is the capability of detecting and quantifying epileptic seizures occurring in electroencephalogram (EEG) signals, which is crucial for treatment efficacy evaluation. In this study, we introduced a seizure detection pipeline based on deep learning models applied to raw EEG signals. This pipeline integrates: a new pre-processing technique which segments continuous raw EEG signals without prior distinction between seizure and seizure-free activities; a post-processing algorithm developed to reassemble EEG segments and allow the identification of seizures start/end; and finally, a new evaluation procedure based on a strict seizure events comparison between predicted and real labels. Models training have been performed using a data splitting strategy which addresses the potential for data leakage. We demonstrated the fundamental differences between a seizure classification and a seizure detection task and showed the differences in performance between the two tasks. Finally, we demonstrated the generalization capabilities across species of our best architecture, combining a Convolutional Neural Network and a Transformer encoder. The model was trained on animal EEGs and tested on human EEGs with a F1-score of 93% on a balanced Bonn dataset.

Title: GraphCroc: Cross-Correlation Autoencoder for Graph Structural Reconstruction

• Authors: Shijin Duan, Ruyi Ding, Jiaxing He, Aidong Adam Ding, Yunsi Fei, Xiaolin Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03396
• Pdf URL: https://arxiv.org/pdf/2410.03396
• Copy Paste: [[2410.03396]] GraphCroc: Cross-Correlation Autoencoder for Graph Structural Reconstruction(https://arxiv.org/abs/2410.03396)
• Keywords: robust
• Abstract: Graph-structured data is integral to many applications, prompting the development of various graph representation methods. Graph autoencoders (GAEs), in particular, reconstruct graph structures from node embeddings. Current GAE models primarily utilize self-correlation to represent graph structures and focus on node-level tasks, often overlooking multi-graph scenarios. Our theoretical analysis indicates that self-correlation generally falls short in accurately representing specific graph features such as islands, symmetrical structures, and directional edges, particularly in smaller or multiple graph contexts. To address these limitations, we introduce a cross-correlation mechanism that significantly enhances the GAE representational capabilities. Additionally, we propose GraphCroc, a new GAE that supports flexible encoder architectures tailored for various downstream tasks and ensures robust structural reconstruction, through a mirrored encoding-decoding process. This model also tackles the challenge of representation bias during optimization by implementing a loss-balancing strategy. Both theoretical analysis and numerical evaluations demonstrate that our methodology significantly outperforms existing self-correlation-based GAEs in graph structure reconstruction.

Title: EBES: Easy Benchmarking for Event Sequences

• Authors: Dmitry Osin, Igor Udovichenko, Viktor Moskvoretskii, Egor Shvetsov, Evgeny Burnaev
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03399
• Pdf URL: https://arxiv.org/pdf/2410.03399
• Copy Paste: [[2410.03399]] EBES: Easy Benchmarking for Event Sequences(https://arxiv.org/abs/2410.03399)
• Keywords: robust
• Abstract: Event sequences, characterized by irregular sampling intervals and a mix of categorical and numerical features, are common data structures in various real-world domains such as healthcare, finance, and user interaction logs. Despite advances in temporal data modeling techniques, there is no standardized benchmarks for evaluating their performance on event sequences. This complicates result comparison across different papers due to varying evaluation protocols, potentially misleading progress in this field. We introduce EBES, a comprehensive benchmarking tool with standardized evaluation scenarios and protocols, focusing on regression and classification problems with sequence-level targets. Our library simplifies benchmarking, dataset addition, and method integration through a unified interface. It includes a novel synthetic dataset and provides preprocessed real-world datasets, including the largest publicly available banking dataset. Our results provide an in-depth analysis of datasets, identifying some as unsuitable for model comparison. We investigate the importance of modeling temporal and sequential components, as well as the robustness and scaling properties of the models. These findings highlight potential directions for future research. Our benchmark aim is to facilitate reproducible research, expediting progress and increasing real-world impacts.

Title: Camel: Communication-Efficient and Maliciously Secure Federated Learning in the Shuffle Model of Differential Privacy

• Authors: Shuangqing Xu, Yifeng Zheng, Zhongyun Hua
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.03407
• Pdf URL: https://arxiv.org/pdf/2410.03407
• Copy Paste: [[2410.03407]] Camel: Communication-Efficient and Maliciously Secure Federated Learning in the Shuffle Model of Differential Privacy(https://arxiv.org/abs/2410.03407)
• Keywords: secure, security, privacy, protect, federate
• Abstract: Federated learning (FL) has rapidly become a compelling paradigm that enables multiple clients to jointly train a model by sharing only gradient updates for aggregation, without revealing their local private data. In order to protect the gradient updates which could also be privacy-sensitive, there has been a line of work studying local differential privacy (LDP) mechanisms to provide a formal privacy guarantee. With LDP mechanisms, clients locally perturb their gradient updates before sharing them out for aggregation. However, such approaches are known for greatly degrading the model utility, due to heavy noise addition. To enable a better privacy-utility tradeoff, a recently emerging trend is to apply the shuffle model of DP in FL, which relies on an intermediate shuffling operation on the perturbed gradient updates to achieve privacy amplification. Following this trend, in this paper, we present Camel, a new communication-efficient and maliciously secure FL framework in the shuffle model of DP. Camel first departs from existing works by ambitiously supporting integrity check for the shuffle computation, achieving security against malicious adversary. Specifically, Camel builds on the trending cryptographic primitive of secret-shared shuffle, with custom techniques we develop for optimizing system-wide communication efficiency, and for lightweight integrity checks to harden the security of server-side computation. In addition, we also derive a significantly tighter bound on the privacy loss through analyzing the Renyi differential privacy (RDP) of the overall FL process. Extensive experiments demonstrate that Camel achieves better privacy-utility trade-offs than the state-of-the-art work, with promising performance.

Title: Predictive Coding for Decision Transformer

• Authors: Tung M. Luu, Donghoon Lee, Chang D. Yoo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03408
• Pdf URL: https://arxiv.org/pdf/2410.03408
• Copy Paste: [[2410.03408]] Predictive Coding for Decision Transformer(https://arxiv.org/abs/2410.03408)
• Keywords: transformer
• Abstract: Recent work in offline reinforcement learning (RL) has demonstrated the effectiveness of formulating decision-making as return-conditioned supervised learning. Notably, the decision transformer (DT) architecture has shown promise across various domains. However, despite its initial success, DTs have underperformed on several challenging datasets in goal-conditioned RL. This limitation stems from the inefficiency of return conditioning for guiding policy learning, particularly in unstructured and suboptimal datasets, resulting in DTs failing to effectively learn temporal compositionality. Moreover, this problem might be further exacerbated in long-horizon sparse-reward tasks. To address this challenge, we propose the Predictive Coding for Decision Transformer (PCDT) framework, which leverages generalized future conditioning to enhance DT methods. PCDT utilizes an architecture that extends the DT framework, conditioned on predictive codings, enabling decision-making based on both past and future factors, thereby improving generalization. Through extensive experiments on eight datasets from the AntMaze and FrankaKitchen environments, our proposed method achieves performance on par with or surpassing existing popular value-based and transformer-based methods in offline goal-conditioned RL. Furthermore, we also evaluate our method on a goal-reaching task with a physical robot.

Title: One2set + Large Language Model: Best Partners for Keyphrase Generation

• Authors: Liangying Shao, Liang Zhang, Minlong Peng, Guoqi Ma, Hao Yue, Mingming Sun, Jinsong Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03421
• Pdf URL: https://arxiv.org/pdf/2410.03421
• Copy Paste: [[2410.03421]] One2set + Large Language Model: Best Partners for Keyphrase Generation(https://arxiv.org/abs/2410.03421)
• Keywords: large language model
• Abstract: Keyphrase generation (KPG) aims to automatically generate a collection of phrases representing the core concepts of a given document. The dominant paradigms in KPG include one2seq and one2set. Recently, there has been increasing interest in applying large language models (LLMs) to KPG. Our preliminary experiments reveal that it is challenging for a single model to excel in both recall and precision. Further analysis shows that: 1) the one2set paradigm owns the advantage of high recall, but suffers from improper assignments of supervision signals during training; 2) LLMs are powerful in keyphrase selection, but existing selection methods often make redundant selections. Given these observations, we introduce a generate-then-select framework decomposing KPG into two steps, where we adopt a one2set-based model as generator to produce candidates and then use an LLM as selector to select keyphrases from these candidates. Particularly, we make two important improvements on our generator and selector: 1) we design an Optimal Transport-based assignment strategy to address the above improper assignments; 2) we model the keyphrase selection as a sequence labeling task to alleviate redundant selections. Experimental results on multiple benchmark datasets show that our framework significantly surpasses state-of-the-art models, especially in absent keyphrase prediction.

Title: A General Framework for Producing Interpretable Semantic Text Embeddings

• Authors: Yiqun Sun, Qiang Huang, Yixuan Tang, Anthony K. H. Tung, Jun Yu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03435
• Pdf URL: https://arxiv.org/pdf/2410.03435
• Copy Paste: [[2410.03435]] A General Framework for Producing Interpretable Semantic Text Embeddings(https://arxiv.org/abs/2410.03435)
• Keywords: interpretability
• Abstract: Semantic text embedding is essential to many tasks in Natural Language Processing (NLP). While black-box models are capable of generating high-quality embeddings, their lack of interpretability limits their use in tasks that demand transparency. Recent approaches have improved interpretability by leveraging domain-expert-crafted or LLM-generated questions, but these methods rely heavily on expert input or well-prompt design, which restricts their generalizability and ability to generate discriminative questions across a wide range of tasks. To address these challenges, we introduce \algo{CQG-MBQA} (Contrastive Question Generation - Multi-task Binary Question Answering), a general framework for producing interpretable semantic text embeddings across diverse tasks. Our framework systematically generates highly discriminative, low cognitive load yes/no questions through the \algo{CQG} method and answers them efficiently with the \algo{MBQA} model, resulting in interpretable embeddings in a cost-effective manner. We validate the effectiveness and interpretability of \algo{CQG-MBQA} through extensive experiments and ablation studies, demonstrating that it delivers embedding quality comparable to many advanced black-box models while maintaining inherently interpretability. Additionally, \algo{CQG-MBQA} outperforms other interpretable text embedding methods across various downstream tasks.

Title: Zebra: In-Context and Generative Pretraining for Solving Parametric PDEs

• Authors: Louis Serrano, Armand Kassaï Koupaï, Thomas X Wang, Pierre Erbacher, Patrick Gallinari
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03437
• Pdf URL: https://arxiv.org/pdf/2410.03437
• Copy Paste: [[2410.03437]] Zebra: In-Context and Generative Pretraining for Solving Parametric PDEs(https://arxiv.org/abs/2410.03437)
• Keywords: robust, transformer, generative, large language model
• Abstract: Solving time-dependent parametric partial differential equations (PDEs) is challenging, as models must adapt to variations in parameters such as coefficients, forcing terms, and boundary conditions. Data-driven neural solvers either train on data sampled from the PDE parameters distribution in the hope that the model generalizes to new instances or rely on gradient-based adaptation and meta-learning to implicitly encode the dynamics from observations. This often comes with increased inference complexity. Inspired by the in-context learning capabilities of large language models (LLMs), we introduce Zebra, a novel generative auto-regressive transformer designed to solve parametric PDEs without requiring gradient adaptation at inference. By leveraging in-context information during both pre-training and inference, Zebra dynamically adapts to new tasks by conditioning on input sequences that incorporate context trajectories or preceding states. This approach enables Zebra to flexibly handle arbitrarily sized context inputs and supports uncertainty quantification through the sampling of multiple solution trajectories. We evaluate Zebra across a variety of challenging PDE scenarios, demonstrating its adaptability, robustness, and superior performance compared to existing approaches.

Title: ToolGen: Unified Tool Retrieval and Calling via Generation

• Authors: Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, Haonan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03439
• Pdf URL: https://arxiv.org/pdf/2410.03439
• Copy Paste: [[2410.03439]] ToolGen: Unified Tool Retrieval and Calling via Generation(https://arxiv.org/abs/2410.03439)
• Keywords: generative, large language model
• Abstract: As large language models (LLMs) advance, their inability to autonomously execute tasks by directly interacting with external tools remains a critical limitation. Traditional methods rely on inputting tool descriptions as context, which is constrained by context length and requires separate, often inefficient, retrieval mechanisms. We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM's parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation. Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains. By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs.

Title: Exploring the Benefit of Activation Sparsity in Pre-training

• Authors: Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin, Yankai Lin, Zhiyuan Zeng, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03440
• Pdf URL: https://arxiv.org/pdf/2410.03440
• Copy Paste: [[2410.03440]] Exploring the Benefit of Activation Sparsity in Pre-training(https://arxiv.org/abs/2410.03440)
• Keywords: transformer
• Abstract: Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pre-training process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to $2\times$ faster inference speed. Codes are available at this https URL.

Title: CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control

• Authors: Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H. Bermano, Michiel van de Panne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03441
• Pdf URL: https://arxiv.org/pdf/2410.03441
• Copy Paste: [[2410.03441]] CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control(https://arxiv.org/abs/2410.03441)
• Keywords: robust, diffusion
• Abstract: Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. this https URL

Title: How Language Models Prioritize Contextual Grammatical Cues?

• Authors: Hamidreza Amirzadeh, Afra Alishahi, Hosein Mohebbi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03447
• Pdf URL: https://arxiv.org/pdf/2410.03447
• Copy Paste: [[2410.03447]] How Language Models Prioritize Contextual Grammatical Cues?(https://arxiv.org/abs/2410.03447)
• Keywords: transformer
• Abstract: Transformer-based language models have shown an excellent ability to effectively capture and utilize contextual information. Although various analysis techniques have been used to quantify and trace the contribution of single contextual cues to a target task such as subject-verb agreement or coreference resolution, scenarios in which multiple relevant cues are available in the context remain underexplored. In this paper, we investigate how language models handle gender agreement when multiple gender cue words are present, each capable of independently disambiguating a target gender pronoun. We analyze two widely used Transformer-based models: BERT, an encoder-based, and GPT-2, a decoder-based model. Our analysis employs two complementary approaches: context mixing analysis, which tracks information flow within the model, and a variant of activation patching, which measures the impact of cues on the model's prediction. We find that BERT tends to prioritize the first cue in the context to form both the target word representations and the model's prediction, while GPT-2 relies more on the final cue. Our findings reveal striking differences in how encoder-based and decoder-based models prioritize and use contextual information for their predictions.

Title: Dynamic Diffusion Transformer

• Authors: Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, Yang You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03456
• Pdf URL: https://arxiv.org/pdf/2410.03456
• Copy Paste: [[2410.03456]] Dynamic Diffusion Transformer(https://arxiv.org/abs/2410.03456)
• Keywords: diffusion, transformer
• Abstract: Diffusion Transformer (DiT), an emerging diffusion model for image generation, has demonstrated superior performance but suffers from substantial computational costs. Our investigations reveal that these costs stem from the static inference paradigm, which inevitably introduces redundant computation in certain diffusion timesteps and spatial regions. To address this inefficiency, we propose Dynamic Diffusion Transformer (DyDiT), an architecture that dynamically adjusts its computation along both timestep and spatial dimensions during generation. Specifically, we introduce a Timestep-wise Dynamic Width (TDW) approach that adapts model width conditioned on the generation timesteps. In addition, we design a Spatial-wise Dynamic Token (SDT) strategy to avoid redundant computation at unnecessary spatial locations. Extensive experiments on various datasets and different-sized models verify the superiority of DyDiT. Notably, with <3% additional fine-tuning iterations, our method reduces the FLOPs of DiT-XL by 51%, accelerates generation by 1.73, and achieves a competitive FID score of 2.07 on ImageNet. The code is publicly available at this https URL Dynamic-Diffusion-Transformer.

Title: Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval Augmented Generation

• Authors: Tobias Leemann, Periklis Petridis, Giuseppe Vietri, Dionysis Manousakas, Aaron Roth, Sergul Aydore
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03461
• Pdf URL: https://arxiv.org/pdf/2410.03461
• Copy Paste: [[2410.03461]] Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval Augmented Generation(https://arxiv.org/abs/2410.03461)
• Keywords: generative, large language model
• Abstract: While retrieval augmented generation (RAG) has been shown to enhance factuality of large language model (LLM) outputs, LLMs still suffer from hallucination, generating incorrect or irrelevant information. One common detection strategy involves prompting the LLM again to assess whether its response is grounded in the retrieved evidence, but this approach is costly. Alternatively, lightweight natural language inference (NLI) models for efficient grounding verification can be used at inference time. While existing pre-trained NLI models offer potential solutions, their performance remains subpar compared to larger models on realistic RAG inputs. RAG inputs are more complex than most datasets used for training NLI models and have characteristics specific to the underlying knowledge base, requiring adaptation of the NLI models to a specific target domain. Additionally, the lack of labeled instances in the target domain makes supervised domain adaptation, e.g., through fine-tuning, infeasible. To address these challenges, we introduce Automatic Generative Domain Adaptation (Auto-GDA). Our framework enables unsupervised domain adaptation through synthetic data generation. Unlike previous methods that rely on handcrafted filtering and augmentation strategies, Auto-GDA employs an iterative process to continuously improve the quality of generated samples using weak labels from less efficient teacher models and discrete optimization to select the most promising augmented samples. Experimental results demonstrate the effectiveness of our approach, with models fine-tuned on synthetic data using Auto-GDA often surpassing the performance of the teacher model and reaching the performance level of LLMs at 10 % of their computational cost.

Title: Linear Transformer Topological Masking with Graph Random Features

• Authors: Isaac Reid, Kumar Avinava Dubey, Deepali Jain, Will Whitney, Amr Ahmed, Joshua Ainslie, Alex Bewley, Mithun Jacob, Aranyak Mehta, David Rendleman, Connor Schenck, Richard E. Turner, René Wagner, Adrian Weller, Krzysztof Choromanski
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.03462
• Pdf URL: https://arxiv.org/pdf/2410.03462
• Copy Paste: [[2410.03462]] Linear Transformer Topological Masking with Graph Random Features(https://arxiv.org/abs/2410.03462)
• Keywords: transformer
• Abstract: When training transformers on graph-structured data, incorporating information about the underlying topology is crucial for good performance. Topological masking, a type of relative position encoding, achieves this by upweighting or downweighting attention depending on the relationship between the query and keys in a graph. In this paper, we propose to parameterise topological masks as a learnable function of a weighted adjacency matrix -- a novel, flexible approach which incorporates a strong structural inductive bias. By approximating this mask with graph random features (for which we prove the first known concentration bounds), we show how this can be made fully compatible with linear attention, preserving $\mathcal{O}(N)$ time and space complexity with respect to the number of input tokens. The fastest previous alternative was $\mathcal{O}(N \log N)$ and only suitable for specific graphs. Our efficient masking algorithms provide strong performance gains for tasks on image and point cloud data, including with $>30$k nodes.

Title: Diffusion State-Guided Projected Gradient for Inverse Problems

• Authors: Rayhan Zirvi, Bahareh Tolooshams, Anima Anandkumar
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03463
• Pdf URL: https://arxiv.org/pdf/2410.03463
• Copy Paste: [[2410.03463]] Diffusion State-Guided Projected Gradient for Inverse Problems(https://arxiv.org/abs/2410.03463)
• Keywords: robust, diffusion
• Abstract: Recent advancements in diffusion models have been effective in learning data priors for solving inverse problems. They leverage diffusion sampling steps for inducing a data prior while using a measurement guidance gradient at each step to impose data consistency. For general inverse problems, approximations are needed when an unconditionally trained diffusion model is used since the measurement likelihood is intractable, leading to inaccurate posterior sampling. In other words, due to their approximations, these methods fail to preserve the generation process on the data manifold defined by the diffusion prior, leading to artifacts in applications such as image restoration. To enhance the performance and robustness of diffusion models in solving inverse problems, we propose Diffusion State-Guided Projected Gradient (DiffStateGrad), which projects the measurement gradient onto a subspace that is a low-rank approximation of an intermediate state of the diffusion process. DiffStateGrad, as a module, can be added to a wide range of diffusion-based inverse solvers to improve the preservation of the diffusion process on the prior manifold and filter out artifact-inducing components. We highlight that DiffStateGrad improves the robustness of diffusion models in terms of the choice of measurement guidance step size and noise while improving the worst-case performance. Finally, we demonstrate that DiffStateGrad improves upon the state-of-the-art on linear and nonlinear image restoration inverse problems.

Title: Is Safer Better? The Impact of Guardrails on the Argumentative Strength of LLMs in Hate Speech Countering

• Authors: Helena Bonaldi, Greta Damo, Nicolás Benjamín Ocampo, Elena Cabrio, Serena Villata, Marco Guerini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03466
• Pdf URL: https://arxiv.org/pdf/2410.03466
• Copy Paste: [[2410.03466]] Is Safer Better? The Impact of Guardrails on the Argumentative Strength of LLMs in Hate Speech Countering(https://arxiv.org/abs/2410.03466)
• Keywords: attack
• Abstract: The potential effectiveness of counterspeech as a hate speech mitigation strategy is attracting increasing interest in the NLG research community, particularly towards the task of automatically producing it. However, automatically generated responses often lack the argumentative richness which characterises expert-produced counterspeech. In this work, we focus on two aspects of counterspeech generation to produce more cogent responses. First, by investigating the tension between helpfulness and harmlessness of LLMs, we test whether the presence of safety guardrails hinders the quality of the generations. Secondly, we assess whether attacking a specific component of the hate speech results in a more effective argumentative strategy to fight online hate. By conducting an extensive human and automatic evaluation, we show how the presence of safety guardrails can be detrimental also to a task that inherently aims at fostering positive social interactions. Moreover, our results show that attacking a specific component of the hate speech, and in particular its implicit negative stereotype and its hateful parts, leads to higher-quality generations.

Title: VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning

• Authors: Han Lin, Tushar Nagarajan, Nicolas Ballas, Mido Assran, Mojtaba Komeili, Mohit Bansal, Koustuv Sinha
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03478
• Pdf URL: https://arxiv.org/pdf/2410.03478
• Copy Paste: [[2410.03478]] VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning(https://arxiv.org/abs/2410.03478)
• Keywords: robust, diffusion, transformer
• Abstract: Procedural video representation learning is an active research area where the objective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations. Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extending compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works. In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR. Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting through iterative denoising - leveraging the recent advances in diffusion transformers (Peebles & Xie, 2023). Empirical studies over a total of five procedural learning tasks across four datasets (NIV, CrossTask, COIN and Ego4D-v2) show that our model advances the strong baselines in long-horizon action anticipation (+2.6% in Verb ED@20, +3.1% in Noun ED@20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc, and +0.90% in mIoU).

Title: A Multimodal Framework for Deepfake Detection

• Authors: Kashish Gandhi, Prutha Kulkarni, Taran Shah, Piyush Chaudhari, Meera Narvekar, Kranti Ghag
• Subjects: cs.CV, cs.AI, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2410.03487
• Pdf URL: https://arxiv.org/pdf/2410.03487
• Copy Paste: [[2410.03487]] A Multimodal Framework for Deepfake Detection(https://arxiv.org/abs/2410.03487)
• Keywords: security, privacy, extraction
• Abstract: The rapid advancement of deepfake technology poses a significant threat to digital media integrity. Deepfakes, synthetic media created using AI, can convincingly alter videos and audio to misrepresent reality. This creates risks of misinformation, fraud, and severe implications for personal privacy and security. Our research addresses the critical issue of deepfakes through an innovative multimodal approach, targeting both visual and auditory elements. This comprehensive strategy recognizes that human perception integrates multiple sensory inputs, particularly visual and auditory information, to form a complete understanding of media content. For visual analysis, a model that employs advanced feature extraction techniques was developed, extracting nine distinct facial characteristics and then applying various machine learning and deep learning models. For auditory analysis, our model leverages mel-spectrogram analysis for feature extraction and then applies various machine learning and deep learningmodels. To achieve a combined analysis, real and deepfake audio in the original dataset were swapped for testing purposes and ensured balanced samples. Using our proposed models for video and audio classification i.e. Artificial Neural Network and VGG19, the overall sample is classified as deepfake if either component is identified as such. Our multimodal framework combines visual and auditory analyses, yielding an accuracy of 94%.

Title: Gradient-based Jailbreak Images for Multimodal Fusion Models

• Authors: Javier Rando, Hannah Korevaar, Erik Brinkman, Ivan Evtimov, Florian Tramèr
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03489
• Pdf URL: https://arxiv.org/pdf/2410.03489
• Copy Paste: [[2410.03489]] Gradient-based Jailbreak Images for Multimodal Fusion Models(https://arxiv.org/abs/2410.03489)
• Keywords: defense, attack
• Abstract: Augmenting language models with image inputs may enable more effective jailbreak attacks through continuous optimization, unlike text inputs that require discrete optimization. However, new multimodal fusion models tokenize all input modalities using non-differentiable functions, which hinders straightforward attacks. In this work, we introduce the notion of a tokenizer shortcut that approximates tokenization with a continuous function and enables continuous optimization. We use tokenizer shortcuts to create the first end-to-end gradient image attacks against multimodal fusion models. We evaluate our attacks on Chameleon models and obtain jailbreak images that elicit harmful information for 72.5% of prompts. Jailbreak images outperform text jailbreaks optimized with the same objective and require 3x lower compute budget to optimize 50x more input tokens. Finally, we find that representation engineering defenses, like Circuit Breakers, trained only on text attacks can effectively transfer to adversarial image inputs.

Title: Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores

• Authors: Robert E. Blackwell, Jon Barry, Anthony G. Cohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03492
• Pdf URL: https://arxiv.org/pdf/2410.03492
• Copy Paste: [[2410.03492]] Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores(https://arxiv.org/abs/2410.03492)
• Keywords: large language model
• Abstract: Large language models (LLMs) are stochastic, and not all models give deterministic answers, even when setting temperature to zero with a fixed random seed. However, few benchmark studies attempt to quantify uncertainty, partly due to the time and cost of repeated experiments. We use benchmarks designed for testing LLMs' capacity to reason about cardinal directions to explore the impact of experimental repeats on mean score and prediction interval. We suggest a simple method for cost-effectively quantifying the uncertainty of a benchmark score and make recommendations concerning reproducible LLM evaluation.

Title: Generative Artificial Intelligence for Navigating Synthesizable Chemical Space

• Authors: Wenhao Gao, Shitong Luo, Connor W. Coley
• Subjects: cs.LG, cs.AI, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.03494
• Pdf URL: https://arxiv.org/pdf/2410.03494
• Copy Paste: [[2410.03494]] Generative Artificial Intelligence for Navigating Synthesizable Chemical Space(https://arxiv.org/abs/2410.03494)
• Keywords: diffusion, transformer, generative
• Abstract: We introduce SynFormer, a generative modeling framework designed to efficiently explore and navigate synthesizable chemical space. Unlike traditional molecular generation approaches, we generate synthetic pathways for molecules to ensure that designs are synthetically tractable. By incorporating a scalable transformer architecture and a diffusion module for building block selection, SynFormer surpasses existing models in synthesizable molecular design. We demonstrate SynFormer's effectiveness in two key applications: (1) local chemical space exploration, where the model generates synthesizable analogs of a reference molecule, and (2) global chemical space exploration, where the model aims to identify optimal molecules according to a black-box property prediction oracle. Additionally, we demonstrate the scalability of our approach via the improvement in performance as more computational resources become available. With our code and trained models openly available, we hope that SynFormer will find use across applications in drug discovery and materials science.

Title: Collaborative and Efficient Personalization with Mixtures of Adaptors

• Authors: Abdulla Jasem Almansoori, Samuel Horváth, Martin Takáč
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03497
• Pdf URL: https://arxiv.org/pdf/2410.03497
• Copy Paste: [[2410.03497]] Collaborative and Efficient Personalization with Mixtures of Adaptors(https://arxiv.org/abs/2410.03497)
• Keywords: robust, federate
• Abstract: Non-iid data is prevalent in real-world federated learning problems. Data heterogeneity can come in different types in terms of distribution shifts. In this work, we are interested in the heterogeneity that comes from concept shifts, i.e., shifts in the prediction across clients. In particular, we consider multi-task learning, where we want the model to adapt to the task of the client. We propose a parameter-efficient framework to tackle this issue, where each client learns to mix between parameter-efficient adaptors according to its task. We use Low-Rank Adaptors (LoRAs) as the backbone and extend its concept to other types of layers. We call our framework Federated Low-Rank Adaptive Learning (FLoRAL). This framework is not an algorithm but rather a model parameterization for a multi-task learning objective, so it can work on top of any algorithm that optimizes this objective, which includes many algorithms from the literature. FLoRAL is memory-efficient, and clients are personalized with small states (e.g., one number per adaptor) as the adaptors themselves are federated. Hence, personalization is--in this sense--federated as well. Even though clients can personalize more freely by training an adaptor locally, we show that collaborative and efficient training of adaptors is possible and performs better. We also show that FLoRAL can outperform an ensemble of full models with optimal cluster assignment, which demonstrates the benefits of federated personalization and the robustness of FLoRAL to overfitting. We show promising experimental results on synthetic datasets, real-world federated multi-task problems such as MNIST, CIFAR-10, and CIFAR-100. We also provide a theoretical analysis of local SGD on a relaxed objective and discuss the effects of aggregation mismatch on convergence.

Title: FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator

• Authors: Sunny Gupta, Nikita Jangid, Amit Sethi
• Subjects: cs.LG, cs.AI, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.03499
• Pdf URL: https://arxiv.org/pdf/2410.03499
• Copy Paste: [[2410.03499]] FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator(https://arxiv.org/abs/2410.03499)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) facilitates data privacy by enabling collaborative in-situ training across decentralized clients. Despite its inherent advantages, FL faces significant challenges of performance and convergence when dealing with data that is not independently and identically distributed (non-i.i.d.). While previous research has primarily addressed the issue of skewed label distribution across clients, this study focuses on the less explored challenge of multi-domain FL, where client data originates from distinct domains with varying feature distributions. We introduce a novel method designed to address these challenges FedStein: Enhancing Multi-Domain Federated Learning Through the James-Stein Estimator. FedStein uniquely shares only the James-Stein (JS) estimates of batch normalization (BN) statistics across clients, while maintaining local BN parameters. The non-BN layer parameters are exchanged via standard FL techniques. Extensive experiments conducted across three datasets and multiple models demonstrate that FedStein surpasses existing methods such as FedAvg and FedBN, with accuracy improvements exceeding 14% in certain domains leading to enhanced domain generalization. The code is available at this https URL

Title: CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

• Authors: Zetian Ouyang, Yishuai Qiu, Linlin Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03502
• Pdf URL: https://arxiv.org/pdf/2410.03502
• Copy Paste: [[2410.03502]] CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios(https://arxiv.org/abs/2410.03502)
• Keywords: large language model
• Abstract: With the proliferation of Large Language Models (LLMs) in diverse domains, there is a particular need for unified evaluation standards in clinical medical scenarios, where models need to be examined very thoroughly. We present CliMedBench, a comprehensive benchmark with 14 expert-guided core clinical scenarios specifically designed to assess the medical ability of LLMs across 7 pivot dimensions. It comprises 33,735 questions derived from real-world medical reports of top-tier tertiary hospitals and authentic examination exercises. The reliability of this benchmark has been confirmed in several ways. Subsequent experiments with existing LLMs have led to the following findings: (i) Chinese medical LLMs underperform on this benchmark, especially where medical reasoning and factual consistency are vital, underscoring the need for advances in clinical knowledge and diagnostic accuracy. (ii) Several general-domain LLMs demonstrate substantial potential in medical clinics, while the limited input capacity of many medical LLMs hinders their practical use. These findings reveal both the strengths and limitations of LLMs in clinical scenarios and offer critical insights for medical research.

Title: Classification-Denoising Networks

• Authors: Louis Thiry, Florentin Guth
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03505
• Pdf URL: https://arxiv.org/pdf/2410.03505
• Copy Paste: [[2410.03505]] Classification-Denoising Networks(https://arxiv.org/abs/2410.03505)
• Keywords: robust
• Abstract: Image classification and denoising suffer from complementary issues of lack of robustness or partially ignoring conditioning information. We argue that they can be alleviated by unifying both tasks through a model of the joint probability of (noisy) images and class labels. Classification is performed with a forward pass followed by conditioning. Using the Tweedie-Miyasawa formula, we evaluate the denoising function with the score, which can be computed by marginalization and back-propagation. The training objective is then a combination of cross-entropy loss and denoising score matching loss integrated over noise levels. Numerical experiments on CIFAR-10 and ImageNet show competitive classification and denoising performance compared to reference deep convolutional classifiers/denoisers, and significantly improves efficiency compared to previous joint approaches. Our model shows an increased robustness to adversarial perturbations compared to a standard discriminative classifier, and allows for a novel interpretation of adversarial gradients as a difference of denoisers.

Title: Stabilized Neural Prediction of Potential Outcomes in Continuous Time

• Authors: Konstantin Hess, Stefan Feuerriegel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03514
• Pdf URL: https://arxiv.org/pdf/2410.03514
• Copy Paste: [[2410.03514]] Stabilized Neural Prediction of Potential Outcomes in Continuous Time(https://arxiv.org/abs/2410.03514)
• Keywords: robust
• Abstract: Patient trajectories from electronic health records are widely used to predict potential outcomes of treatments over time, which then allows to personalize care. Yet, existing neural methods for this purpose have a key limitation: while some adjust for time-varying confounding, these methods assume that the time series are recorded in discrete time. In other words, they are constrained to settings where measurements and treatments are conducted at fixed time steps, even though this is unrealistic in medical practice. In this work, we aim to predict potential outcomes in continuous time. The latter is of direct practical relevance because it allows for modeling patient trajectories where measurements and treatments take place at arbitrary, irregular timestamps. We thus propose a new method called stabilized continuous time inverse propensity network (SCIP-Net). For this, we further derive stabilized inverse propensity weights for robust prediction of the potential outcomes. To the best of our knowledge, our SCIP-Net is the first neural method that performs proper adjustments for time-varying confounding in continuous time.

Title: LCMDC: Large-scale Chinese Medical Dialogue Corpora for Automatic Triage and Medical Consultation

• Authors: Xinyuan Wang, Haozhou Li, Dingfang Zheng, Qinke Peng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03521
• Pdf URL: https://arxiv.org/pdf/2410.03521
• Copy Paste: [[2410.03521]] LCMDC: Large-scale Chinese Medical Dialogue Corpora for Automatic Triage and Medical Consultation(https://arxiv.org/abs/2410.03521)
• Keywords: privacy
• Abstract: The global COVID-19 pandemic underscored major deficiencies in traditional healthcare systems, hastening the advancement of online medical services, especially in medical triage and consultation. However, existing studies face two main challenges. First, the scarcity of large-scale, publicly available, domain-specific medical datasets due to privacy concerns, with current datasets being small and limited to a few diseases, limiting the effectiveness of triage methods based on Pre-trained Language Models (PLMs). Second, existing methods lack medical knowledge and struggle to accurately understand professional terms and expressions in patient-doctor consultations. To overcome these obstacles, we construct the Large-scale Chinese Medical Dialogue Corpora (LCMDC), comprising a Coarse-grained Triage dataset with 439,630 samples, a Fine-grained Diagnosis dataset with 199,600 samples, and a Medical Consultation dataset with 472,418 items, thereby addressing the data shortage in this field. Moreover, we further propose a novel triage system that combines BERT-based supervised learning with prompt learning, as well as a GPT-based medical consultation model using reinforcement learning. To enhance domain knowledge acquisition, we pre-trained PLMs using our self-constructed background corpus. Experimental results on the LCMDC demonstrate the efficacy of our proposed systems.

Title: A Probabilistic Perspective on Unlearning and Alignment for Large Language Models

• Authors: Yan Scholten, Stephan Günnemann, Leo Schwinn
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03523
• Pdf URL: https://arxiv.org/pdf/2410.03523
• Copy Paste: [[2410.03523]] A Probabilistic Perspective on Unlearning and Alignment for Large Language Models(https://arxiv.org/abs/2410.03523)
• Keywords: large language model
• Abstract: Comprehensive evaluation of Large Language Models (LLMs) is an open research problem. Existing evaluations rely on deterministic point estimates generated via greedy decoding. However, we find that deterministic evaluations fail to capture the whole output distribution of a model, yielding inaccurate estimations of model capabilities. This is particularly problematic in critical contexts such as unlearning and alignment, where precise model evaluations are crucial. To remedy this, we introduce the first formal probabilistic evaluation framework in LLMs. Namely, we derive novel metrics with high-probability guarantees concerning the output distribution of a model. Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment. Through a case study focused on unlearning, we reveal that deterministic evaluations falsely indicate successful unlearning, whereas our probabilistic evaluations demonstrate that most if not all of the supposedly unlearned information remains accessible in these models. Additionally, we propose a novel unlearning loss based on entropy optimization and adaptive temperature scaling, which significantly improves unlearning in probabilistic settings on recent benchmarks. Our proposed shift from point estimates to probabilistic evaluations of output distributions represents an important step toward comprehensive evaluations of LLMs. this https URL

Title: Steering Large Language Models between Code Execution and Textual Reasoning

• Authors: Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, Chi Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03524
• Pdf URL: https://arxiv.org/pdf/2410.03524
• Copy Paste: [[2410.03524]] Steering Large Language Models between Code Execution and Textual Reasoning(https://arxiv.org/abs/2410.03524)
• Keywords: large language model
• Abstract: While a lot of recent research focuses on enhancing the textual reasoning capabilities of Large Language Models (LLMs) by optimizing the multi-agent framework or reasoning chains, several benchmark tasks can be solved with 100% success through direct coding, which is more scalable and avoids the computational overhead associated with textual iterating and searching. Textual reasoning has inherent limitations in solving tasks with challenges in math, logics, optimization, and searching, which is unlikely to be solved by simply scaling up the model and data size. The recently released OpenAI GPT Code Interpreter and multi-agent frameworks such as AutoGen have demonstrated remarkable proficiency of integrating code generation and execution to solve complex tasks using LLMs. However, based on our experiments on 7 existing popular methods for steering code/text generation in both single- and multi-turn settings with 14 tasks and 6 types of LLMs (including the new O1-preview), currently there is no optimal method to correctly steer LLMs to write code when needed. We discover some interesting patterns on when models use code vs. textual reasoning with the evolution to task complexity and model sizes, which even result in an astonishingly inverse scaling law. We also discover that results from LLM written code are not always better than using textual reasoning, even if the task could be solved through code. To mitigate the above issues, we propose three methods to better steer LLM code/text generation and achieve a notable improvement. The costs of token lengths and runtime are thoroughly discussed for all the methods. We believe the problem of steering LLM code/text generation is critical for future research and has much space for further improvement. Project Page, Datasets, and Codes are available at this https URL.

Title: MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction

• Authors: Han Jiang, Junwen Duan, Zhe Qu, Jianxin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03531
• Pdf URL: https://arxiv.org/pdf/2410.03531
• Copy Paste: [[2410.03531]] MARE: Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction(https://arxiv.org/abs/2410.03531)
• Keywords: extraction
• Abstract: Unsupervised rationale extraction aims to extract text snippets to support model predictions without explicit rationale annotation. Researchers have made many efforts to solve this task. Previous works often encode each aspect independently, which may limit their ability to capture meaningful internal correlations between aspects. While there has been significant work on mitigating spurious correlations, our approach focuses on leveraging the beneficial internal correlations to improve multi-aspect rationale extraction. In this paper, we propose a Multi-Aspect Rationale Extractor (MARE) to explain and predict multiple aspects simultaneously. Concretely, we propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on hard deletion to encode multiple text chunks simultaneously. Furthermore, multiple special tokens are prepended in front of the text with each corresponding to one certain aspect. Finally, multi-task training is deployed to reduce the training overhead. Experimental results on two unsupervised rationale extraction benchmarks show that MARE achieves state-of-the-art performance. Ablation studies further demonstrate the effectiveness of our method. Our codes have been available at this https URL.

Title: NRGBoost: Energy-Based Generative Boosted Trees

• Authors: João Bravo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03535
• Pdf URL: https://arxiv.org/pdf/2410.03535
• Copy Paste: [[2410.03535]] NRGBoost: Energy-Based Generative Boosted Trees(https://arxiv.org/abs/2410.03535)
• Keywords: generative
• Abstract: Despite the rise to dominance of deep learning in unstructured data domains, tree-based methods such as Random Forests (RF) and Gradient Boosted Decision Trees (GBDT) are still the workhorses for handling discriminative tasks on tabular data. We explore generative extensions of these popular algorithms with a focus on explicitly modeling the data density (up to a normalization constant), thus enabling other applications besides sampling. As our main contribution we propose an energy-based generative boosting algorithm that is analogous to the second order boosting implemented in popular packages like XGBoost. We show that, despite producing a generative model capable of handling inference tasks over any input variable, our proposed algorithm can achieve similar discriminative performance to GBDT on a number of real world tabular datasets, outperforming alternative generative approaches. At the same time, we show that it is also competitive with neural network based models for sampling.

Title: Ward: Provable RAG Dataset Inference via LLM Watermarks

• Authors: Nikola Jovanović, Robin Staab, Maximilian Baader, Martin Vechev
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.03537
• Pdf URL: https://arxiv.org/pdf/2410.03537
• Copy Paste: [[2410.03537]] Ward: Provable RAG Dataset Inference via LLM Watermarks(https://arxiv.org/abs/2410.03537)
• Keywords: robust, watermark
• Abstract: Retrieval-Augmented Generation (RAG) improves LLMs by enabling them to incorporate external data during generation. This raises concerns for data owners regarding unauthorized use of their content in RAG systems. Despite its importance, the challenge of detecting such unauthorized usage remains underexplored, with existing datasets and methodologies from adjacent fields being ill-suited for its study. In this work, we take several steps to bridge this gap. First, we formalize this problem as (black-box) RAG Dataset Inference (RAG-DI). To facilitate research on this challenge, we further introduce a novel dataset specifically designed for benchmarking RAG-DI methods under realistic conditions, and propose a set of baseline approaches. Building on this foundation, we introduce Ward, a RAG-DI method based on LLM watermarks that enables data owners to obtain rigorous statistical guarantees regarding the usage of their dataset in a RAG system. In our experimental evaluation, we show that Ward consistently outperforms all baselines across many challenging settings, achieving higher accuracy, superior query efficiency and robustness. Our work provides a foundation for future studies of RAG-DI and highlights LLM watermarks as a promising approach to this problem.

Title: Re-examining Sexism and Misogyny Classification with Annotator Attitudes

• Authors: Aiqi Jiang, Nikolas Vitsakis, Tanvi Dinkar, Gavin Abercrombie, Ioannis Konstas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03543
• Pdf URL: https://arxiv.org/pdf/2410.03543
• Copy Paste: [[2410.03543]] Re-examining Sexism and Misogyny Classification with Annotator Attitudes(https://arxiv.org/abs/2410.03543)
• Keywords: large language model
• Abstract: Gender-Based Violence (GBV) is an increasing problem online, but existing datasets fail to capture the plurality of possible annotator perspectives or ensure the representation of affected groups. We revisit two important stages in the moderation pipeline for GBV: (1) manual data labelling; and (2) automated classification. For (1), we examine two datasets to investigate the relationship between annotator identities and attitudes and the responses they give to two GBV labelling tasks. To this end, we collect demographic and attitudinal information from crowd-sourced annotators using three validated surveys from Social Psychology. We find that higher Right Wing Authoritarianism scores are associated with a higher propensity to label text as sexist, while for Social Dominance Orientation and Neosexist Attitudes, higher scores are associated with a negative tendency to do so. For (2), we conduct classification experiments using Large Language Models and five prompting strategies, including infusing prompts with annotator information. We find: (i) annotator attitudes affect the ability of classifiers to predict their labels; (ii) including attitudinal information can boost performance when we use well-structured brief annotator descriptions; and (iii) models struggle to reflect the increased complexity and imbalanced classes of the new label sets.

Title: Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding

• Authors: Wei Wu, Chao Wang, Liyi Chen, Mingze Yin, Yiheng Zhu, Kun Fu, Jieping Ye, Hui Xiong, Zheng Wang
• Subjects: cs.CL, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.03553
• Pdf URL: https://arxiv.org/pdf/2410.03553
• Copy Paste: [[2410.03553]] Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding(https://arxiv.org/abs/2410.03553)
• Keywords: large language model
• Abstract: Proteins, as essential biomolecules, play a central role in biological processes, including metabolic reactions and DNA replication. Accurate prediction of their properties and functions is crucial in biological applications. Recent development of protein language models (pLMs) with supervised fine tuning provides a promising solution to this problem. However, the fine-tuned model is tailored for particular downstream prediction task, and achieving general-purpose protein understanding remains a challenge. In this paper, we introduce Structure-Enhanced Protein Instruction Tuning (SEPIT) framework to bridge this gap. Our approach integrates a noval structure-aware module into pLMs to inform them with structural knowledge, and then connects these enhanced pLMs to large language models (LLMs) to generate understanding of proteins. In this framework, we propose a novel two-stage instruction tuning pipeline that first establishes a basic understanding of proteins through caption-based instructions and then refines this understanding using a mixture of experts (MoEs) to learn more complex properties and functional information with the same amount of activated parameters. Moreover, we construct the largest and most comprehensive protein instruction dataset to date, which allows us to train and evaluate the general-purpose protein understanding model. Extensive experimental results on open-ended generation and closed-set answer tasks demonstrate the superior performance of SEPIT over both closed-source general LLMs and open-source LLMs trained with protein knowledge.

Title: Artificial intelligence inspired freeform optics design: a review

• Authors: Lei Feng, Jingxing Liao, Jingna Yang
• Subjects: cs.LG, physics.optics
• Abstract URL: https://arxiv.org/abs/2410.03554
• Pdf URL: https://arxiv.org/pdf/2410.03554
• Copy Paste: [[2410.03554]] Artificial intelligence inspired freeform optics design: a review(https://arxiv.org/abs/2410.03554)
• Keywords: interpretability
• Abstract: Integrating artificial intelligence (AI) techniques such as machine learning and deep learning into freeform optics design has significantly enhanced design efficiency, expanded the design space, and led to innovative solutions. This article reviews the latest developments in AI applications within this field, highlighting their roles in initial design generation, optimization, and performance prediction. It also addresses the benefits of AI, such as improved accuracy and performance, alongside challenges like data requirements, model interpretability, and computational complexity. Despite these challenges, the future of AI in freeform optics design looks promising, with potential advancements in hybrid design methods, interpretable AI, AI-driven manufacturing, and targeted research for specific applications. Collaboration among researchers, engineers, and designers is essential to fully harness AI's potential and drive innovation in optics.

Title: BodyShapeGPT: SMPL Body Shape Manipulation with LLMs

• Authors: Baldomero R. Árbol, Dan Casas
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03556
• Pdf URL: https://arxiv.org/pdf/2410.03556
• Copy Paste: [[2410.03556]] BodyShapeGPT: SMPL Body Shape Manipulation with LLMs(https://arxiv.org/abs/2410.03556)
• Keywords: generative, large language model
• Abstract: Generative AI models provide a wide range of tools capable of performing complex tasks in a fraction of the time it would take a human. Among these, Large Language Models (LLMs) stand out for their ability to generate diverse texts, from literary narratives to specialized responses in different fields of knowledge. This paper explores the use of fine-tuned LLMs to identify physical descriptions of people, and subsequently create accurate representations of avatars using the SMPL-X model by inferring shape parameters. We demonstrate that LLMs can be trained to understand and manipulate the shape space of SMPL, allowing the control of 3D human shapes through natural language. This approach promises to improve human-machine interaction and opens new avenues for customization and simulation in virtual environments.

Title: Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features

• Authors: Benyuan Meng, Qianqian Xu, Zitai Wang, Xiaochun Cao, Qingming Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03558
• Pdf URL: https://arxiv.org/pdf/2410.03558
• Copy Paste: [[2410.03558]] Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features(https://arxiv.org/abs/2410.03558)
• Keywords: diffusion, segmentation
• Abstract: Diffusion models are initially designed for image generation. Recent research shows that the internal signals within their backbones, named activations, can also serve as dense features for various discriminative tasks such as semantic segmentation. Given numerous activations, selecting a small yet effective subset poses a fundamental problem. To this end, the early study of this field performs a large-scale quantitative comparison of the discriminative ability of the activations. However, we find that many potential activations have not been evaluated, such as the queries and keys used to compute attention scores. Moreover, recent advancements in diffusion architectures bring many new activations, such as those within embedded ViT modules. Both combined, activation selection remains unresolved but overlooked. To tackle this issue, this paper takes a further step with a much broader range of activations evaluated. Considering the significant increase in activations, a full-scale quantitative comparison is no longer operational. Instead, we seek to understand the properties of these activations, such that the activations that are clearly inferior can be filtered out in advance via simple qualitative evaluation. After careful analysis, we discover three properties universal among diffusion models, enabling this study to go beyond specific models. On top of this, we present effective feature selection solutions for several popular diffusion models. Finally, the experiments across multiple discriminative tasks validate the superiority of our method over the SOTA competitors. Our code is available at this https URL.

Title: A Survey on Offensive AI Within Cybersecurity

• Authors: Sahil Girhepuje, Aviral Verma, Gaurav Raina
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03566
• Pdf URL: https://arxiv.org/pdf/2410.03566
• Copy Paste: [[2410.03566]] A Survey on Offensive AI Within Cybersecurity(https://arxiv.org/abs/2410.03566)
• Keywords: security, attack
• Abstract: Artificial Intelligence (AI) has witnessed major growth and integration across various domains. As AI systems become increasingly prevalent, they also become targets for threat actors to manipulate their functionality for malicious purposes. This survey paper on offensive AI will comprehensively cover various aspects related to attacks against and using AI systems. It will delve into the impact of offensive AI practices on different domains, including consumer, enterprise, and public digital infrastructure. The paper will explore adversarial machine learning, attacks against AI models, infrastructure, and interfaces, along with offensive techniques like information gathering, social engineering, and weaponized AI. Additionally, it will discuss the consequences and implications of offensive AI, presenting case studies, insights, and avenues for further research.

Title: Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)

• Authors: Abrar Rahman, Garry Bowlin, Binit Mohanty, Sean McGunigal
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03568
• Pdf URL: https://arxiv.org/pdf/2410.03568
• Copy Paste: [[2410.03568]] Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)(https://arxiv.org/abs/2410.03568)
• Keywords: large language model
• Abstract: This paper presents a comprehensive study on the tokenization techniques employed by state-of-the-art large language models (LLMs) and their implications on the cost and availability of services across different languages, especially low resource languages. The analysis considers multiple LLMs, including GPT-4 (using cl100k_base embeddings), GPT-3 (with p50k_base embeddings), and DaVinci (employing r50k_base embeddings), as well as the widely used BERT base tokenizer. The study evaluates the tokenization variability observed across these models and investigates the challenges of linguistic representation in subword tokenization. The research underscores the importance of fostering linguistically-aware development practices, especially for languages that are traditionally under-resourced. Moreover, this paper introduces case studies that highlight the real-world implications of tokenization choices, particularly in the context of electronic health record (EHR) systems. This research aims to promote generalizable Internationalization (I18N) practices in the development of AI services in this domain and beyond, with a strong emphasis on inclusivity, particularly for languages traditionally underrepresented in AI applications.

Title: Teaching Transformers Modular Arithmetic at Scale

• Authors: Eshika Saxena, Alberto Alfarano, Emily Wenger, Kristin Lauter
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03569
• Pdf URL: https://arxiv.org/pdf/2410.03569
• Copy Paste: [[2410.03569]] Teaching Transformers Modular Arithmetic at Scale(https://arxiv.org/abs/2410.03569)
• Keywords: transformer
• Abstract: Modular addition is, on its face, a simple operation: given $N$ elements in $\mathbb{Z}_q$, compute their sum modulo $q$. Yet, scalable machine learning solutions to this problem remain elusive: prior work trains ML models that sum $N \le 6$ elements mod $q \le 1000$. Promising applications of ML models for cryptanalysis-which often involve modular arithmetic with large $N$ and $q$-motivate reconsideration of this problem. This work proposes three changes to the modular addition model training pipeline: more diverse training data, an angular embedding, and a custom loss function. With these changes, we demonstrate success with our approach for $N = 256, q = 3329$, a case which is interesting for cryptographic applications, and a significant increase in $N$ and $q$ over prior work. These techniques also generalize to other modular arithmetic problems, motivating future work.

Title: HyResPINNs: Adaptive Hybrid Residual Networks for Learning Optimal Combinations of Neural and RBF Components for Physics-Informed Modeling

• Authors: Madison Cooley, Robert M. Kirby, Shandian Zhe, Varun Shankar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03573
• Pdf URL: https://arxiv.org/pdf/2410.03573
• Copy Paste: [[2410.03573]] HyResPINNs: Adaptive Hybrid Residual Networks for Learning Optimal Combinations of Neural and RBF Components for Physics-Informed Modeling(https://arxiv.org/abs/2410.03573)
• Keywords: robust
• Abstract: Physics-informed neural networks (PINNs) are an increasingly popular class of techniques for the numerical solution of partial differential equations (PDEs), where neural networks are trained using loss functions regularized by relevant PDE terms to enforce physical constraints. We present a new class of PINNs called HyResPINNs, which augment traditional PINNs with adaptive hybrid residual blocks that combine the outputs of a standard neural network and a radial basis function (RBF) network. A key feature of our method is the inclusion of adaptive combination parameters within each residual block, which dynamically learn to weigh the contributions of the neural network and RBF network outputs. Additionally, adaptive connections between residual blocks allow for flexible information flow throughout the network. We show that HyResPINNs are more robust to training point locations and neural network architectures than traditional PINNs. Moreover, HyResPINNs offer orders of magnitude greater accuracy than competing methods on certain problems, with only modest increases in training costs. We demonstrate the strengths of our approach on challenging PDEs, including the Allen-Cahn equation and the Darcy-Flow equation. Our results suggest that HyResPINNs effectively bridge the gap between traditional numerical methods and modern machine learning-based solvers.

Title: Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

• Authors: Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, Xuming Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03577
• Pdf URL: https://arxiv.org/pdf/2410.03577
• Copy Paste: [[2410.03577]] Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models(https://arxiv.org/abs/2410.03577)
• Keywords: large language model
• Abstract: Despite their impressive capabilities, Multimodal Large Language Models (MLLMs) are susceptible to hallucinations, especially assertively fabricating content not present in the visual inputs. To address the aforementioned challenge, we follow a common cognitive process - when one's initial memory of critical on-sight details fades, it is intuitive to look at them a second time to seek a factual and accurate answer. Therefore, we introduce Memory-space Visual Retracing (MemVR), a novel hallucination mitigation paradigm that without the need for external knowledge retrieval or additional fine-tuning. In particular, we treat visual prompts as supplementary evidence to be reinjected into MLLMs via Feed Forward Network (FFN) as key-value memory, when the model is uncertain or even amnesic about question-relevant visual memories. Comprehensive experimental evaluations demonstrate that MemVR significantly mitigates hallucination issues across various MLLMs and excels in general benchmarks without incurring added time overhead, thus emphasizing its potential for widespread applicability.

Title: Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments

• Authors: Omar Sharif, Joseph Gatto, Madhusudan Basak, Sarah M. Preum
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03594
• Pdf URL: https://arxiv.org/pdf/2410.03594
• Copy Paste: [[2410.03594]] Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments(https://arxiv.org/abs/2410.03594)
• Keywords: extraction, generative
• Abstract: Prior works formulate the extraction of event-specific arguments as a span extraction problem, where event arguments are explicit -- i.e. assumed to be contiguous spans of text in a document. In this study, we revisit this definition of Event Extraction (EE) by introducing two key argument types that cannot be modeled by existing EE frameworks. First, implicit arguments are event arguments which are not explicitly mentioned in the text, but can be inferred through context. Second, scattered arguments are event arguments that are composed of information scattered throughout the text. These two argument types are crucial to elicit the full breadth of information required for proper event modeling. To support the extraction of explicit, implicit, and scattered arguments, we develop a novel dataset, DiscourseEE, which includes 7,464 argument annotations from online health discourse. Notably, 51.2% of the arguments are implicit, and 17.4% are scattered, making DiscourseEE a unique corpus for complex event extraction. Additionally, we formulate argument extraction as a text generation problem to facilitate the extraction of complex argument types. We provide a comprehensive evaluation of state-of-the-art models and highlight critical open challenges in generative event extraction. Our data and codebase are available at this https URL.

Title: Efficiently Identifying Watermarked Segments in Mixed-Source Texts

• Authors: Xuandong Zhao, Chenwen Liao, Yu-Xiang Wang, Lei Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.03600
• Pdf URL: https://arxiv.org/pdf/2410.03600
• Copy Paste: [[2410.03600]] Efficiently Identifying Watermarked Segments in Mixed-Source Texts(https://arxiv.org/abs/2410.03600)
• Keywords: watermark, large language model
• Abstract: Text watermarks in large language models (LLMs) are increasingly used to detect synthetic text, mitigating misuse cases like fake news and academic dishonesty. While existing watermarking detection techniques primarily focus on classifying entire documents as watermarked or not, they often neglect the common scenario of identifying individual watermark segments within longer, mixed-source documents. Drawing inspiration from plagiarism detection systems, we propose two novel methods for partial watermark detection. First, we develop a geometry cover detection framework aimed at determining whether there is a watermark segment in long text. Second, we introduce an adaptive online learning algorithm to pinpoint the precise location of watermark segments within the text. Evaluated on three popular watermarking techniques (KGW-Watermark, Unigram-Watermark, and Gumbel-Watermark), our approach achieves high accuracy, significantly outperforming baseline methods. Moreover, our framework is adaptable to other watermarking techniques, offering new insights for precise watermark detection.

Title: How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework

• Authors: Yinuo Ren, Haoxuan Chen, Grant M. Rotskoff, Lexing Ying
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.03601
• Pdf URL: https://arxiv.org/pdf/2410.03601
• Copy Paste: [[2410.03601]] How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework(https://arxiv.org/abs/2410.03601)
• Keywords: diffusion
• Abstract: Discrete diffusion models have gained increasing attention for their ability to model complex distributions with tractable sampling and inference. However, the error analysis for discrete diffusion models remains less well-understood. In this work, we propose a comprehensive framework for the error analysis of discrete diffusion models based on Lévy-type stochastic integrals. By generalizing the Poisson random measure to that with a time-independent and state-dependent intensity, we rigorously establish a stochastic integral formulation of discrete diffusion models and provide the corresponding change of measure theorems that are intriguingly analogous to Itô integrals and Girsanov's theorem for their continuous counterparts. Our framework unifies and strengthens the current theoretical results on discrete diffusion models and obtains the first error bound for the $\tau$-leaping scheme in KL divergence. With error sources clearly identified, our analysis gives new insight into the mathematical properties of discrete diffusion models and offers guidance for the design of efficient and accurate algorithms for real-world discrete diffusion model applications.

Title: Large Language Model Performance Benchmarking on Mobile Platforms: A Thorough Evaluation

• Authors: Jie Xiao, Qianyi Huang, Xu Chen, Chen Tian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03613
• Pdf URL: https://arxiv.org/pdf/2410.03613
• Copy Paste: [[2410.03613]] Large Language Model Performance Benchmarking on Mobile Platforms: A Thorough Evaluation(https://arxiv.org/abs/2410.03613)
• Keywords: privacy, large language model
• Abstract: As large language models (LLMs) increasingly integrate into every aspect of our work and daily lives, there are growing concerns about user privacy, which push the trend toward local deployment of these models. There are a number of lightweight LLMs (e.g., Gemini Nano, LLAMA2 7B) that can run locally on smartphones, providing users with greater control over their personal data. As a rapidly emerging application, we are concerned about their performance on commercial-off-the-shelf mobile devices. To fully understand the current landscape of LLM deployment on mobile platforms, we conduct a comprehensive measurement study on mobile devices. We evaluate both metrics that affect user experience, including token throughput, latency, and battery consumption, as well as factors critical to developers, such as resource utilization, DVFS strategies, and inference engines. In addition, we provide a detailed analysis of how these hardware capabilities and system dynamics affect on-device LLM performance, which may help developers identify and address bottlenecks for mobile LLM applications. We also provide comprehensive comparisons across the mobile system-on-chips (SoCs) from major vendors, highlighting their performance differences in handling LLM workloads. We hope that this study can provide insights for both the development of on-device LLMs and the design for future mobile system architecture.

Title: A Global Medical Data Security and Privacy Preserving Standards Identification Framework for Electronic Healthcare Consumers

• Authors: Vinaytosh Mishra, Kishu Gupta, Deepika Saxena, Ashutosh Kumar Singh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03621
• Pdf URL: https://arxiv.org/pdf/2410.03621
• Copy Paste: [[2410.03621]] A Global Medical Data Security and Privacy Preserving Standards Identification Framework for Electronic Healthcare Consumers(https://arxiv.org/abs/2410.03621)
• Keywords: security, privacy
• Abstract: Electronic Health Records (EHR) are crucial for the success of digital healthcare, with a focus on putting consumers at the center of this transformation. However, the digitalization of healthcare records brings along security and privacy risks for personal data. The major concern is that different countries have varying standards for the security and privacy of medical data. This paper proposed a novel and comprehensive framework to standardize these rules globally, bringing them together on a common platform. To support this proposal, the study reviews existing literature to understand the research interest in this issue. It also examines six key laws and standards related to security and privacy, identifying twenty concepts. The proposed framework utilized K-means clustering to categorize these concepts and identify five key factors. Finally, an Ordinal Priority Approach is applied to determine the preferred implementation of these factors in the context of EHRs. The proposed study provides a descriptive then prescriptive framework for the implementation of privacy and security in the context of electronic health records. Therefore, the findings of the proposed framework are useful for professionals and policymakers in improving the security and privacy associated with EHRs.

Title: Robust Offline Imitation Learning from Diverse Auxiliary Data

• Authors: Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li, Konstantinos Karydis, Amit K. Roy-Chowdhury
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03626
• Pdf URL: https://arxiv.org/pdf/2410.03626
• Copy Paste: [[2410.03626]] Robust Offline Imitation Learning from Diverse Auxiliary Data(https://arxiv.org/abs/2410.03626)
• Keywords: robust
• Abstract: Offline imitation learning enables learning a policy solely from a set of expert demonstrations, without any environment interaction. To alleviate the issue of distribution shift arising due to the small amount of expert data, recent works incorporate large numbers of auxiliary demonstrations alongside the expert data. However, the performance of these approaches rely on assumptions about the quality and composition of the auxiliary data. However, they are rarely successful when those assumptions do not hold. To address this limitation, we propose Robust Offline Imitation from Diverse Auxiliary Data (ROIDA). ROIDA first identifies high-quality transitions from the entire auxiliary dataset using a learned reward function. These high-reward samples are combined with the expert demonstrations for weighted behavioral cloning. For lower-quality samples, ROIDA applies temporal difference learning to steer the policy towards high-reward states, improving long-term returns. This two-pronged approach enables our framework to effectively leverage both high and low-quality data without any assumptions. Extensive experiments validate that ROIDA achieves robust and consistent performance across multiple auxiliary datasets with diverse ratios of expert and non-expert demonstrations. ROIDA effectively leverages unlabeled auxiliary data, outperforming prior methods reliant on specific data assumptions.

Title: Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models

• Authors: Chumeng Liang, Jiaxuan You
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03640
• Pdf URL: https://arxiv.org/pdf/2410.03640
• Copy Paste: [[2410.03640]] Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models(https://arxiv.org/abs/2410.03640)
• Keywords: attack, membership infer, fair, diffusion
• Abstract: Membership inference attacks (MIAs) on diffusion models have emerged as potential evidence of unauthorized data usage in training pre-trained diffusion models. These attacks aim to detect the presence of specific images in training datasets of diffusion models. Our study delves into the evaluation of state-of-the-art MIAs on diffusion models and reveals critical flaws and overly optimistic performance estimates in existing MIA evaluation. We introduce CopyMark, a more realistic MIA benchmark that distinguishes itself through the support for pre-trained diffusion models, unbiased datasets, and fair evaluation pipelines. Through extensive experiments, we demonstrate that the effectiveness of current MIA methods significantly degrades under these more practical conditions. Based on our results, we alert that MIA, in its current state, is not a reliable approach for identifying unauthorized data usage in pre-trained diffusion models. To the best of our knowledge, we are the first to discover the performance overestimation of MIAs on diffusion models and present a unified benchmark for more realistic evaluation. Our code is available on GitHub: \url{this https URL}.

Title: Aligning LLMs with Individual Preferences via Interaction

• Authors: Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, Heng Ji
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.03642
• Pdf URL: https://arxiv.org/pdf/2410.03642
• Copy Paste: [[2410.03642]] Aligning LLMs with Individual Preferences via Interaction(https://arxiv.org/abs/2410.03642)
• Keywords: large language model
• Abstract: As large language models (LLMs) demonstrate increasingly advanced capabilities, aligning their behaviors with human values and preferences becomes crucial for their wide adoption. While previous research focuses on general alignment to principles such as helpfulness, harmlessness, and honesty, the need to account for individual and diverse preferences has been largely overlooked, potentially undermining customized human experiences. To address this gap, we train LLMs that can ''interact to align'', essentially cultivating the meta-skill of LLMs to implicitly infer the unspoken personalized preferences of the current user through multi-turn conversations, and then dynamically align their following behaviors and responses to these inferred preferences. Our approach involves establishing a diverse pool of 3,310 distinct user personas by initially creating seed examples, which are then expanded through iterative self-generation and filtering. Guided by distinct user personas, we leverage multi-LLM collaboration to develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures. Finally, we apply supervised fine-tuning and reinforcement learning to enhance LLMs using this dataset. For evaluation, we establish the ALOE (ALign With CustOmized PrEferences) benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations. Experimental results demonstrate the effectiveness of our method in enabling dynamic, personalized alignment via interaction.

Title: Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need

• Authors: Xianlong Wang, Minghui Li, Wei Liu, Hangtao Zhang, Shengshan Hu, Yechao Zhang, Ziqi Zhou, Hai Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.03644
• Pdf URL: https://arxiv.org/pdf/2410.03644
• Copy Paste: [[2410.03644]] Unlearnable 3D Point Clouds: Class-wise Transformation Is All You Need(https://arxiv.org/abs/2410.03644)
• Keywords: protect
• Abstract: Traditional unlearnable strategies have been proposed to prevent unauthorized users from training on the 2D image data. With more 3D point cloud data containing sensitivity information, unauthorized usage of this new type data has also become a serious concern. To address this, we propose the first integral unlearnable framework for 3D point clouds including two processes: (i) we propose an unlearnable data protection scheme, involving a class-wise setting established by a category-adaptive allocation strategy and multi-transformations assigned to samples; (ii) we propose a data restoration scheme that utilizes class-wise inverse matrix transformation, thus enabling authorized-only training for unlearnable data. This restoration process is a practical issue overlooked in most existing unlearnable literature, \ie, even authorized users struggle to gain knowledge from 3D unlearnable data. Both theoretical and empirical results (including 6 datasets, 16 models, and 2 tasks) demonstrate the effectiveness of our proposed unlearnable framework. Our code is available at \url{this https URL}

Title: Dorami: Privilege Separating Security Monitor on RISC-V TEEs

• Authors: Mark Kuhne, Stavros Volos, Shweta Shinde
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.03653
• Pdf URL: https://arxiv.org/pdf/2410.03653
• Copy Paste: [[2410.03653]] Dorami: Privilege Separating Security Monitor on RISC-V TEEs(https://arxiv.org/abs/2410.03653)
• Keywords: security, protect, attack
• Abstract: TEE implementations on RISC-V offer an enclave abstraction by introducing a trusted component called the security monitor (SM). The SM performs critical tasks such as isolating enclaves from each other as well as from the OS by using privileged ISA instructions that enforce the physical memory protection. However, the SM executes at the highest privilege layer on the platform (machine-mode) along side firmware that is not only large in size but also includes third-party vendor code specific to the platform. In this paper, we present Dorami - a privilege separation approach that isolates the SM from the firmware thus reducing the attack surface on TEEs. Dorami re-purposes existing ISA features to enforce its isolation and achieves its goals without large overheads.

Title: Geometric Representation Condition Improves Equivariant Molecule Generation

• Authors: Zian Li, Cai Zhou, Xiyuan Wang, Xingang Peng, Muhan Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03655
• Pdf URL: https://arxiv.org/pdf/2410.03655
• Copy Paste: [[2410.03655]] Geometric Representation Condition Improves Equivariant Molecule Generation(https://arxiv.org/abs/2410.03655)
• Keywords: diffusion, generative
• Abstract: Recent advancements in molecular generative models have demonstrated substantial potential in accelerating scientific discovery, particularly in drug design. However, these models often face challenges in generating high-quality molecules, especially in conditional scenarios where specific molecular properties must be satisfied. In this work, we introduce GeoRCG, a general framework to enhance the performance of molecular generative models by integrating geometric representation conditions. We decompose the molecule generation process into two stages: first, generating an informative geometric representation; second, generating a molecule conditioned on the representation. Compared to directly generating a molecule, the relatively easy-to-generate representation in the first-stage guides the second-stage generation to reach a high-quality molecule in a more goal-oriented and much faster way. Leveraging EDM as the base generator, we observe significant quality improvements in unconditional molecule generation on the widely-used QM9 and GEOM-DRUG datasets. More notably, in the challenging conditional molecular generation task, our framework achieves an average 31\% performance improvement over state-of-the-art approaches, highlighting the superiority of conditioning on semantically rich geometric representations over conditioning on individual property values as in previous approaches. Furthermore, we show that, with such representation guidance, the number of diffusion steps can be reduced to as small as 100 while maintaining superior generation quality than that achieved with 1,000 steps, thereby significantly accelerating the generation process.

Title: RAFT: Realistic Attacks to Fool Text Detectors

• Authors: James Wang, Ran Li, Junfeng Yang, Chengzhi Mao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.03658
• Pdf URL: https://arxiv.org/pdf/2410.03658
• Copy Paste: [[2410.03658]] RAFT: Realistic Attacks to Fool Text Detectors(https://arxiv.org/abs/2410.03658)
• Keywords: attack, robust, large language model
• Abstract: Large language models (LLMs) have exhibited remarkable fluency across various tasks. However, their unethical applications, such as disseminating disinformation, have become a growing concern. Although recent works have proposed a number of LLM detection methods, their robustness and reliability remain unclear. In this paper, we present RAFT: a grammar error-free black-box attack against existing LLM detectors. In contrast to previous attacks for language models, our method exploits the transferability of LLM embeddings at the word-level while preserving the original text quality. We leverage an auxiliary embedding to greedily select candidate words to perturb against the target detector. Experiments reveal that our attack effectively compromises all detectors in the study across various domains by up to 99%, and are transferable across source models. Manual human evaluation studies show our attacks are realistic and indistinguishable from original human-written text. We also show that examples generated by RAFT can be used to train adversarially robust detectors. Our work shows that current LLM detectors are not adversarially robust, underscoring the urgent need for more resilient detection mechanisms.

Title: Enhance Reasoning by Learning from Mistakes: Peer-Review Knowledge Distillation from Multiple Large Language Models

• Authors: Zhuochun Li, Yuelyu Ji, Rui Meng, Daqing He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03663
• Pdf URL: https://arxiv.org/pdf/2410.03663
• Copy Paste: [[2410.03663]] Enhance Reasoning by Learning from Mistakes: Peer-Review Knowledge Distillation from Multiple Large Language Models(https://arxiv.org/abs/2410.03663)
• Keywords: large language model
• Abstract: Large language models (LLMs) have exhibited complex reasoning abilities by generating question rationales and demonstrated exceptional performance in natural language processing (NLP) tasks. However, these reasoning capabilities generally emerge in models with tens of billions of parameters, creating significant computational challenges for real-world deployment. Recent research has concentrated on improving open-source smaller models through knowledge distillation (KD) from commercial LLMs. Nevertheless, most of these studies rely solely on the responses from one single LLM as the gold rationale for training. In this paper, we introduce a novel Mistake-Aware Peer-Review Distillation (MAPD) approach: 1) Instead of merely obtaining gold rationales from teachers, our method asks teachers to identify and explain the student's mistakes, providing customized instruction learning data. 2) We design a simulated peer-review process between teacher LLMs, which selects only the generated rationales above the acceptance threshold. This reduces the chance of teachers guessing correctly with flawed rationale, improving instructional data quality. Comprehensive experiments and analysis on mathematical, commonsense, and logical reasoning tasks demonstrate the effectiveness of our method.

Title: Estimating Body and Hand Motion in an Ego-sensed World

• Authors: Brent Yi, Vickie Ye, Maya Zheng, Lea Müller, Georgios Pavlakos, Yi Ma, Jitendra Malik, Angjoo Kanazawa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.03665
• Pdf URL: https://arxiv.org/pdf/2410.03665
• Copy Paste: [[2410.03665]] Estimating Body and Hand Motion in an Ego-sensed World(https://arxiv.org/abs/2410.03665)
• Keywords: diffusion
• Abstract: We present EgoAllo, a system for human motion estimation from a head-mounted device. Using only egocentric SLAM poses and images, EgoAllo guides sampling from a conditional diffusion model to estimate 3D body pose, height, and hand parameters that capture the wearer's actions in the allocentric coordinate frame of the scene. To achieve this, our key insight is in representation: we propose spatial and temporal invariance criteria for improving model performance, from which we derive a head motion conditioning parameterization that improves estimation by up to 18%. We also show how the bodies estimated by our system can improve the hands: the resulting kinematic and temporal constraints result in over 40% lower hand estimation errors compared to noisy monocular estimates. Project page: this https URL