2024-09-19

Title: Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning

• Authors: Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, David Saltiel, Beatrice Guez
• Subjects: cs.CL, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2409.11408
• Pdf URL: https://arxiv.org/pdf/2409.11408
• Copy Paste: [[2409.11408]] Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning(https://arxiv.org/abs/2409.11408)
• Keywords: generative
• Abstract: In this paper, we demonstrate that non-generative, small-sized models such as FinBERT and FinDRoBERTa, when fine-tuned, can outperform GPT-3.5 and GPT-4 models in zero-shot learning settings in sentiment analysis for financial news. These fine-tuned models show comparable results to GPT-3.5 when it is fine-tuned on the task of determining market sentiment from daily financial news summaries sourced from Bloomberg. To fine-tune and compare these models, we created a novel database, which assigns a market score to each piece of news without human interpretation bias, systematically identifying the mentioned companies and analyzing whether their stocks have gone up, down, or remained neutral. Furthermore, the paper shows that the assumptions of Condorcet's Jury Theorem do not hold suggesting that fine-tuned small models are not independent of the fine-tuned GPT models, indicating behavioural similarities. Lastly, the resulted fine-tuned models are made publicly available on HuggingFace, providing a resource for further research in financial sentiment analysis and text classification.

Title: Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows

• Authors: Simone Marullo, Matteo Tiezzi, Marco Gori, Stefano Melacci
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11441
• Pdf URL: https://arxiv.org/pdf/2409.11441
• Copy Paste: [[2409.11441]] Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows(https://arxiv.org/abs/2409.11441)
• Keywords: self-supervised
• Abstract: Learning with neural networks from a continuous stream of visual information presents several challenges due to the non-i.i.d. nature of the data. However, it also offers novel opportunities to develop representations that are consistent with the information flow. In this paper we investigate the case of unsupervised continual learning of pixel-wise features subject to multiple motion-induced constraints, therefore named motion-conjugated feature representations. Differently from existing approaches, motion is not a given signal (either ground-truth or estimated by external modules), but is the outcome of a progressive and autonomous learning process, occurring at various levels of the feature hierarchy. Multiple motion flows are estimated with neural networks and characterized by different levels of abstractions, spanning from traditional optical flow to other latent signals originating from higher-level features, hence called higher-order motions. Continuously learning to develop consistent multi-order flows and representations is prone to trivial solutions, which we counteract by introducing a self-supervised contrastive loss, spatially-aware and based on flow-induced similarity. We assess our model on photorealistic synthetic streams and real-world videos, comparing to pre-trained state-of-the art feature extractors (also based on Transformers) and to recent unsupervised learning models, significantly outperforming these alternatives.

Title: Adaptive Anomaly Detection in Network Flows with Low-Rank Tensor Decompositions and Deep Unrolling

• Authors: Lukas Schynol, Marius Pesavento
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.11529
• Pdf URL: https://arxiv.org/pdf/2409.11529
• Copy Paste: [[2409.11529]] Adaptive Anomaly Detection in Network Flows with Low-Rank Tensor Decompositions and Deep Unrolling(https://arxiv.org/abs/2409.11529)
• Keywords: anomaly
• Abstract: Anomaly detection (AD) is increasingly recognized as a key component for ensuring the resilience of future communication systems. While deep learning has shown state-of-the-art AD performance, its application in critical systems is hindered by concerns regarding training data efficiency, domain adaptation and interpretability. This work considers AD in network flows using incomplete measurements, leveraging a robust tensor decomposition approach and deep unrolling techniques to address these challenges. We first propose a novel block-successive convex approximation algorithm based on a regularized model-fitting objective where the normal flows are modeled as low-rank tensors and anomalies as sparse. An augmentation of the objective is introduced to decrease the computational cost. We apply deep unrolling to derive a novel deep network architecture based on our proposed algorithm, treating the regularization parameters as learnable weights. Inspired by Bayesian approaches, we extend the model architecture to perform online adaptation to per-flow and per-time-step statistics, improving AD performance while maintaining a low parameter count and preserving the problem's permutation equivariances. To optimize the deep network weights for detection performance, we employ a homotopy optimization approach based on an efficient approximation of the area under the receiver operating characteristic curve. Extensive experiments on synthetic and real-world data demonstrate that our proposed deep network architecture exhibits a high training data efficiency, outperforms reference methods, and adapts seamlessly to varying network topologies.

Title: Balancing Optimality and Diversity: Human-Centered Decision Making through Generative Curation

• Authors: Michael Lingzhi Li, Shixiang Zhu
• Subjects: cs.LG, cs.HC, math.OC
• Abstract URL: https://arxiv.org/abs/2409.11535
• Pdf URL: https://arxiv.org/pdf/2409.11535
• Copy Paste: [[2409.11535]] Balancing Optimality and Diversity: Human-Centered Decision Making through Generative Curation(https://arxiv.org/abs/2409.11535)
• Keywords: generative
• Abstract: The surge in data availability has inundated decision-makers with an overwhelming array of choices. While existing approaches focus on optimizing decisions based on quantifiable metrics, practical decision-making often requires balancing measurable quantitative criteria with unmeasurable qualitative factors embedded in the broader context. In such cases, algorithms can generate high-quality recommendations, but the final decision rests with the human, who must weigh both dimensions. We define the process of selecting the optimal set of algorithmic recommendations in this context as human-centered decision making. To address this challenge, we introduce a novel framework called generative curation, which optimizes the true desirability of decision options by integrating both quantitative and qualitative aspects. Our framework uses a Gaussian process to model unknown qualitative factors and derives a diversity metric that balances quantitative optimality with qualitative diversity. This trade-off enables the generation of a manageable subset of diverse, near-optimal actions that are robust to unknown qualitative preferences. To operationalize this framework, we propose two implementation approaches: a generative neural network architecture that produces a distribution $\pi$ to efficiently sample a diverse set of near-optimal actions, and a sequential optimization method to iteratively generates solutions that can be easily incorporated into complex optimization formulations. We validate our approach with extensive datasets, demonstrating its effectiveness in enhancing decision-making processes across a range of complex environments, with significant implications for policy and management.

Title: Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey

• Authors: Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D. Yao, Shi-Xiong Zhang, Sambit Sahu
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.11564
• Pdf URL: https://arxiv.org/pdf/2409.11564
• Copy Paste: [[2409.11564]] Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey(https://arxiv.org/abs/2409.11564)
• Keywords: generative
• Abstract: Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth examination of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.

Title: HEARTS: A Holistic Framework for Explainable, Sustainable and Robust Text Stereotype Detection

• Authors: Theo King, Zekun Wu, Adriano Koshiyama, Emre Kazim, Philip Treleaven
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11579
• Pdf URL: https://arxiv.org/pdf/2409.11579
• Copy Paste: [[2409.11579]] HEARTS: A Holistic Framework for Explainable, Sustainable and Robust Text Stereotype Detection(https://arxiv.org/abs/2409.11579)
• Keywords: in-context
• Abstract: Stereotypes are generalised assumptions about societal groups, and even state-of-the-art LLMs using in-context learning struggle to identify them accurately. Due to the subjective nature of stereotypes, where what constitutes a stereotype can vary widely depending on cultural, social, and individual perspectives, robust explainability is crucial. Explainable models ensure that these nuanced judgments can be understood and validated by human users, promoting trust and accountability. We address these challenges by introducing HEARTS (Holistic Framework for Explainable, Sustainable, and Robust Text Stereotype Detection), a framework that enhances model performance, minimises carbon footprint, and provides transparent, interpretable explanations. We establish the Expanded Multi-Grain Stereotype Dataset (EMGSD), comprising 57,201 labeled texts across six groups, including under-represented demographics like LGBTQ+ and regional stereotypes. Ablation studies confirm that BERT models fine-tuned on EMGSD outperform those trained on individual components. We then analyse a fine-tuned, carbon-efficient ALBERT-V2 model using SHAP to generate token-level importance values, ensuring alignment with human understanding, and calculate explainability confidence scores by comparing SHAP and LIME outputs. Finally, HEARTS is applied to assess stereotypical bias in 12 LLM outputs, revealing a gradual reduction in bias over time within model families.

Title: ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering

• Authors: Priyesh Vakharia, Abigail Kufeldt, Max Meyers, Ian Lane, Leilani Gilpin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11589
• Pdf URL: https://arxiv.org/pdf/2409.11589
• Copy Paste: [[2409.11589]] ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering(https://arxiv.org/abs/2409.11589)
• Keywords: generative
• Abstract: Neurosymbolic approaches can add robustness to opaque neural systems by incorporating explainable symbolic representations. However, previous approaches have not used formal logic to contextualize queries to and validate outputs of large language models (LLMs). We propose \systemname{}, a novel neurosymbolic framework, to improve the robustness and reliability of LLMs in question-answering tasks. We provide \systemname{} with a domain-specific knowledge base, a logical reasoning system, and an integration to an existing LLM. This framework has two capabilities (1) context gathering: generating explainable and relevant context for a given query, and (2) validation: confirming and validating the factual accuracy of a statement in accordance with a knowledge base (KB). Our work opens a new area of neurosymbolic generative AI text validation and user personalization.

Title: Self-Contrastive Forward-Forward Algorithm

• Authors: Xing Chen, Dongshu Liu, Jeremie Laydevant, Julie Grollier
• Subjects: cs.LG, cs.AI, cs.CV, cs.ET, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.11593
• Pdf URL: https://arxiv.org/pdf/2409.11593
• Copy Paste: [[2409.11593]] Self-Contrastive Forward-Forward Algorithm(https://arxiv.org/abs/2409.11593)
• Keywords: self-supervised
• Abstract: The Forward-Forward (FF) algorithm is a recent, purely forward-mode learning method, that updates weights locally and layer-wise and supports supervised as well as unsupervised learning. These features make it ideal for applications such as brain-inspired learning, low-power hardware neural networks, and distributed learning in large models. However, while FF has shown promise on written digit recognition tasks, its performance on natural images and time-series remains a challenge. A key limitation is the need to generate high-quality negative examples for contrastive learning, especially in unsupervised tasks, where versatile solutions are currently lacking. To address this, we introduce the Self-Contrastive Forward-Forward (SCFF) method, inspired by self-supervised contrastive learning. SCFF generates positive and negative examples applicable across different datasets, surpassing existing local forward algorithms for unsupervised classification accuracy on MNIST (MLP: 98.7%), CIFAR-10 (CNN: 80.75%), and STL-10 (CNN: 77.3%). Additionally, SCFF is the first to enable FF training of recurrent neural networks, opening the door to more complex tasks and continuous-time video and text processing.

Title: Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model

• Authors: Derek Jollie, Jingmin Sun, Zecheng Zhang, Hayden Schaeffer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11609
• Pdf URL: https://arxiv.org/pdf/2409.11609
• Copy Paste: [[2409.11609]] Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model(https://arxiv.org/abs/2409.11609)
• Keywords: foundation model
• Abstract: Symbolic encoding has been used in multi-operator learning as a way to embed additional information for distinct time-series data. For spatiotemporal systems described by time-dependent partial differential equations, the equation itself provides an additional modality to identify the system. The utilization of symbolic expressions along side time-series samples allows for the development of multimodal predictive neural networks. A key challenge with current approaches is that the symbolic information, i.e. the equations, must be manually preprocessed (simplified, rearranged, etc.) to match and relate to the existing token library, which increases costs and reduces flexibility, especially when dealing with new differential equations. We propose a new token library based on SymPy to encode differential equations as an additional modality for time-series models. The proposed approach incurs minimal cost, is automated, and maintains high prediction accuracy for forecasting tasks. Additionally, we include a Bayesian filtering module that connects the different modalities to refine the learned equation. This improves the accuracy of the learned symbolic representation and the predicted time-series.

Title: PieClam: A Universal Graph Autoencoder Based on Overlapping Inclusive and Exclusive Communities

• Authors: Daniel Zilberg, Ron Levie
• Subjects: cs.LG, cs.SI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.11618
• Pdf URL: https://arxiv.org/pdf/2409.11618
• Copy Paste: [[2409.11618]] PieClam: A Universal Graph Autoencoder Based on Overlapping Inclusive and Exclusive Communities(https://arxiv.org/abs/2409.11618)
• Keywords: generative, anomaly
• Abstract: We propose PieClam (Prior Inclusive Exclusive Cluster Affiliation Model): a probabilistic graph model for representing any graph as overlapping generalized communities. Our method can be interpreted as a graph autoencoder: nodes are embedded into a code space by an algorithm that maximizes the log-likelihood of the decoded graph, given the input graph. PieClam is a community affiliation model that extends well-known methods like BigClam in two main manners. First, instead of the decoder being defined via pairwise interactions between the nodes in the code space, we also incorporate a learned prior on the distribution of nodes in the code space, turning our method into a graph generative model. Secondly, we generalize the notion of communities by allowing not only sets of nodes with strong connectivity, which we call inclusive communities, but also sets of nodes with strong disconnection, which we call exclusive communities. To model both types of communities, we propose a new type of decoder based the Lorentz inner product, which we prove to be much more expressive than standard decoders based on standard inner products or norm distances. By introducing a new graph similarity measure, that we call the log cut distance, we show that PieClam is a universal autoencoder, able to uniformly approximately reconstruct any graph. Our method is shown to obtain competitive performance in graph anomaly detection benchmarks.

Title: PainDiffusion: Can robot express pain?

• Authors: Quang Tien Dam, Tri Tung Nguyen Nguyen, Dinh Tuan Tran, Joo-Ho Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11635
• Pdf URL: https://arxiv.org/pdf/2409.11635
• Copy Paste: [[2409.11635]] PainDiffusion: Can robot express pain?(https://arxiv.org/abs/2409.11635)
• Keywords: diffusion
• Abstract: Pain is a more intuitive and user-friendly way of communicating problems, making it especially useful in rehabilitation nurse training robots. While most previous methods have focused on classifying or recognizing pain expressions, these approaches often result in unnatural, jiggling robot faces. We introduce PainDiffusion, a model that generates facial expressions in response to pain stimuli, with controllable pain expressiveness and emotion status. PainDiffusion leverages diffusion forcing to roll out predictions over arbitrary lengths using a conditioned temporal U-Net. It operates as a latent diffusion model within EMOCA's facial expression latent space, ensuring a compact data representation and quick rendering time. For training data, we process the BioVid Heatpain Database, extracting expression codes and subject identity configurations. We also propose a novel set of metrics to evaluate pain expressions, focusing on expressiveness, diversity, and the appropriateness of model-generated outputs. Finally, we demonstrate that PainDiffusion outperforms the autoregressive method, both qualitatively and quantitatively. Code, videos, and further analysis are available at: \href{this https URL}{this https URL}.

Title: Few-Shot Class-Incremental Learning with Non-IID Decentralized Data

• Authors: Cuiwei Liu, Siang Xu, Huaijun Qiu, Jing Zhang, Zhi Liu, Liang Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11657
• Pdf URL: https://arxiv.org/pdf/2409.11657
• Copy Paste: [[2409.11657]] Few-Shot Class-Incremental Learning with Non-IID Decentralized Data(https://arxiv.org/abs/2409.11657)
• Keywords: generative
• Abstract: Few-shot class-incremental learning is crucial for developing scalable and adaptive intelligent systems, as it enables models to acquire new classes with minimal annotated data while safeguarding the previously accumulated knowledge. Nonetheless, existing methods deal with continuous data streams in a centralized manner, limiting their applicability in scenarios that prioritize data privacy and security. To this end, this paper introduces federated few-shot class-incremental learning, a decentralized machine learning paradigm tailored to progressively learn new classes from scarce data distributed across multiple clients. In this learning paradigm, clients locally update their models with new classes while preserving data privacy, and then transmit the model updates to a central server where they are aggregated globally. However, this paradigm faces several issues, such as difficulties in few-shot learning, catastrophic forgetting, and data heterogeneity. To address these challenges, we present a synthetic data-driven framework that leverages replay buffer data to maintain existing knowledge and facilitate the acquisition of new knowledge. Within this framework, a noise-aware generative replay module is developed to fine-tune local models with a balance of new and replay data, while generating synthetic data of new classes to further expand the replay buffer for future tasks. Furthermore, a class-specific weighted aggregation strategy is designed to tackle data heterogeneity by adaptively aggregating class-specific parameters based on local models performance on synthetic data. This enables effective global model optimization without direct access to client data. Comprehensive experiments across three widely-used datasets underscore the effectiveness and preeminence of the introduced framework.

Title: RUIE: Retrieval-based Unified Information Extraction using Large Language Model

• Authors: Xincheng Liao, Junwen Duan, Yixi Huang, Jianxin Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11673
• Pdf URL: https://arxiv.org/pdf/2409.11673
• Copy Paste: [[2409.11673]] RUIE: Retrieval-based Unified Information Extraction using Large Language Model(https://arxiv.org/abs/2409.11673)
• Keywords: in-context
• Abstract: Unified information extraction (UIE) aims to complete all information extraction tasks using a single model or framework. While previous work has primarily focused on instruction-tuning large language models (LLMs) with constructed datasets, these methods require significant computational resources and struggle to generalize to unseen tasks. To address these limitations, we propose RUIE (Retrieval-based Unified Information Extraction), a framework that leverages in-context learning to enable rapid generalization while reducing computational costs. The key challenge in RUIE is selecting the most beneficial demonstrations for LLMs to effectively handle diverse IE tasks. To achieve this, we integrate LLM preferences for ranking candidate demonstrations and design a keyword-enhanced reward model to capture fine-grained relationships between queries and demonstrations. We then train a bi-encoder retriever for UIE through contrastive learning and knowledge distillation. To the best of our knowledge, RUIE is the first trainable retrieval framework for UIE. Experimental results on 8 held-out datasets demonstrate RUIE's effectiveness in generalizing to unseen tasks, with average F1-score improvements of 19.22 and 3.13 compared to instruction-tuning methods and other retrievers, respectively. Further analysis confirms RUIE's adaptability to LLMs of varying sizes and the importance of its key components.

Title: SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation

• Authors: Mingze Sun, Chen Guo, Puhua Jiang, Shiwei Mao, Yurun Chen, Ruqi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11682
• Pdf URL: https://arxiv.org/pdf/2409.11682
• Copy Paste: [[2409.11682]] SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation(https://arxiv.org/abs/2409.11682)
• Keywords: diffusion
• Abstract: In this paper, we propose SRIF, a novel Semantic shape Registration framework based on diffusion-based Image morphing and Flow estimation. More concretely, given a pair of extrinsically aligned shapes, we first render them from multi-views, and then utilize an image interpolation framework based on diffusion models to generate sequences of intermediate images between them. The images are later fed into a dynamic 3D Gaussian splatting framework, with which we reconstruct and post-process for intermediate point clouds respecting the image morphing processing. In the end, tailored for the above, we propose a novel registration module to estimate continuous normalizing flow, which deforms source shape consistently towards the target, with intermediate point clouds as weak guidance. Our key insight is to leverage large vision models (LVMs) to associate shapes and therefore obtain much richer semantic information on the relationship between shapes than the ad-hoc feature extraction and alignment. As a consequence, SRIF achieves high-quality dense correspondences on challenging shape pairs, but also delivers smooth, semantically meaningful interpolation in between. Empirical evidence justifies the effectiveness and superiority of our method as well as specific design choices. The code is released at this https URL.

Title: Recurrent Interpolants for Probabilistic Time Series Prediction

• Authors: Yu Chen, Marin Biloš, Sarthak Mittal, Wei Deng, Kashif Rasul, Anderson Schneider
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.11684
• Pdf URL: https://arxiv.org/pdf/2409.11684
• Copy Paste: [[2409.11684]] Recurrent Interpolants for Probabilistic Time Series Prediction(https://arxiv.org/abs/2409.11684)
• Keywords: diffusion, generative
• Abstract: Sequential models such as recurrent neural networks or transformer-based models became \textit{de facto} tools for multivariate time series forecasting in a probabilistic fashion, with applications to a wide range of datasets, such as finance, biology, medicine, etc. Despite their adeptness in capturing dependencies, assessing prediction uncertainty, and efficiency in training, challenges emerge in modeling high-dimensional complex distributions and cross-feature dependencies. To tackle these issues, recent works delve into generative modeling by employing diffusion or flow-based models. Notably, the integration of stochastic differential equations or probability flow successfully extends these methods to probabilistic time series imputation and forecasting. However, scalability issues necessitate a computational-friendly framework for large-scale generative model-based predictions. This work proposes a novel approach by blending the computational efficiency of recurrent neural networks with the high-quality probabilistic modeling of the diffusion model, which addresses challenges and advances generative models' application in time series forecasting. Our method relies on the foundation of stochastic interpolants and the extension to a broader conditional generation framework with additional control features, offering insights for future developments in this dynamic field.

Title: GUNet: A Graph Convolutional Network United Diffusion Model for Stable and Diversity Pose Generation

• Authors: Shuowen Liang, Sisi Li, Qingyun Wang, Cen Zhang, Kaiquan Zhu, Tian Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11689
• Pdf URL: https://arxiv.org/pdf/2409.11689
• Copy Paste: [[2409.11689]] GUNet: A Graph Convolutional Network United Diffusion Model for Stable and Diversity Pose Generation(https://arxiv.org/abs/2409.11689)
• Keywords: diffusion, generative
• Abstract: Pose skeleton images are an important reference in pose-controllable image generation. In order to enrich the source of skeleton images, recent works have investigated the generation of pose skeletons based on natural language. These methods are based on GANs. However, it remains challenging to perform diverse, structurally correct and aesthetically pleasing human pose skeleton generation with various textual inputs. To address this problem, we propose a framework with GUNet as the main model, PoseDiffusion. It is the first generative framework based on a diffusion model and also contains a series of variants fine-tuned based on a stable diffusion model. PoseDiffusion demonstrates several desired properties that outperform existing methods. 1) Correct Skeletons. GUNet, a denoising model of PoseDiffusion, is designed to incorporate graphical convolutional neural networks. It is able to learn the spatial relationships of the human skeleton by introducing skeletal information during the training process. 2) Diversity. We decouple the key points of the skeleton and characterise them separately, and use cross-attention to introduce textual conditions. Experimental results show that PoseDiffusion outperforms existing SoTA algorithms in terms of stability and diversity of text-driven pose skeleton generation. Qualitative analyses further demonstrate its superiority for controllable generation in Stable Diffusion.

Title: ORB-SfMLearner: ORB-Guided Self-supervised Visual Odometry with Selective Online Adaptation

• Authors: Yanlin Jin, Rui-Yang Ju, Haojun Liu, Yuzhong Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11692
• Pdf URL: https://arxiv.org/pdf/2409.11692
• Copy Paste: [[2409.11692]] ORB-SfMLearner: ORB-Guided Self-supervised Visual Odometry with Selective Online Adaptation(https://arxiv.org/abs/2409.11692)
• Keywords: self-supervised
• Abstract: Deep visual odometry, despite extensive research, still faces limitations in accuracy and generalizability that prevent its broader application. To address these challenges, we propose an Oriented FAST and Rotated BRIEF (ORB)-guided visual odometry with selective online adaptation named ORB-SfMLearner. We present a novel use of ORB features for learning-based ego-motion estimation, leading to more robust and accurate results. We also introduce the cross-attention mechanism to enhance the explainability of PoseNet and have revealed that driving direction of the vehicle can be explained through attention weights, marking a novel exploration in this area. To improve generalizability, our selective online adaptation allows the network to rapidly and selectively adjust to the optimal parameters across different domains. Experimental results on KITTI and vKITTI datasets show that our method outperforms previous state-of-the-art deep visual odometry methods in terms of ego-motion accuracy and generalizability.

Title: Free-VSC: Free Semantics from Visual Foundation Models for Unsupervised Video Semantic Compression

• Authors: Yuan Tian, Guo Lu, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11718
• Pdf URL: https://arxiv.org/pdf/2409.11718
• Copy Paste: [[2409.11718]] Free-VSC: Free Semantics from Visual Foundation Models for Unsupervised Video Semantic Compression(https://arxiv.org/abs/2409.11718)
• Keywords: foundation model
• Abstract: Unsupervised video semantic compression (UVSC), i.e., compressing videos to better support various analysis tasks, has recently garnered attention. However, the semantic richness of previous methods remains limited, due to the single semantic learning objective, limited training data, etc. To address this, we propose to boost the UVSC task by absorbing the off-the-shelf rich semantics from VFMs. Specifically, we introduce a VFMs-shared semantic alignment layer, complemented by VFM-specific prompts, to flexibly align semantics between the compressed video and various VFMs. This allows different VFMs to collaboratively build a mutually-enhanced semantic space, guiding the learning of the compression model. Moreover, we introduce a dynamic trajectory-based inter-frame compression scheme, which first estimates the semantic trajectory based on the historical content, and then traverses along the trajectory to predict the future semantics as the coding context. This reduces the overall bitcost of the system, further improving the compression efficiency. Our approach outperforms previous coding methods on three mainstream tasks and six datasets.

Title: Human-like Affective Cognition in Foundation Models

• Authors: Kanishk Gandhi, Zoe Lynch, Jan-Philipp Fränken, Kayla Patterson, Sharon Wambu, Tobias Gerstenberg, Desmond C. Ong, Noah D. Goodman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11733
• Pdf URL: https://arxiv.org/pdf/2409.11733
• Copy Paste: [[2409.11733]] Human-like Affective Cognition in Foundation Models(https://arxiv.org/abs/2409.11733)
• Keywords: foundation model
• Abstract: Understanding emotions is fundamental to human interaction and experience. Humans easily infer emotions from situations or facial expressions, situations from emotions, and do a variety of other \emph{affective cognition}. How adept is modern AI at these inferences? We introduce an evaluation framework for testing affective cognition in foundation models. Starting from psychological theory, we generate 1,280 diverse scenarios exploring relationships between appraisals, emotions, expressions, and outcomes. We evaluate the abilities of foundation models (GPT-4, Claude-3, Gemini-1.5-Pro) and humans (N = 567) across carefully selected conditions. Our results show foundation models tend to agree with human intuitions, matching or exceeding interparticipant agreement. In some conditions, models are ``superhuman'' -- they better predict modal human judgements than the average human. All models benefit from chain-of-thought reasoning. This suggests foundation models have acquired a human-like understanding of emotions and their influence on beliefs and behavior.

Title: InverseMeetInsert: Robust Real Image Editing via Geometric Accumulation Inversion in Guided Diffusion Models

• Authors: Yan Zheng, Lemeng Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11734
• Pdf URL: https://arxiv.org/pdf/2409.11734
• Copy Paste: [[2409.11734]] InverseMeetInsert: Robust Real Image Editing via Geometric Accumulation Inversion in Guided Diffusion Models(https://arxiv.org/abs/2409.11734)
• Keywords: diffusion
• Abstract: In this paper, we introduce Geometry-Inverse-Meet-Pixel-Insert, short for GEO, an exceptionally versatile image editing technique designed to cater to customized user requirements at both local and global scales. Our approach seamlessly integrates text prompts and image prompts to yield diverse and precise editing outcomes. Notably, our method operates without the need for training and is driven by two key contributions: (i) a novel geometric accumulation loss that enhances DDIM inversion to faithfully preserve pixel space geometry and layout, and (ii) an innovative boosted image prompt technique that combines pixel-level editing for text-only inversion with latent space geometry guidance for standard classifier-free reversion. Leveraging the publicly available Stable Diffusion model, our approach undergoes extensive evaluation across various image types and challenging prompt editing scenarios, consistently delivering high-fidelity editing results for real images.

Title: Knowledge Adaptation Network for Few-Shot Class-Incremental Learning

• Authors: Ye Wang, Yaxiong Wang, Guoshuai Zhao, Xueming Qian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11770
• Pdf URL: https://arxiv.org/pdf/2409.11770
• Copy Paste: [[2409.11770]] Knowledge Adaptation Network for Few-Shot Class-Incremental Learning(https://arxiv.org/abs/2409.11770)
• Keywords: foundation model
• Abstract: Few-shot class-incremental learning (FSCIL) aims to incrementally recognize new classes using a few samples while maintaining the performance on previously learned classes. One of the effective methods to solve this challenge is to construct prototypical evolution classifiers. Despite the advancement achieved by most existing methods, the classifier weights are simply initialized using mean features. Because representations for new classes are weak and biased, we argue such a strategy is suboptimal. In this paper, we tackle this issue from two aspects. Firstly, thanks to the development of foundation models, we employ a foundation model, the CLIP, as the network pedestal to provide a general representation for each class. Secondly, to generate a more reliable and comprehensive instance representation, we propose a Knowledge Adapter (KA) module that summarizes the data-specific knowledge from training data and fuses it into the general representation. Additionally, to tune the knowledge learned from the base classes to the upcoming classes, we propose a mechanism of Incremental Pseudo Episode Learning (IPEL) by simulating the actual FSCIL. Taken together, our proposed method, dubbed as Knowledge Adaptation Network (KANet), achieves competitive performance on a wide range of datasets, including CIFAR100, CUB200, and ImageNet-R.

Title: The Factuality of Large Language Models in the Legal Domain

• Authors: Rajaa El Hamdani, Thomas Bonald, Fragkiskos Malliaros, Nils Holzenberger, Fabian Suchanek
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11798
• Pdf URL: https://arxiv.org/pdf/2409.11798
• Copy Paste: [[2409.11798]] The Factuality of Large Language Models in the Legal Domain(https://arxiv.org/abs/2409.11798)
• Keywords: in-context
• Abstract: This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%.

Title: Latent fingerprint enhancement for accurate minutiae detection

• Authors: Abdul Wahab, Tariq Mahmood Khan, Shahzaib Iqbal, Bandar AlShammari, Bandar Alhaqbani, Imran Razzak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11802
• Pdf URL: https://arxiv.org/pdf/2409.11802
• Copy Paste: [[2409.11802]] Latent fingerprint enhancement for accurate minutiae detection(https://arxiv.org/abs/2409.11802)
• Keywords: generative
• Abstract: Identification of suspects based on partial and smudged fingerprints, commonly referred to as fingermarks or latent fingerprints, presents a significant challenge in the field of fingerprint recognition. Although fixed-length embeddings have shown effectiveness in recognising rolled and slap fingerprints, the methods for matching latent fingerprints have primarily centred around local minutiae-based embeddings, failing to fully exploit global representations for matching purposes. Consequently, enhancing latent fingerprints becomes critical to ensuring robust identification for forensic investigations. Current approaches often prioritise restoring ridge patterns, overlooking the fine-macroeconomic details crucial for accurate fingerprint recognition. To address this, we propose a novel approach that uses generative adversary networks (GANs) to redefine Latent Fingerprint Enhancement (LFE) through a structured approach to fingerprint generation. By directly optimising the minutiae information during the generation process, the model produces enhanced latent fingerprints that exhibit exceptional fidelity to ground-truth instances. This leads to a significant improvement in identification performance. Our framework integrates minutiae locations and orientation fields, ensuring the preservation of both local and structural fingerprint features. Extensive evaluations conducted on two publicly available datasets demonstrate our method's dominance over existing state-of-the-art techniques, highlighting its potential to significantly enhance latent fingerprint recognition accuracy in forensic applications.

Title: Constraint Guided AutoEncoders for Joint Optimization of Condition Indicator Estimation and Anomaly Detection in Machine Condition Monitoring

• Authors: Maarten Meire, Quinten Van Baelen, Ted Ooijevaar, Peter Karsmakers
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11807
• Pdf URL: https://arxiv.org/pdf/2409.11807
• Copy Paste: [[2409.11807]] Constraint Guided AutoEncoders for Joint Optimization of Condition Indicator Estimation and Anomaly Detection in Machine Condition Monitoring(https://arxiv.org/abs/2409.11807)
• Keywords: anomaly
• Abstract: The main goal of machine condition monitoring is, as the name implies, to monitor the condition of industrial applications. The objective of this monitoring can be mainly split into two problems. A diagnostic problem, where normal data should be distinguished from anomalous data, otherwise called Anomaly Detection (AD), or a prognostic problem, where the aim is to predict the evolution of a Condition Indicator (CI) that reflects the condition of an asset throughout its life time. When considering machine condition monitoring, it is expected that this CI shows a monotonic behavior, as the condition of a machine gradually degrades over time. This work proposes an extension to Constraint Guided AutoEncoders (CGAE), which is a robust AD method, that enables building a single model that can be used for both AD and CI estimation. For the purpose of improved CI estimation the extension incorporates a constraint that enforces the model to have monotonically increasing CI predictions over time. Experimental results indicate that the proposed algorithm performs similar, or slightly better, than CGAE, with regards to AD, while improving the monotonic behavior of the CI.

Title: ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images

• Authors: Abhinaw Jagtap, Nachiket Tapas, R. G. Brajesh
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.11874
• Pdf URL: https://arxiv.org/pdf/2409.11874
• Copy Paste: [[2409.11874]] ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images(https://arxiv.org/abs/2409.11874)
• Keywords: diffusion, generative
• Abstract: In the fast-evolving field of Generative AI, platforms like MidJourney, DALL-E, and Stable Diffusion have transformed Text-to-Image (T2I) Generation. However, despite their impressive ability to create high-quality images, they often struggle to generate accurate text within these images. Theoretically, if we could achieve accurate text generation in AI images in a ``zero-shot'' manner, it would not only make AI-generated images more meaningful but also democratize the graphic design industry. The first step towards this goal is to create a robust scoring matrix for evaluating text accuracy in AI-generated images. Although there are existing bench-marking methods like CLIP SCORE and T2I-CompBench++, there's still a gap in systematically evaluating text and typography in AI-generated images, especially with diffusion-based methods. In this paper, we introduce a novel evaluation matrix designed explicitly for quantifying the performance of text and typography generation within AI-generated images. We have used letter by letter matching strategy to compute the exact matching scores from the reference text to the AI generated text. Our novel approach to calculate the score takes care of multiple redundancies such as repetition of words, case sensitivity, mixing of words, irregular incorporation of letters etc. Moreover, we have developed a Novel method named as brevity adjustment to handle excess text. In addition we have also done a quantitative analysis of frequent errors arise due to frequently used words and less frequently used words. Project page is available at: this https URL.

Title: Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction

• Authors: Caihong Wang, Du Xu, Zonghang Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.11890
• Pdf URL: https://arxiv.org/pdf/2409.11890
• Copy Paste: [[2409.11890]] Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction(https://arxiv.org/abs/2409.11890)
• Keywords: anomaly
• Abstract: In the era of rapid Internet development, log data has become indispensable for recording the operations of computer devices and software. These data provide valuable insights into system behavior and necessitate thorough analysis. Recent advances in text analysis have enabled deep learning to achieve significant breakthroughs in log anomaly detection. However, the high cost of manual annotation and the dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. It leverages the expressive power of large models for log content analysis and the capability of graph structures to encapsulate correlations between logs. It retains key log information while integrating the causal relationships between logs to achieve effective feature extraction. Additionally, we introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor. By employing graph clustering algorithms for log anomaly detection, Log2graphs enables the identification of abnormal logs without the need for labeled data. We comprehensively evaluate the feature extraction capability of DualGCN-LogAE and the anomaly detection performance of Log2graphs using public log datasets across five different scenarios. Our evaluation metrics include detection accuracy and graph clustering quality scores. Experimental results demonstrate that the log features extracted by DualGCN-LogAE outperform those obtained by other methods on classic classifiers. Moreover, Log2graphs surpasses existing unsupervised log detection methods, providing a robust tool for advancing log anomaly detection research.

Title: Multi-Grid Graph Neural Networks with Self-Attention for Computational Mechanics

• Authors: Paul Garnier, Jonathan Viquerat, Elie Hachem
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2409.11899
• Pdf URL: https://arxiv.org/pdf/2409.11899
• Copy Paste: [[2409.11899]] Multi-Grid Graph Neural Networks with Self-Attention for Computational Mechanics(https://arxiv.org/abs/2409.11899)
• Keywords: self-supervised
• Abstract: Advancement in finite element methods have become essential in various disciplines, and in particular for Computational Fluid Dynamics (CFD), driving research efforts for improved precision and efficiency. While Convolutional Neural Networks (CNNs) have found success in CFD by mapping meshes into images, recent attention has turned to leveraging Graph Neural Networks (GNNs) for direct mesh processing. This paper introduces a novel model merging Self-Attention with Message Passing in GNNs, achieving a 15\% reduction in RMSE on the well known flow past a cylinder benchmark. Furthermore, a dynamic mesh pruning technique based on Self-Attention is proposed, that leads to a robust GNN-based multigrid approach, also reducing RMSE by 15\%. Additionally, a new self-supervised training method based on BERT is presented, resulting in a 25\% RMSE reduction. The paper includes an ablation study and outperforms state-of-the-art models on several challenging datasets, promising advancements similar to those recently achieved in natural language and image processing. Finally, the paper introduces a dataset with meshes larger than existing ones by at least an order of magnitude. Code and Datasets will be released at this https URL.

Title: Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation

• Authors: Dimitrios Christodoulou, Mads Kuhlmann-Jørgensen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11904
• Pdf URL: https://arxiv.org/pdf/2409.11904
• Copy Paste: [[2409.11904]] Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation(https://arxiv.org/abs/2409.11904)
• Keywords: diffusion
• Abstract: Efficiently evaluating the performance of text-to-image models is difficult as it inherently requires subjective judgment and human preference, making it hard to compare different models and quantify the state of the art. Leveraging Rapidata's technology, we present an efficient annotation framework that sources human feedback from a diverse, global pool of annotators. Our study collected over 2 million annotations across 4,512 images, evaluating four prominent models (DALL-E 3, Flux.1, MidJourney, and Stable Diffusion) on style preference, coherence, and text-to-image alignment. We demonstrate that our approach makes it feasible to comprehensively rank image generation models based on a vast pool of annotators and show that the diverse annotator demographics reflect the world population, significantly decreasing the risk of biases.

Title: LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Foundation Models

• Authors: Amaia Cardiel, Eloi Zablocki, Oriane Siméoni, Elias Ramzi, Matthieu Cord
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11919
• Pdf URL: https://arxiv.org/pdf/2409.11919
• Copy Paste: [[2409.11919]] LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Foundation Models(https://arxiv.org/abs/2409.11919)
• Keywords: foundation model
• Abstract: Vision Language Models (VLMs) have shown impressive performances on numerous tasks but their zero-shot capabilities can be limited compared to dedicated or fine-tuned models. Yet, fine-tuning VLMs comes with limitations as it requires `white-box' access to the model's architecture and weights as well as expertise to design the fine-tuning objectives and optimize the hyper-parameters, which are specific to each VLM and downstream task. In this work, we propose LLM-wrapper, a novel approach to adapt VLMs in a `black-box' manner by leveraging large language models (LLMs) so as to reason on their outputs. We demonstrate the effectiveness of LLM-wrapper on Referring Expression Comprehension (REC), a challenging open-vocabulary task that requires spatial and semantic reasoning. Our approach significantly boosts the performance of off-the-shelf models, resulting in competitive results when compared with classic fine-tuning.

Title: Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models

• Authors: Lorenzo Mandelli, Stefano Berretti
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11920
• Pdf URL: https://arxiv.org/pdf/2409.11920
• Copy Paste: [[2409.11920]] Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models(https://arxiv.org/abs/2409.11920)
• Keywords: diffusion
• Abstract: In this paper, we address the challenge of generating realistic 3D human motions for action classes that were never seen during the training phase. Our approach involves decomposing complex actions into simpler movements, specifically those observed during training, by leveraging the knowledge of human motion contained in GPTs models. These simpler movements are then combined into a single, realistic animation using the properties of diffusion models. Our claim is that this decomposition and subsequent recombination of simple movements can synthesize an animation that accurately represents the complex input action. This method operates during the inference phase and can be integrated with any pre-trained diffusion model, enabling the synthesis of motion classes not present in the training data. We evaluate our method by dividing two benchmark human motion datasets into basic and complex actions, and then compare its performance against the state-of-the-art.

Title: MitoSeg: Mitochondria Segmentation Tool

• Authors: Faris Serdar Taşel, Efe Çiftci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11974
• Pdf URL: https://arxiv.org/pdf/2409.11974
• Copy Paste: [[2409.11974]] MitoSeg: Mitochondria Segmentation Tool(https://arxiv.org/abs/2409.11974)
• Keywords: generative
• Abstract: Recent studies suggest a potential link between the physical structure of mitochondria and neurodegenerative diseases. With advances in Electron Microscopy techniques, it has become possible to visualize the boundary and internal membrane structures of mitochondria in detail. It is crucial to automatically segment mitochondria from these images to investigate the relationship between mitochondria and diseases. In this paper, we present a software solution for mitochondrial segmentation, highlighting mitochondria boundaries in electron microscopy tomography images and generating corresponding 3D meshes.

Title: ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation

• Authors: Peiyu Li, Xiaobao Huang, Yijun Tian, Nitesh V. Chawla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12010
• Pdf URL: https://arxiv.org/pdf/2409.12010
• Copy Paste: [[2409.12010]] ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation(https://arxiv.org/abs/2409.12010)
• Keywords: foundation model
• Abstract: Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub.

Title: LEMON: Localized Editing with Mesh Optimization and Neural Shaders

• Authors: Furkan Mert Algan, Umut Yazgan, Driton Salihu, Cem Eteke, Eckehard Steinbach
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12024
• Pdf URL: https://arxiv.org/pdf/2409.12024
• Copy Paste: [[2409.12024]] LEMON: Localized Editing with Mesh Optimization and Neural Shaders(https://arxiv.org/abs/2409.12024)
• Keywords: diffusion
• Abstract: In practical use cases, polygonal mesh editing can be faster than generating new ones, but it can still be challenging and time-consuming for users. Existing solutions for this problem tend to focus on a single task, either geometry or novel view synthesis, which often leads to disjointed results between the mesh and view. In this work, we propose LEMON, a mesh editing pipeline that combines neural deferred shading with localized mesh optimization. Our approach begins by identifying the most important vertices in the mesh for editing, utilizing a segmentation model to focus on these key regions. Given multi-view images of an object, we optimize a neural shader and a polygonal mesh while extracting the normal map and the rendered image from each view. By using these outputs as conditioning data, we edit the input images with a text-to-image diffusion model and iteratively update our dataset while deforming the mesh. This process results in a polygonal mesh that is edited according to the given text instruction, preserving the geometric characteristics of the initial mesh while focusing on the most significant areas. We evaluate our pipeline using the DTU dataset, demonstrating that it generates finely-edited meshes more rapidly than the current state-of-the-art methods. We include our code and additional results in the supplementary material.

Title: On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery

• Authors: BW Sheffield, Jeffrey Ellen, Ben Whitmore
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12026
• Pdf URL: https://arxiv.org/pdf/2409.12026
• Copy Paste: [[2409.12026]] On Vision Transformers for Classification Tasks in Side-Scan Sonar Imagery(https://arxiv.org/abs/2409.12026)
• Keywords: self-supervised
• Abstract: Side-scan sonar (SSS) imagery presents unique challenges in the classification of man-made objects on the seafloor due to the complex and varied underwater environments. Historically, experts have manually interpreted SSS images, relying on conventional machine learning techniques with hand-crafted features. While Convolutional Neural Networks (CNNs) significantly advanced automated classification in this domain, they often fall short when dealing with diverse seafloor textures, such as rocky or ripple sand bottoms, where false positive rates may increase. Recently, Vision Transformers (ViTs) have shown potential in addressing these limitations by utilizing a self-attention mechanism to capture global information in image patches, offering more flexibility in processing spatial hierarchies. This paper rigorously compares the performance of ViT models alongside commonly used CNN architectures, such as ResNet and ConvNext, for binary classification tasks in SSS imagery. The dataset encompasses diverse geographical seafloor types and is balanced between the presence and absence of man-made objects. ViT-based models exhibit superior classification performance across f1-score, precision, recall, and accuracy metrics, although at the cost of greater computational resources. CNNs, with their inductive biases, demonstrate better computational efficiency, making them suitable for deployment in resource-constrained environments like underwater vehicles. Future research directions include exploring self-supervised learning for ViTs and multi-modal fusion to further enhance performance in challenging underwater environments.

Title: Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidance

• Authors: Jaehoon Joo, Taejin Jeong, Seongjae Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12099
• Pdf URL: https://arxiv.org/pdf/2409.12099
• Copy Paste: [[2409.12099]] Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidance(https://arxiv.org/abs/2409.12099)
• Keywords: diffusion, generative
• Abstract: Understanding how humans process visual information is one of the crucial steps for unraveling the underlying mechanism of brain activity. Recently, this curiosity has motivated the fMRI-to-image reconstruction task; given the fMRI data from visual stimuli, it aims to reconstruct the corresponding visual stimuli. Surprisingly, leveraging powerful generative models such as the Latent Diffusion Model (LDM) has shown promising results in reconstructing complex visual stimuli such as high-resolution natural images from vision datasets. Despite the impressive structural fidelity of these reconstructions, they often lack details of small objects, ambiguous shapes, and semantic nuances. Consequently, the incorporation of additional semantic knowledge, beyond mere visuals, becomes imperative. In light of this, we exploit how modern LDMs effectively incorporate multi-modal guidance (text guidance, visual guidance, and image layout) for structurally and semantically plausible image generations. Specifically, inspired by the two-streams hypothesis suggesting that perceptual and semantic information are processed in different brain regions, our framework, Brain-Streams, maps fMRI signals from these brain regions to appropriate embeddings. That is, by extracting textual guidance from semantic information regions and visual guidance from perceptual information regions, Brain-Streams provides accurate multi-modal guidance to LDMs. We validate the reconstruction ability of Brain-Streams both quantitatively and qualitatively on a real fMRI dataset comprising natural image stimuli and fMRI data.

Title: Measuring Human and AI Values based on Generative Psychometrics with Large Language Models

• Authors: Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, Guojie Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12106
• Pdf URL: https://arxiv.org/pdf/2409.12106
• Copy Paste: [[2409.12106]] Measuring Human and AI Values based on Generative Psychometrics with Large Language Models(https://arxiv.org/abs/2409.12106)
• Keywords: generative
• Abstract: Human values and their measurement are long-standing interdisciplinary inquiry. Recent advances in AI have sparked renewed interest in this area, with large language models (LLMs) emerging as both tools and subjects of value measurement. This work introduces Generative Psychometrics for Values (GPV), an LLM-based, data-driven value measurement paradigm, theoretically grounded in text-revealed selective perceptions. We begin by fine-tuning an LLM for accurate perception-level value measurement and verifying the capability of LLMs to parse texts into perceptions, forming the core of the GPV pipeline. Applying GPV to human-authored blogs, we demonstrate its stability, validity, and superiority over prior psychological tools. Then, extending GPV to LLM value measurement, we advance the current art with 1) a psychometric methodology that measures LLM values based on their scalable and free-form outputs, enabling context-specific measurement; 2) a comparative analysis of measurement paradigms, indicating response biases of prior methods; and 3) an attempt to bridge LLM values and their safety, revealing the predictive power of different value systems and the impacts of various values on LLM safety. Through interdisciplinary efforts, we aim to leverage AI for next-generation psychometrics and psychometrics for value-aligned AI.

Title: MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion

• Authors: Kalakonda Sai Shashank, Shubh Maheshwari, Ravi Kiran Sarvadevabhatla
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.12140
• Pdf URL: https://arxiv.org/pdf/2409.12140
• Copy Paste: [[2409.12140]] MoRAG -- Multi-Fusion Retrieval Augmented Generation for Human Motion(https://arxiv.org/abs/2409.12140)
• Keywords: diffusion
• Abstract: We introduce MoRAG, a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs), we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore, by utilizing low-level, part-specific motion information, we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module, improving the performance of motion diffusion models. Code, pretrained models and sample videos will be made available at: this https URL

Title: JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation

• Authors: Sai Tanmay Reddy Chakkera, Aggelina Chatziagapi, Dimitris Samaras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.12156
• Pdf URL: https://arxiv.org/pdf/2409.12156
• Copy Paste: [[2409.12156]] JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation(https://arxiv.org/abs/2409.12156)
• Keywords: self-supervised
• Abstract: We introduce a novel method for joint expression and audio-guided talking face generation. Recent approaches either struggle to preserve the speaker identity or fail to produce faithful facial expressions. To address these challenges, we propose a NeRF-based network. Since we train our network on monocular videos without any ground truth, it is essential to learn disentangled representations for audio and expression. We first learn audio features in a self-supervised manner, given utterances from multiple subjects. By incorporating a contrastive learning technique, we ensure that the learned audio features are aligned to the lip motion and disentangled from the muscle motion of the rest of the face. We then devise a transformer-based architecture that learns expression features, capturing long-range facial expressions and disentangling them from the speech-specific mouth movements. Through quantitative and qualitative evaluation, we demonstrate that our method can synthesize high-fidelity talking face videos, achieving state-of-the-art facial expression transfer along with lip synchronization to unseen audio.

Title: NSSR-DIL: Null-Shot Image Super-Resolution Using Deep Identity Learning

• Authors: Sree Rama Vamsidhar S, Rama Krishna Gorthi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12165
• Pdf URL: https://arxiv.org/pdf/2409.12165
• Copy Paste: [[2409.12165]] NSSR-DIL: Null-Shot Image Super-Resolution Using Deep Identity Learning(https://arxiv.org/abs/2409.12165)
• Keywords: self-supervised
• Abstract: The present State-of-the-Art (SotA) Image Super-Resolution (ISR) methods employ Deep Learning (DL) techniques using a large amount of image data. The primary limitation to extending the existing SotA ISR works for real-world instances is their computational and time complexities. In this paper, contrary to the existing methods, we present a novel and computationally efficient ISR algorithm that is independent of the image dataset to learn the ISR task. The proposed algorithm reformulates the ISR task from generating the Super-Resolved (SR) images to computing the inverse of the kernels that span the degradation space. We introduce Deep Identity Learning, exploiting the identity relation between the degradation and inverse degradation models. The proposed approach neither relies on the ISR dataset nor on a single input low-resolution (LR) image (like the self-supervised method i.e. ZSSR) to model the ISR task. Hence we term our model as Null-Shot Super-Resolution Using Deep Identity Learning (NSSR-DIL). The proposed NSSR-DIL model requires fewer computational resources, at least by an order of 10, and demonstrates a competitive performance on benchmark ISR datasets. Another salient aspect of our proposition is that the NSSR-DIL framework detours retraining the model and remains the same for varying scale factors like X2, X3, and X4. This makes our highly efficient ISR model more suitable for real-world applications.

Title: Expanding Expressivity in Transformer Models with M\"obiusAttention

• Authors: Anna-Maria Halacheva, Mojtaba Nayyeri, Steffen Staab
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.12175
• Pdf URL: https://arxiv.org/pdf/2409.12175
• Copy Paste: [[2409.12175]] Expanding Expressivity in Transformer Models with M\"obiusAttention(https://arxiv.org/abs/2409.12175)
• Keywords: foundation model
• Abstract: Attention mechanisms and Transformer architectures have revolutionized Natural Language Processing (NLP) by enabling exceptional modeling of long-range dependencies and capturing intricate linguistic patterns. However, their inherent reliance on linear operations in the form of matrix multiplications limits their ability to fully capture inter-token relationships on their own. We propose MöbiusAttention, a novel approach that integrates Möbius transformations within the attention mechanism of Transformer-based models. Möbius transformations are non-linear operations in spaces over complex numbers with the ability to map between various geometries. By incorporating these properties, MöbiusAttention empowers models to learn more intricate geometric relationships between tokens and capture a wider range of information through complex-valued weight vectors. We build and pre-train a BERT and a RoFormer version enhanced with MöbiusAttention, which we then finetune on the GLUE benchmark. We evaluate empirically our approach against the baseline BERT and RoFormer models on a range of downstream tasks. Our approach compares favorably against the baseline models, even with smaller number of parameters suggesting the enhanced expressivity of MöbiusAttention. This research paves the way for exploring the potential of Möbius transformations in the complex projective space to enhance the expressivity and performance of foundation models.

Title: A Controlled Study on Long Context Extension and Generalization in LLMs

• Authors: Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, Alexander M. Rush
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12181
• Pdf URL: https://arxiv.org/pdf/2409.12181
• Copy Paste: [[2409.12181]] A Controlled Study on Long Context Extension and Generalization in LLMs(https://arxiv.org/abs/2409.12181)
• Keywords: in-context
• Abstract: Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. However, owing to differences in data and model classes, it has been challenging to compare these approaches, leading to uncertainty as to how to evaluate long-context performance and whether it differs from standard evaluation. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data. Our study yields several insights into long-context behavior. First, we reaffirm the critical role of perplexity as a general-purpose performance indicator even in longer-context tasks. Second, we find that current approximate attention methods systematically underperform across long-context tasks. Finally, we confirm that exact fine-tuning based methods are generally effective within the range of their extension, whereas extrapolation remains challenging. All codebases, models, and checkpoints will be made available open-source, promoting transparency and facilitating further research in this critical area of AI development.

Title: Massively Multi-Person 3D Human Motion Forecasting with Scene Context

• Authors: Felix B Mueller, Julian Tanke, Juergen Gall
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.12189
• Pdf URL: https://arxiv.org/pdf/2409.12189
• Copy Paste: [[2409.12189]] Massively Multi-Person 3D Human Motion Forecasting with Scene Context(https://arxiv.org/abs/2409.12189)
• Keywords: diffusion
• Abstract: Forecasting long-term 3D human motion is challenging: the stochasticity of human behavior makes it hard to generate realistic human motion from the input sequence alone. Information on the scene environment and the motion of nearby people can greatly aid the generation process. We propose a scene-aware social transformer model (SAST) to forecast long-term (10s) human motion motion. Unlike previous models, our approach can model interactions between both widely varying numbers of people and objects in a scene. We combine a temporal convolutional encoder-decoder architecture with a Transformer-based bottleneck that allows us to efficiently combine motion and scene information. We model the conditional motion distribution using denoising diffusion models. We benchmark our approach on the Humans in Kitchens dataset, which contains 1 to 16 persons and 29 to 50 objects that are visible simultaneously. Our model outperforms other approaches in terms of realism and diversity on different metrics and in a user study. Code is available at this https URL.

Title: Vista3D: Unravel the 3D Darkside of a Single Image

• Authors: Qiuhong Shen, Xingyi Yang, Michael Bi Mi, Xinchao Wang
• Subjects: cs.CV, cs.AI, cs.GT, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.12193
• Pdf URL: https://arxiv.org/pdf/2409.12193
• Copy Paste: [[2409.12193]] Vista3D: Unravel the 3D Darkside of a Single Image(https://arxiv.org/abs/2409.12193)
• Keywords: diffusion
• Abstract: We embark on the age-old quest: unveiling the hidden dimensions of objects from mere glimpses of their visible parts. To address this, we present Vista3D, a framework that realizes swift and consistent 3D generation within a mere 5 minutes. At the heart of Vista3D lies a two-phase approach: the coarse phase and the fine phase. In the coarse phase, we rapidly generate initial geometry with Gaussian Splatting from a single image. In the fine phase, we extract a Signed Distance Function (SDF) directly from learned Gaussian Splatting, optimizing it with a differentiable isosurface representation. Furthermore, it elevates the quality of generation by using a disentangled representation with two independent implicit functions to capture both visible and obscured aspects of objects. Additionally, it harmonizes gradients from 2D diffusion prior with 3D-aware diffusion priors by angular diffusion prior composition. Through extensive evaluation, we demonstrate that Vista3D effectively sustains a balance between the consistency and diversity of the generated 3D objects. Demos and code will be available at this https URL.