2024-02-01

Title: Customizing Language Model Responses with Contrastive In-Context Learning

• Authors: Xiang Gao, Kamalika Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.17390
• Pdf URL: https://arxiv.org/pdf/2401.17390
• Copy Paste: [[2401.17390]] Customizing Language Model Responses with Contrastive In-Context Learning(https://arxiv.org/abs/2401.17390)
• Keywords: in-context
• Abstract: Large language models (LLMs) are becoming increasingly important for machine learning applications. However, it can be challenging to align LLMs with our intent, particularly when we want to generate content that is preferable over others or when we want the LLM to respond in a certain style or tone that is hard to describe. To address this challenge, we propose an approach that uses contrastive examples to better describe our intent. This involves providing positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid. The negative examples can be retrieved from labeled data, written by a human, or generated by the LLM itself. Before generating an answer, we ask the model to analyze the examples to teach itself what to avoid. This reasoning step provides the model with the appropriate articulation of the user's need and guides it towards generting a better answer. We tested our approach on both synthesized and real-world datasets, including StackExchange and Reddit, and found that it significantly improves performance compared to standard few-shot prompting

Title: Superiority of Multi-Head Attention in In-Context Linear Regression

• Authors: Yingqian Cui, Jie Ren, Pengfei He, Jiliang Tang, Yue Xing
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.17426
• Pdf URL: https://arxiv.org/pdf/2401.17426
• Copy Paste: [[2401.17426]] Superiority of Multi-Head Attention in In-Context Linear Regression(https://arxiv.org/abs/2401.17426)
• Keywords: in-context
• Abstract: We present a theoretical analysis of the performance of transformer with softmax attention in in-context learning with linear regression tasks. While the existing literature predominantly focuses on the convergence of transformers with single-/multi-head attention, our research centers on comparing their performance. We conduct an exact theoretical analysis to demonstrate that multi-head attention with a substantial embedding dimension performs better than single-head attention. When the number of in-context examples D increases, the prediction loss using single-/multi-head attention is in O(1/D), and the one for multi-head attention has a smaller multiplicative constant. In addition to the simplest data distribution setting, we consider more scenarios, e.g., noisy labels, local examples, correlated features, and prior knowledge. We observe that, in general, multi-head attention is preferred over single-head attention. Our results verify the effectiveness of the design of multi-head attention in the transformer architecture.

Title: Towards Visual Syntactical Understanding

• Authors: Sayeed Shafayet Chowdhury, Soumyadeep Chandra, Kaushik Roy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17497
• Pdf URL: https://arxiv.org/pdf/2401.17497
• Copy Paste: [[2401.17497]] Towards Visual Syntactical Understanding(https://arxiv.org/abs/2401.17497)
• Keywords: anomaly
• Abstract: Syntax is usually studied in the realm of linguistics and refers to the arrangement of words in a sentence. Similarly, an image can be considered as a visual 'sentence', with the semantic parts of the image acting as 'words'. While visual syntactic understanding occurs naturally to humans, it is interesting to explore whether deep neural networks (DNNs) are equipped with such reasoning. To that end, we alter the syntax of natural images (e.g. swapping the eye and nose of a face), referred to as 'incorrect' images, to investigate the sensitivity of DNNs to such syntactic anomaly. Through our experiments, we discover an intriguing property of DNNs where we observe that state-of-the-art convolutional neural networks, as well as vision transformers, fail to discriminate between syntactically correct and incorrect images when trained on only correct ones. To counter this issue and enable visual syntactic understanding with DNNs, we propose a three-stage framework- (i) the 'words' (or the sub-features) in the image are detected, (ii) the detected words are sequentially masked and reconstructed using an autoencoder, (iii) the original and reconstructed parts are compared at each location to determine syntactic correctness. The reconstruction module is trained with BERT-like masked autoencoding for images, with the motivation to leverage language model inspired training to better capture the syntax. Note, our proposed approach is unsupervised in the sense that the incorrect images are only used during testing and the correct versus incorrect labels are never used for training. We perform experiments on CelebA, and AFHQ datasets and obtain classification accuracy of 92.10%, and 90.89%, respectively. Notably, the approach generalizes well to ImageNet samples which share common classes with CelebA and AFHQ without explicitly training on them.

Title: FEUDA: Frustratingly Easy Prompt Based Unsupervised Domain Adaptation

• Authors: Rheeya Uppaal, Yixuan Li, Junjie Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17514
• Pdf URL: https://arxiv.org/pdf/2401.17514
• Copy Paste: [[2401.17514]] FEUDA: Frustratingly Easy Prompt Based Unsupervised Domain Adaptation(https://arxiv.org/abs/2401.17514)
• Keywords: self-supervised
• Abstract: A major thread of unsupervised domain adaptation (UDA) methods uses unlabeled data from both source and target domains to learn domain-invariant representations for adaptation. However, these methods showcase certain limitations, encouraging the use of self-supervised learning through continued pre-training. The necessity of continued pre-training or learning domain-invariant representations is still unclear in the prompt-based classification framework, where an input example is modified by a template and then fed into a language model (LM) to generate a label string. To examine this new paradigm of UDA in the prompt-based setup, we propose a frustratingly easy UDA method (FEUDA) that trains an autoregressive LM on both unlabeled and labeled examples using two different instruction-tuning tasks. Specifically, the first task trains the LM on unlabeled texts from both domains via masked language modeling (MLM), and the other uses supervised instruction-tuning on source-labeled data for classification. We conduct extensive experiments on 24 real-world domain pairs to show the effectiveness of our method over strong domain-invariant learning methods. Our analysis sheds light on why masked language modeling improves target-domain classification performance in prompt-based UDA. We discover that MLM helps the model learn both semantic and background knowledge of a domain, which are both beneficial for downstream classification.

Title: Game-Theoretic Unlearnable Example Generator

• Authors: Shuang Liu, Yihan Wang, Xiao-Shan Gao
• Subjects: cs.LG, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.17523
• Pdf URL: https://arxiv.org/pdf/2401.17523
• Copy Paste: [[2401.17523]] Game-Theoretic Unlearnable Example Generator(https://arxiv.org/abs/2401.17523)
• Keywords: generative
• Abstract: Unlearnable example attacks are data poisoning attacks aiming to degrade the clean test accuracy of deep learning by adding imperceptible perturbations to the training samples, which can be formulated as a bi-level optimization problem. However, directly solving this optimization problem is intractable for deep neural networks. In this paper, we investigate unlearnable example attacks from a game-theoretic perspective, by formulating the attack as a nonzero sum Stackelberg game. First, the existence of game equilibria is proved under the normal setting and the adversarial training setting. It is shown that the game equilibrium gives the most powerful poison attack in that the victim has the lowest test accuracy among all networks within the same hypothesis space, when certain loss functions are used. Second, we propose a novel attack method, called the Game Unlearnable Example (GUE), which has three main gradients. (1) The poisons are obtained by directly solving the equilibrium of the Stackelberg game with a first-order algorithm. (2) We employ an autoencoder-like generative network model as the poison attacker. (3) A novel payoff function is introduced to evaluate the performance of the poison. Comprehensive experiments demonstrate that GUE can effectively poison the model in various scenarios. Furthermore, the GUE still works by using a relatively small percentage of the training data to train the generator, and the poison generator can generalize to unseen data well. Our implementation code can be found at https://github.com/hong-xian/gue.

Title: Enhancing Score-Based Sampling Methods with Ensembles

• Authors: Tobias Bischoff, Bryan Riel
• Subjects: cs.LG, stat.CO
• Abstract URL: https://arxiv.org/abs/2401.17539
• Pdf URL: https://arxiv.org/pdf/2401.17539
• Copy Paste: [[2401.17539]] Enhancing Score-Based Sampling Methods with Ensembles(https://arxiv.org/abs/2401.17539)
• Keywords: diffusion, generative
• Abstract: We introduce ensembles within score-based sampling methods to develop gradient-free approximate sampling techniques that leverage the collective dynamics of particle ensembles to compute approximate reverse diffusion drifts. We introduce the underlying methodology, emphasizing its relationship with generative diffusion models and the previously introduced F\"ollmer sampler. We demonstrate the efficacy of ensemble strategies through various examples, ranging from low- to medium-dimensionality sampling problems, including multi-modal and highly non-Gaussian probability distributions, and provide comparisons to traditional methods like NUTS. Our findings highlight the potential of ensemble strategies for modeling complex probability distributions in situations where gradients are unavailable. Finally, we showcase its application in the context of Bayesian inversion problems within the geophysical sciences.

Title: Task-Oriented Diffusion Model Compression

• Authors: Geonung Kim, Beomsu Kim, Eunhyeok Park, Sunghyun Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17547
• Pdf URL: https://arxiv.org/pdf/2401.17547
• Copy Paste: [[2401.17547]] Task-Oriented Diffusion Model Compression(https://arxiv.org/abs/2401.17547)
• Keywords: diffusion
• Abstract: As recent advancements in large-scale Text-to-Image (T2I) diffusion models have yielded remarkable high-quality image generation, diverse downstream Image-to-Image (I2I) applications have emerged. Despite the impressive results achieved by these I2I models, their practical utility is hampered by their large model size and the computational burden of the iterative denoising process. In this paper, we explore the compression potential of these I2I models in a task-oriented manner and introduce a novel method for reducing both model size and the number of timesteps. Through extensive experiments, we observe key insights and use our empirical knowledge to develop practical solutions that aim for near-optimal results with minimal exploration costs. We validate the effectiveness of our method by applying it to InstructPix2Pix for image editing and StableSR for image restoration. Our approach achieves satisfactory output quality with 39.2% and 56.4% reduction in model footprint and 81.4% and 68.7% decrease in latency to InstructPix2Pix and StableSR, respectively.

Title: Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning

• Authors: Yuelyu Ji, Zeshui Yu, Yanshan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17602
• Pdf URL: https://arxiv.org/pdf/2401.17602
• Copy Paste: [[2401.17602]] Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning(https://arxiv.org/abs/2401.17602)
• Keywords: in-context
• Abstract: In this study, we aim to address the task of assertion detection when extracting medical concepts from clinical notes, a key process in clinical natural language processing (NLP). Assertion detection in clinical NLP usually involves identifying assertion types for medical concepts in the clinical text, namely certainty (whether the medical concept is positive, negated, possible, or hypothetical), temporality (whether the medical concept is for present or the past history), and experiencer (whether the medical concept is described for the patient or a family member). These assertion types are essential for healthcare professionals to quickly and clearly understand the context of medical conditions from unstructured clinical texts, directly influencing the quality and outcomes of patient care. Although widely used, traditional methods, particularly rule-based NLP systems and machine learning or deep learning models, demand intensive manual efforts to create patterns and tend to overlook less common assertion types, leading to an incomplete understanding of the context. To address this challenge, our research introduces a novel methodology that utilizes Large Language Models (LLMs) pre-trained on a vast array of medical data for assertion detection. We enhanced the current method with advanced reasoning techniques, including Tree of Thought (ToT), Chain of Thought (CoT), and Self-Consistency (SC), and refine it further with Low-Rank Adaptation (LoRA) fine-tuning. We first evaluated the model on the i2b2 2010 assertion dataset. Our method achieved a micro-averaged F-1 of 0.89, with 0.11 improvements over the previous works. To further assess the generalizability of our approach, we extended our evaluation to a local dataset that focused on sleep concept extraction. Our approach achieved an F-1 of 0.74, which is 0.31 higher than the previous method.

Title: Topology-Aware Latent Diffusion for 3D Shape Generation

• Authors: Jiangbei Hu, Ben Fei, Baixin Xu, Fei Hou, Weidong Yang, Shengfa Wang, Na Lei, Chen Qian, Ying He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17603
• Pdf URL: https://arxiv.org/pdf/2401.17603
• Copy Paste: [[2401.17603]] Topology-Aware Latent Diffusion for 3D Shape Generation(https://arxiv.org/abs/2401.17603)
• Keywords: diffusion, generative
• Abstract: We introduce a new generative model that combines latent diffusion with persistent homology to create 3D shapes with high diversity, with a special emphasis on their topological characteristics. Our method involves representing 3D shapes as implicit fields, then employing persistent homology to extract topological features, including Betti numbers and persistence diagrams. The shape generation process consists of two steps. Initially, we employ a transformer-based autoencoding module to embed the implicit representation of each 3D shape into a set of latent vectors. Subsequently, we navigate through the learned latent space via a diffusion model. By strategically incorporating topological features into the diffusion process, our generative module is able to produce a richer variety of 3D shapes with different topological structures. Furthermore, our framework is flexible, supporting generation tasks constrained by a variety of inputs, including sparse and partial point clouds, as well as sketches. By modifying the persistence diagrams, we can alter the topology of the shapes generated from these input modalities.

Title: Graph Multi-Similarity Learning for Molecular Property Prediction

• Authors: Hao Xu, Zhengyang Zhou, Pengyu Hong
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2401.17615
• Pdf URL: https://arxiv.org/pdf/2401.17615
• Copy Paste: [[2401.17615]] Graph Multi-Similarity Learning for Molecular Property Prediction(https://arxiv.org/abs/2401.17615)
• Keywords: self-supervised
• Abstract: Effective molecular representation learning is essential for molecular property prediction. Contrastive learning, a prominent self-supervised approach for molecular representation learning, relies on establishing positive and negative pairs. However, this binary similarity categorization oversimplifies the nature of complex molecular relationships and overlooks the degree of relative similarities among molecules, posing challenges to the effectiveness and generality of representation learning. In response to this challenge, we propose the Graph Multi-Similarity Learning for Molecular Property Prediction (GraphMSL) framework. GraphMSL incorporates a generalized multi-similarity metric in a continuous scale, capturing self-similarity and relative similarities. The unimodal multi-similarity metrics are derived from various chemical modalities, and the fusion of these metrics into a multimodal form significantly enhances the effectiveness of GraphMSL. In addition, the flexibility of fusion function can reshape the focus of the model to convey different chemical semantics. GraphMSL proves effective in drug discovery evaluations through various downstream tasks and post-hoc analysis of learnt representations. Its notable performance suggests significant potential for the exploration of new drug candidates.

Title: Unveiling the Power of Self-supervision for Multi-view Multi-human Association and Tracking

• Authors: Wei Feng, Feifan Wang, Ruize Han, Zekun Qian, Song Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.17617
• Pdf URL: https://arxiv.org/pdf/2401.17617
• Copy Paste: [[2401.17617]] Unveiling the Power of Self-supervision for Multi-view Multi-human Association and Tracking(https://arxiv.org/abs/2401.17617)
• Keywords: self-supervised
• Abstract: Multi-view multi-human association and tracking (MvMHAT), is a new but important problem for multi-person scene video surveillance, aiming to track a group of people over time in each view, as well as to identify the same person across different views at the same time, which is different from previous MOT and multi-camera MOT tasks only considering the over-time human tracking. This way, the videos for MvMHAT require more complex annotations while containing more information for self learning. In this work, we tackle this problem with a self-supervised learning aware end-to-end network. Specifically, we propose to take advantage of the spatial-temporal self-consistency rationale by considering three properties of reflexivity, symmetry and transitivity. Besides the reflexivity property that naturally holds, we design the self-supervised learning losses based on the properties of symmetry and transitivity, for both appearance feature learning and assignment matrix optimization, to associate the multiple humans over time and across views. Furthermore, to promote the research on MvMHAT, we build two new large-scale benchmarks for the network training and testing of different algorithms. Extensive experiments on the proposed benchmarks verify the effectiveness of our method. We have released the benchmark and code to the public.

Title: Spatial-and-Frequency-aware Restoration method for Images based on Diffusion Models

• Authors: Kyungsung Lee, Donggyu Lee, Myungjoo Kang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17629
• Pdf URL: https://arxiv.org/pdf/2401.17629
• Copy Paste: [[2401.17629]] Spatial-and-Frequency-aware Restoration method for Images based on Diffusion Models(https://arxiv.org/abs/2401.17629)
• Keywords: diffusion
• Abstract: Diffusion models have recently emerged as a promising framework for Image Restoration (IR), owing to their ability to produce high-quality reconstructions and their compatibility with established methods. Existing methods for solving noisy inverse problems in IR, considers the pixel-wise data-fidelity. In this paper, we propose SaFaRI, a spatial-and-frequency-aware diffusion model for IR with Gaussian noise. Our model encourages images to preserve data-fidelity in both the spatial and frequency domains, resulting in enhanced reconstruction quality. We comprehensively evaluate the performance of our model on a variety of noisy inverse problems, including inpainting, denoising, and super-resolution. Our thorough evaluation demonstrates that SaFaRI achieves state-of-the-art performance on both the ImageNet datasets and FFHQ datasets, outperforming existing zero-shot IR methods in terms of LPIPS and FID metrics.

Title: What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis

• Authors: Takanori Ashihara, Marc Delcroix, Takafumi Moriya, Kohei Matsuura, Taichi Asami, Yusuke Ijima
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.17632
• Pdf URL: https://arxiv.org/pdf/2401.17632
• Copy Paste: [[2401.17632]] What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis(https://arxiv.org/abs/2401.17632)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) has attracted increased attention for learning meaningful speech representations. Speech SSL models, such as WavLM, employ masked prediction training to encode general-purpose representations. In contrast, speaker SSL models, exemplified by DINO-based models, adopt utterance-level training objectives primarily for speaker representation. Understanding how these models represent information is essential for refining model efficiency and effectiveness. Unlike the various analyses of speech SSL, there has been limited investigation into what information speaker SSL captures and how its representation differs from speech SSL or other fully-supervised speaker models. This paper addresses these fundamental questions. We explore the capacity to capture various speech properties by applying SUPERB evaluation probing tasks to speech and speaker SSL models. We also examine which layers are predominantly utilized for each task to identify differences in how speech is represented. Furthermore, we conduct direct comparisons to measure the similarities between layers within and across models. Our analysis unveils that 1) the capacity to represent content information is somewhat unrelated to enhanced speaker representation, 2) specific layers of speech SSL models would be partly specialized in capturing linguistic information, and 3) speaker SSL models tend to disregard linguistic information but exhibit more sophisticated speaker representation.

Title: A primer on synthetic health data

• Authors: Jennifer Anne Bartell, Sander Boisen Valentin, Anders Krogh, Henning Langberg, Martin Bøgsted
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17653
• Pdf URL: https://arxiv.org/pdf/2401.17653
• Copy Paste: [[2401.17653]] A primer on synthetic health data(https://arxiv.org/abs/2401.17653)
• Keywords: generative
• Abstract: Recent advances in deep generative models have greatly expanded the potential to create realistic synthetic health datasets. These synthetic datasets aim to preserve the characteristics, patterns, and overall scientific conclusions derived from sensitive health datasets without disclosing patient identity or sensitive information. Thus, synthetic data can facilitate safe data sharing that supports a range of initiatives including the development of new predictive models, advanced health IT platforms, and general project ideation and hypothesis development. However, many questions and challenges remain, including how to consistently evaluate a synthetic dataset's similarity and predictive utility in comparison to the original real dataset and risk to privacy when shared. Additional regulatory and governance issues have not been widely addressed. In this primer, we map the state of synthetic health data, including generation and evaluation methods and tools, existing examples of deployment, the regulatory and ethical landscape, access and governance options, and opportunities for further development.

Title: An attempt to generate new bridge types from latent space of energy-based model

• Authors: Hongjun Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.17657
• Pdf URL: https://arxiv.org/pdf/2401.17657
• Copy Paste: [[2401.17657]] An attempt to generate new bridge types from latent space of energy-based model(https://arxiv.org/abs/2401.17657)
• Keywords: generative
• Abstract: Use energy-based model for bridge-type innovation. The loss function is explained by the game theory, the logic is clear and the formula is simple and clear. Thus avoid the use of maximum likelihood estimation to explain the loss function and eliminate the need for Monte Carlo methods to solve the normalized denominator. Assuming that the bridge-type population follows a Boltzmann distribution, a neural network is constructed to represent the energy function. Use Langevin dynamics technology to generate a new sample with low energy value, thus a generative model of bridge-type based on energy is established. Train energy function on symmetric structured image dataset of three span beam bridge, arch bridge, cable-stayed bridge, and suspension bridge to accurately calculate the energy values of real and fake samples. Sampling from latent space, using gradient descent algorithm, the energy function transforms the sampling points into low energy score samples, thereby generating new bridge types different from the dataset. Due to unstable and slow training in this attempt, the possibility of generating new bridge types is rare and the image definition of generated images is low.

Title: Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generation

• Authors: Yuanhuiyi Lyu, Xu Zheng, Lin Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.17664
• Pdf URL: https://arxiv.org/pdf/2401.17664
• Copy Paste: [[2401.17664]] Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generation(https://arxiv.org/abs/2401.17664)
• Keywords: diffusion, generative
• Abstract: The multifaceted nature of human perception and comprehension indicates that, when we think, our body can naturally take any combination of senses, a.k.a., modalities and form a beautiful picture in our brain. For example, when we see a cattery and simultaneously perceive the cat's purring sound, our brain can construct a picture of a cat in the cattery. Intuitively, generative AI models should hold the versatility of humans and be capable of generating images from any combination of modalities efficiently and collaboratively. This paper presents ImgAny, a novel end-to-end multi-modal generative model that can mimic human reasoning and generate high-quality images. Our method serves as the first attempt in its capacity of efficiently and flexibly taking any combination of seven modalities, ranging from language, audio to vision modalities, including image, point cloud, thermal, depth, and event data. Our key idea is inspired by human-level cognitive processes and involves the integration and harmonization of multiple input modalities at both the entity and attribute levels without specific tuning across modalities. Accordingly, our method brings two novel training-free technical branches: 1) Entity Fusion Branch ensures the coherence between inputs and outputs. It extracts entity features from the multi-modal representations powered by our specially constructed entity knowledge graph; 2) Attribute Fusion Branch adeptly preserves and processes the attributes. It efficiently amalgamates distinct attributes from diverse input modalities via our proposed attribute knowledge graph. Lastly, the entity and attribute features are adaptively fused as the conditional inputs to the pre-trained Stable Diffusion model for image generation. Extensive experiments under diverse modality combinations demonstrate its exceptional capability for visual content creation.

Title: Enhancing Large Language Model with Decomposed Reasoning for Emotion Cause Pair Extraction

• Authors: Jialiang Wu, Yi Shen, Ziheng Zhang, Longjun Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17716
• Pdf URL: https://arxiv.org/pdf/2401.17716
• Copy Paste: [[2401.17716]] Enhancing Large Language Model with Decomposed Reasoning for Emotion Cause Pair Extraction(https://arxiv.org/abs/2401.17716)
• Keywords: in-context
• Abstract: Emotion-Cause Pair Extraction (ECPE) involves extracting clause pairs representing emotions and their causes in a document. Existing methods tend to overfit spurious correlations, such as positional bias in existing benchmark datasets, rather than capturing semantic features. Inspired by recent work, we explore leveraging large language model (LLM) to address ECPE task without additional training. Despite strong capabilities, LLMs suffer from uncontrollable outputs, resulting in mediocre performance. To address this, we introduce chain-of-thought to mimic human cognitive process and propose the Decomposed Emotion-Cause Chain (DECC) framework. Combining inducing inference and logical pruning, DECC guides LLMs to tackle ECPE task. We further enhance the framework by incorporating in-context learning. Experiment results demonstrate the strength of DECC compared to state-of-the-art supervised fine-tuning methods. Finally, we analyze the effectiveness of each component and the robustness of the method in various scenarios, including different LLM bases, rebalanced datasets, and multi-pair extraction.

Title: M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval

• Authors: Xingning Dong, Zipeng Feng, Chunluan Zhou, Xuzheng Yu, Ming Yang, Qingpei Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17797
• Pdf URL: https://arxiv.org/pdf/2401.17797
• Copy Paste: [[2401.17797]] M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval(https://arxiv.org/abs/2401.17797)
• Keywords: foundation model
• Abstract: We present a Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards effective and efficient zero-shot video-text retrieval, dubbed M2-RAAP. Upon popular image-text models like CLIP, most current adaptation-based video-text pre-training methods are confronted by three major issues, i.e., noisy data corpus, time-consuming pre-training, and limited performance gain. Towards this end, we conduct a comprehensive study including four critical steps in video-text pre-training. Specifically, we investigate 1) data filtering and refinement, 2) video input type selection, 3) temporal modeling, and 4) video feature enhancement. We then summarize this empirical study into the M2-RAAP recipe, where our technical contributions lie in 1) the data filtering and text re-writing pipeline resulting in 1M high-quality bilingual video-text pairs, 2) the replacement of video inputs with key-frames to accelerate pre-training, and 3) the Auxiliary-Caption-Guided (ACG) strategy to enhance video features. We conduct extensive experiments by adapting three image-text foundation models on two refined video-text datasets from different languages, validating the robustness and reproducibility of M2-RAAP for adaptation-based pre-training. Results demonstrate that M2-RAAP yields superior performance with significantly reduced data (-90%) and time consumption (-95%), establishing a new SOTA on four English zero-shot retrieval datasets and two Chinese ones. We are preparing our refined bilingual data annotations and codebase, which will be available at https://github.com/alipay/Ant-Multi-Modal-Framework/tree/main/prj/M2_RAAP.

Title: Advances in 3D Generation: A Survey

• Authors: Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, Ying Shan
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.17807
• Pdf URL: https://arxiv.org/pdf/2401.17807
• Copy Paste: [[2401.17807]] Advances in 3D Generation: A Survey(https://arxiv.org/abs/2401.17807)
• Keywords: generative
• Abstract: Generating 3D models lies at the core of computer graphics and has been the focus of decades of research. With the emergence of advanced neural representations and generative models, the field of 3D content generation is developing rapidly, enabling the creation of increasingly high-quality and diverse 3D models. The rapid growth of this field makes it difficult to stay abreast of all recent developments. In this survey, we aim to introduce the fundamental methodologies of 3D generation methods and establish a structured roadmap, encompassing 3D representation, generation methods, datasets, and corresponding applications. Specifically, we introduce the 3D representations that serve as the backbone for 3D generation. Furthermore, we provide a comprehensive overview of the rapidly growing literature on generation methods, categorized by the type of algorithmic paradigms, including feedforward generation, optimization-based generation, procedural generation, and generative novel view synthesis. Lastly, we discuss available datasets, applications, and open challenges. We hope this survey will help readers explore this exciting topic and foster further advancements in the field of 3D content generation.

Title: Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis

• Authors: Jianing Li, Xi Nan, Ming Lu, Li Du, Shanghang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17862
• Pdf URL: https://arxiv.org/pdf/2401.17862
• Copy Paste: [[2401.17862]] Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis(https://arxiv.org/abs/2401.17862)
• Keywords: in-context
• Abstract: Multi-modal large language models (MLLMs) have demonstrated remarkable vision-language capabilities, primarily due to the exceptional in-context understanding and multi-task learning strengths of large language models (LLMs). The advent of visual instruction tuning has further enhanced MLLMs' performance in vision-language understanding. However, while existing MLLMs adeptly recognize \textit{what} objects are in an image, they still face challenges in effectively discerning \textit{where} these objects are, particularly along the distance (scene depth) axis. To overcome this limitation in MLLMs, we introduce Proximity Question Answering (Proximity QA), a novel framework designed to enable MLLMs to infer the proximity relationship between objects in images. The framework operates in two phases: the first phase focuses on guiding the models to understand the relative depth of objects, and the second phase further encourages the models to infer the proximity relationships between objects based on their depth perceptions. We also propose a VQA dataset called Proximity-110K, containing additional instructions that incorporate depth information and the proximity relationships of objects. We have conducted extensive experiments to validate Proximity QA's superior ability in depth perception and proximity analysis, outperforming other state-of-the-art MLLMs. Code and dataset will be released at \textcolor{magenta}{https://github.com/NorthSummer/ProximityQA.git}.

Title: Efficient Subseasonal Weather Forecast using Teleconnection-informed Transformers

• Authors: Shan Zhao, Zhitong Xiong, Xiao Xiang Zhu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.17870
• Pdf URL: https://arxiv.org/pdf/2401.17870
• Copy Paste: [[2401.17870]] Efficient Subseasonal Weather Forecast using Teleconnection-informed Transformers(https://arxiv.org/abs/2401.17870)
• Keywords: foundation model
• Abstract: Subseasonal forecasting, which is pivotal for agriculture, water resource management, and early warning of disasters, faces challenges due to the chaotic nature of the atmosphere. Recent advances in machine learning (ML) have revolutionized weather forecasting by achieving competitive predictive skills to numerical models. However, training such foundation models requires thousands of GPU days, which causes substantial carbon emissions and limits their broader applicability. Moreover, ML models tend to fool the pixel-wise error scores by producing smoothed results which lack physical consistency and meteorological meaning. To deal with the aforementioned problems, we propose a teleconnection-informed transformer. Our architecture leverages the pretrained Pangu model to achieve good initial weights and integrates a teleconnection-informed temporal module to improve predictability in an extended temporal range. Remarkably, by adjusting 1.1% of the Pangu model's parameters, our method enhances predictability on four surface and five upper-level atmospheric variables at a two-week lead time. Furthermore, the teleconnection-filtered features improve the spatial granularity of outputs significantly, indicating their potential physical consistency. Our research underscores the importance of atmospheric and oceanic teleconnections in driving future weather conditions. Besides, it presents a resource-efficient pathway for researchers to leverage existing foundation models on versatile downstream tasks.

Title: AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error

• Authors: Jonas Ricker, Denis Lukovnikov, Asja Fischer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17879
• Pdf URL: https://arxiv.org/pdf/2401.17879
• Copy Paste: [[2401.17879]] AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error(https://arxiv.org/abs/2401.17879)
• Keywords: diffusion
• Abstract: With recent text-to-image models, anyone can generate deceptively realistic images with arbitrary contents, fueling the growing threat of visual disinformation. A key enabler for generating high-resolution images with low computational cost has been the development of latent diffusion models (LDMs). In contrast to conventional diffusion models, LDMs perform the denoising process in the low-dimensional latent space of a pre-trained autoencoder (AE) instead of the high-dimensional image space. Despite their relevance, the forensic analysis of LDMs is still in its infancy. In this work we propose AEROBLADE, a novel detection method which exploits an inherent component of LDMs: the AE used to transform images between image and latent space. We find that generated images can be more accurately reconstructed by the AE than real images, allowing for a simple detection approach based on the reconstruction error. Most importantly, our method is easy to implement and does not require any training, yet nearly matches the performance of detectors that rely on extensive training. We empirically demonstrate that AEROBLADE is effective against state-of-the-art LDMs including Stable Diffusion and Midjourney. Beyond detection, our approach allows for the qualitative analysis of images, which can be leveraged for identifying inpainted regions.

Title: Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation

• Authors: Maoyuan Ye, Jing Zhang, Juhua Liu, Chenyu Liu, Baocai Yin, Cong Liu, Bo Du, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17904
• Pdf URL: https://arxiv.org/pdf/2401.17904
• Copy Paste: [[2401.17904]] Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation(https://arxiv.org/abs/2401.17904)
• Keywords: foundation model
• Abstract: The Segment Anything Model (SAM), a profound vision foundation model pre-trained on a large-scale dataset, breaks the boundaries of general segmentation and sparks various downstream applications. This paper introduces Hi-SAM, a unified model leveraging SAM for hierarchical text segmentation. Hi-SAM excels in text segmentation across four hierarchies, including stroke, word, text-line, and paragraph, while realizing layout analysis as well. Specifically, we first turn SAM into a high-quality text stroke segmentation (TSS) model through a parameter-efficient fine-tuning approach. We use this TSS model to iteratively generate the text stroke labels in a semi-automatical manner, unifying labels across the four text hierarchies in the HierText dataset. Subsequently, with these complete labels, we launch the end-to-end trainable Hi-SAM based on the TSS architecture with a customized hierarchical mask decoder. During inference, Hi-SAM offers both automatic mask generation (AMG) mode and promptable segmentation mode. In terms of the AMG mode, Hi-SAM segments text stroke foreground masks initially, then samples foreground points for hierarchical text mask generation and achieves layout analysis in passing. As for the promptable mode, Hi-SAM provides word, text-line, and paragraph masks with a single point click. Experimental results show the state-of-the-art performance of our TSS model: 84.86% fgIOU on Total-Text and 88.96% fgIOU on TextSeg for text stroke segmentation. Moreover, compared to the previous specialist for joint hierarchical detection and layout analysis on HierText, Hi-SAM achieves significant improvements: 4.73% PQ and 5.39% F1 on the text-line level, 5.49% PQ and 7.39% F1 on the paragraph level layout analysis, requiring 20x fewer training epochs. The code is available at https://github.com/ymy-k/Hi-SAM.

Title: Paramanu: A Family of Novel Efficient Indic Generative Foundation Language Models

• Authors: Mitodru Niyogi, Arnab Bhattacharya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.18034
• Pdf URL: https://arxiv.org/pdf/2401.18034
• Copy Paste: [[2401.18034]] Paramanu: A Family of Novel Efficient Indic Generative Foundation Language Models(https://arxiv.org/abs/2401.18034)
• Keywords: generative
• Abstract: We present Gyan AI Paramanu ("atom"), a family of novel language models for Indian languages. It is a collection of auto-regressive monolingual, bilingual, and multilingual Indic language models pretrained from scratch on a single GPU for 10 Indian languages (Assamese, Bangla, Hindi, Konkani, Maithili, Marathi, Odia, Sanskrit, Tamil, Telugu) across 5 scripts (Bangla, Devanagari, Odia, Tamil, Telugu) of varying sizes ranging from 13.29M to 367.5M.The models are pretrained with a context size of 1024 on a single GPU. The models are very efficient, small, fast, and powerful. We have also developed an efficient most advanced Indic tokenizer that can even tokenize unseen languages. In order to avoid the "curse of multi-linguality" in our multilingual mParamanu model, we pretrained on comparable corpora by typological grouping using the same script. We performed human evaluation of our pretrained models for open end text generation on grammar, coherence, creativity, and factuality metrics for Bangla, Hindi, and Sanskrit. Our Bangla, Hindi, and Sanskrit models outperformed GPT-3.5-Turbo (ChatGPT), Bloom 7B, LLaMa-2 7B, OPT 6.7B, GPT-J 6B, GPTNeo 1.3B, GPT2-XL large language models (LLMs) by a large margin despite being smaller in size by 66 to 20 times compared to standard 7B LLMs. To run inference on our pretrained models, CPU is enough, and GPU is not needed. We also instruction-tuned our pretrained Bangla, Hindi, Marathi, Tamil, and Telugu models on 23k instructions in respective languages. Our pretrained and instruction-tuned models which are first of its kind, most powerful efficient small generative language models ever developed for Indic languages, and the various results lead to the conclusion that high quality generative language models are possible without high amount of compute power and humongous number of parameters. We plan to release our models at https://www.bharatgpts.com.

Title: Optimizing contrastive learning for cortical folding pattern detection

• Authors: Aymeric Gaudin (1), Louise Guillon (1), Clara Fischer (1), Arnaud Cachia (2), Denis Rivière (1), Jean-François Mangin (1), Joël Chavas (1) ((1) Neurospin, Gif-sur-Yvette, France, (2) LaPsyDé, Laboratoire A.Binet-Sorbonne, Paris, France)
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.18035
• Pdf URL: https://arxiv.org/pdf/2401.18035
• Copy Paste: [[2401.18035]] Optimizing contrastive learning for cortical folding pattern detection(https://arxiv.org/abs/2401.18035)
• Keywords: self-supervised
• Abstract: The human cerebral cortex has many bumps and grooves called gyri and sulci. Even though there is a high inter-individual consistency for the main cortical folds, this is not the case when we examine the exact shapes and details of the folding patterns. Because of this complexity, characterizing the cortical folding variability and relating them to subjects' behavioral characteristics or pathologies is still an open scientific problem. Classical approaches include labeling a few specific patterns, either manually or semi-automatically, based on geometric distances, but the recent availability of MRI image datasets of tens of thousands of subjects makes modern deep-learning techniques particularly attractive. Here, we build a self-supervised deep-learning model to detect folding patterns in the cingulate region. We train a contrastive self-supervised model (SimCLR) on both Human Connectome Project (1101 subjects) and UKBioBank (21070 subjects) datasets with topological-based augmentations on the cortical skeletons, which are topological objects that capture the shape of the folds. We explore several backbone architectures (convolutional network, DenseNet, and PointNet) for the SimCLR. For evaluation and testing, we perform a linear classification task on a database manually labeled for the presence of the "double-parallel" folding pattern in the cingulate region, which is related to schizophrenia characteristics. The best model, giving a test AUC of 0.76, is a convolutional network with 6 layers, a 10-dimensional latent space, a linear projection head, and using the branch-clipping augmentation. This is the first time that a self-supervised deep learning model has been applied to cortical skeletons on such a large dataset and quantitatively evaluated. We can now envisage the next step: applying it to other brain regions to detect other biomarkers.

Title: Multipath parsing in the brain

• Authors: Berta Franzluebbers, Donald Dunagan, Miloš Stanojević, Jan Buys, John T. Hale
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.18046
• Pdf URL: https://arxiv.org/pdf/2401.18046
• Copy Paste: [[2401.18046]] Multipath parsing in the brain(https://arxiv.org/abs/2401.18046)
• Keywords: generative
• Abstract: Humans understand sentences word-by-word, in the order that they hear them. This incrementality entails resolving temporary ambiguities about syntactic relationships. We investigate how humans process these syntactic ambiguities by correlating predictions from incremental generative dependency parsers with timecourse data from people undergoing functional neuroimaging while listening to an audiobook. In particular, we compare competing hypotheses regarding the number of developing syntactic analyses in play during word-by-word comprehension: one vs more than one. This comparison involves evaluating syntactic surprisal from a state-of-the-art dependency parser with LLM-adapted encodings against an existing fMRI dataset. In both English and Chinese data, we find evidence for multipath parsing. Brain regions associated with this multipath effect include bilateral superior temporal gyrus.

Title: Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

• Authors: Daniel Geng, Andrew Owens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.18085
• Pdf URL: https://arxiv.org/pdf/2401.18085
• Copy Paste: [[2401.18085]] Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators(https://arxiv.org/abs/2401.18085)
• Keywords: diffusion
• Abstract: Diffusion models are capable of generating impressive images conditioned on text descriptions, and extensions of these models allow users to edit images at a relatively coarse scale. However, the ability to precisely edit the layout, position, pose, and shape of objects in images with diffusion models is still difficult. To this end, we propose motion guidance, a zero-shot technique that allows a user to specify dense, complex motion fields that indicate where each pixel in an image should move. Motion guidance works by steering the diffusion sampling process with the gradients through an off-the-shelf optical flow network. Specifically, we design a guidance loss that encourages the sample to have the desired motion, as estimated by a flow network, while also being visually similar to the source image. By simultaneously sampling from a diffusion model and guiding the sample to have low guidance loss, we can obtain a motion-edited image. We demonstrate that our technique works on complex motions and produces high quality edits of real and generated images.