2024-02-29

Title: Time Series Analysis in Compressor-Based Machines: A Survey

• Authors: Francesca Forbicini, Nicolò Oreste Pinciroli Vago, Piero Fraternali
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17802
• Pdf URL: https://arxiv.org/pdf/2402.17802
• Copy Paste: [[2402.17802]] Time Series Analysis in Compressor-Based Machines: A Survey(https://arxiv.org/abs/2402.17802)
• Keywords: diffusion
• Abstract: In both industrial and residential contexts, compressor-based machines, such as refrigerators, HVAC systems, heat pumps and chillers, are essential to fulfil production and consumers' needs. The diffusion of sensors and IoT connectivity supports the development of monitoring systems able to detect and predict faults, identify behavioural shifts and forecast the operational status of machines and of their components. The focus of this paper is to survey the recent research on such tasks as Fault Detection, Fault Prediction, Forecasting and Change Point Detection applied to multivariate time series characterizing the operations of compressor-based machines. Specifically, Fault Detection detects and diagnoses faults, Fault Prediction predicts such occurrences, forecasting anticipates the future value of characteristic variables of machines and Change Point Detection identifies significant variations in the behaviour of the appliances, such as a change in the working regime. We identify and classify the approaches to the above-mentioned tasks, compare the algorithms employed, highlight the gaps in the current status of the art and discuss the most promising future research directions in the field.

Title: Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

• Authors: Zhenting Qi, Hanlin Zhang, Eric Xing, Sham Kakade, Himabindu Lakkaraju
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.17840
• Pdf URL: https://arxiv.org/pdf/2402.17840
• Copy Paste: [[2402.17840]] Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems(https://arxiv.org/abs/2402.17840)
• Keywords: in-context
• Abstract: Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.

Title: Representation learning in multiplex graphs: Where and how to fuse information?

• Authors: Piotr Bielak, Tomasz Kajdanowicz
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.17906
• Pdf URL: https://arxiv.org/pdf/2402.17906
• Copy Paste: [[2402.17906]] Representation learning in multiplex graphs: Where and how to fuse information?(https://arxiv.org/abs/2402.17906)
• Keywords: self-supervised
• Abstract: In recent years, unsupervised and self-supervised graph representation learning has gained popularity in the research community. However, most proposed methods are focused on homogeneous networks, whereas real-world graphs often contain multiple node and edge types. Multiplex graphs, a special type of heterogeneous graphs, possess richer information, provide better modeling capabilities and integrate more detailed data from potentially different sources. The diverse edge types in multiplex graphs provide more context and insights into the underlying processes of representation learning. In this paper, we tackle the problem of learning representations for nodes in multiplex networks in an unsupervised or self-supervised manner. To that end, we explore diverse information fusion schemes performed at different levels of the graph processing pipeline. The detailed analysis and experimental evaluation of various scenarios inspired us to propose improvements in how to construct GNN architectures that deal with multiplex graphs.

Title: Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models

• Authors: Ashkan Taghipour, Morteza Ghahremani, Mohammed Bennamoun, Aref Miri Rekavandi, Hamid Laga, Farid Boussaid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17910
• Pdf URL: https://arxiv.org/pdf/2402.17910
• Copy Paste: [[2402.17910]] Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models(https://arxiv.org/abs/2402.17910)
• Keywords: diffusion
• Abstract: While latent diffusion models (LDMs) excel at creating imaginative images, they often lack precision in semantic fidelity and spatial control over where objects are generated. To address these deficiencies, we introduce the Box-it-to-Bind-it (B2B) module - a novel, training-free approach for improving spatial control and semantic accuracy in text-to-image (T2I) diffusion models. B2B targets three key challenges in T2I: catastrophic neglect, attribute binding, and layout guidance. The process encompasses two main steps: i) Object generation, which adjusts the latent encoding to guarantee object generation and directs it within specified bounding boxes, and ii) attribute binding, guaranteeing that generated objects adhere to their specified attributes in the prompt. B2B is designed as a compatible plug-and-play module for existing T2I models, markedly enhancing model performance in addressing the key challenges. We evaluate our technique using the established CompBench and TIFA score benchmarks, demonstrating significant performance improvements compared to existing methods. The source code will be made publicly available at https://github.com/nextaistudio/BoxIt2BindIt.

Title: All in a Single Image: Large Multimodal Models are In-Image Learners

• Authors: Lei Wang, Wanyu Xu, Zhiqiang Hu, Yihuai Lan, Shan Dong, Hao Wang, Roy Ka-Wei Lee, Ee-Peng Lim
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.17971
• Pdf URL: https://arxiv.org/pdf/2402.17971
• Copy Paste: [[2402.17971]] All in a Single Image: Large Multimodal Models are In-Image Learners(https://arxiv.org/abs/2402.17971)
• Keywords: in-context
• Abstract: This paper introduces a new in-context learning (ICL) mechanism called In-Image Learning (I$^2$L) that combines demonstration examples, visual cues, and instructions into a single image to enhance the capabilities of GPT-4V. Unlike previous approaches that rely on converting images to text or incorporating visual input into language models, I$^2$L consolidates all information into one image and primarily leverages image processing, understanding, and reasoning abilities. This has several advantages: it avoids inaccurate textual descriptions of complex images, provides flexibility in positioning demonstration examples, reduces the input burden, and avoids exceeding input limits by eliminating the need for multiple images and lengthy text. To further combine the strengths of different ICL methods, we introduce an automatic strategy to select the appropriate ICL method for a data example in a given task. We conducted experiments on MathVista and Hallusionbench to test the effectiveness of I$^2$L in complex multimodal reasoning tasks and mitigating language hallucination and visual illusion. Additionally, we explored the impact of image resolution, the number of demonstration examples, and their positions on the effectiveness of I$^2$L. Our code is publicly available at https://github.com/AGI-Edgerunners/IIL.

Title: Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning

• Authors: Zeyang Liu, Lipeng Wan, Xinrui Yang, Zhuoran Chen, Xingyu Chen, Xuguang Lan
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2402.17978
• Pdf URL: https://arxiv.org/pdf/2402.17978
• Copy Paste: [[2402.17978]] Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning(https://arxiv.org/abs/2402.17978)
• Keywords: diffusion, generative
• Abstract: Effective exploration is crucial to discovering optimal strategies for multi-agent reinforcement learning (MARL) in complex coordination tasks. Existing methods mainly utilize intrinsic rewards to enable committed exploration or use role-based learning for decomposing joint action spaces instead of directly conducting a collective search in the entire action-observation space. However, they often face challenges obtaining specific joint action sequences to reach successful states in long-horizon tasks. To address this limitation, we propose Imagine, Initialize, and Explore (IIE), a novel method that offers a promising solution for efficient multi-agent exploration in complex scenarios. IIE employs a transformer model to imagine how the agents reach a critical state that can influence each other's transition functions. Then, we initialize the environment at this state using a simulator before the exploration phase. We formulate the imagination as a sequence modeling problem, where the states, observations, prompts, actions, and rewards are predicted autoregressively. The prompt consists of timestep-to-go, return-to-go, influence value, and one-shot demonstration, specifying the desired state and trajectory as well as guiding the action generation. By initializing agents at the critical states, IIE significantly increases the likelihood of discovering potentially important under-explored regions. Despite its simplicity, empirical results demonstrate that our method outperforms multi-agent exploration baselines on the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 environments. Particularly, IIE shows improved performance in the sparse-reward SMAC tasks and produces more effective curricula over the initialized states than other generative methods, such as CVAE-GAN and diffusion models.

Title: PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis

• Authors: Jason J. Yu, Tristan Aumentado-Armstrong, Fereshteh Forghani, Konstantinos G. Derpanis, Marcus A. Brubaker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.17986
• Pdf URL: https://arxiv.org/pdf/2402.17986
• Copy Paste: [[2402.17986]] PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis(https://arxiv.org/abs/2402.17986)
• Keywords: generative
• Abstract: This paper considers the problem of generative novel view synthesis (GNVS), generating novel, plausible views of a scene given a limited number of known views. Here, we propose a set-based generative model that can simultaneously generate multiple, self-consistent new views, conditioned on any number of known views. Our approach is not limited to generating a single image at a time and can condition on zero, one, or more views. As a result, when generating a large number of views, our method is not restricted to a low-order autoregressive generation approach and is better able to maintain generated image quality over large sets of images. We evaluate the proposed model on standard NVS datasets and show that it outperforms the state-of-the-art image-based GNVS baselines. Further, we show that the model is capable of generating sets of camera views that have no natural sequential ordering, like loops and binocular trajectories, and significantly outperforms other methods on such tasks.

Title: Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints

• Authors: Lingkai Kong, Yuanqi Du, Wenhao Mu, Kirill Neklyudov, Valentin De Bortol, Haorui Wang, Dongxia Wu, Aaron Ferber, Yi-An Ma, Carla P. Gomes, Chao Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18012
• Pdf URL: https://arxiv.org/pdf/2402.18012
• Copy Paste: [[2402.18012]] Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints(https://arxiv.org/abs/2402.18012)
• Keywords: diffusion
• Abstract: Addressing real-world optimization problems becomes particularly challenging when analytic objective functions or constraints are unavailable. While numerous studies have addressed the issue of unknown objectives, limited research has focused on scenarios where feasibility constraints are not given explicitly. Overlooking these constraints can lead to spurious solutions that are unrealistic in practice. To deal with such unknown constraints, we propose to perform optimization within the data manifold using diffusion models. To constrain the optimization process to the data manifold, we reformulate the original optimization problem as a sampling problem from the product of the Boltzmann distribution defined by the objective function and the data distribution learned by the diffusion model. To enhance sampling efficiency, we propose a two-stage framework that begins with a guided diffusion process for warm-up, followed by a Langevin dynamics stage for further correction. Theoretical analysis shows that the initial stage results in a distribution focused on feasible solutions, thereby providing a better initialization for the later stage. Comprehensive experiments on a synthetic dataset, six real-world black-box optimization datasets, and a multi-objective optimization dataset show that our method achieves better or comparable performance with previous state-of-the-art baselines.

Title: Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

• Authors: Kexun Zhang, Yee Man Choi, Zhenqiao Song, Taiqi He, William Yang Wang, Lei Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18025
• Pdf URL: https://arxiv.org/pdf/2402.18025
• Copy Paste: [[2402.18025]] Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions(https://arxiv.org/abs/2402.18025)
• Keywords: in-context
• Abstract: How can large language models (LLMs) process and translate endangered languages? Many languages lack a large corpus to train a decent LLM; therefore existing LLMs rarely perform well in unseen, endangered languages. On the contrary, we observe that 2000 endangered languages, though without a large corpus, have a grammar book or a dictionary. We propose LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. Our key insight is to demonstrate linguistic knowledge of an unseen language in an LLM's prompt, including a dictionary, a grammar book, and morphologically analyzed input text. We implement LINGOLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LINGOLLM elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions. Our findings demonstrate the tremendous value of linguistic knowledge in the age of LLMs for endangered languages. Our data, code, and model generations can be found at https://github.com/LLiLab/llm4endangeredlang.

Title: Breaking the Black-Box: Confidence-Guided Model Inversion Attack for Distribution Shift

• Authors: Xinhao Liu, Yingzhao Jiang, Zetao Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18027
• Pdf URL: https://arxiv.org/pdf/2402.18027
• Copy Paste: [[2402.18027]] Breaking the Black-Box: Confidence-Guided Model Inversion Attack for Distribution Shift(https://arxiv.org/abs/2402.18027)
• Keywords: generative
• Abstract: Model inversion attacks (MIAs) seek to infer the private training data of a target classifier by generating synthetic images that reflect the characteristics of the target class through querying the model. However, prior studies have relied on full access to the target model, which is not practical in real-world scenarios. Additionally, existing black-box MIAs assume that the image prior and target model follow the same distribution. However, when confronted with diverse data distribution settings, these methods may result in suboptimal performance in conducting attacks. To address these limitations, this paper proposes a \textbf{C}onfidence-\textbf{G}uided \textbf{M}odel \textbf{I}nversion attack method called CG-MI, which utilizes the latent space of a pre-trained publicly available generative adversarial network (GAN) as prior information and gradient-free optimizer, enabling high-resolution MIAs across different data distributions in a black-box setting. Our experiments demonstrate that our method significantly \textbf{outperforms the SOTA black-box MIA by more than 49\% for Celeba and 58\% for Facescrub in different distribution settings}. Furthermore, our method exhibits the ability to generate high-quality images \textbf{comparable to those produced by white-box attacks}. Our method provides a practical and effective solution for black-box model inversion attacks.

Title: OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine

• Authors: Xiaosong Wang, Xiaofan Zhang, Guotai Wang, Junjun He, Zhongyu Li, Wentao Zhu, Yi Guo, Qi Dou, Xiaoxiao Li, Dequan Wang, Liang Hong, Qicheng Lao, Tong Ruan, Yukun Zhou, Yixue Li, Jie Zhao, Kang Li, Xin Sun, Lifeng Zhu, Shaoting Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18028
• Pdf URL: https://arxiv.org/pdf/2402.18028
• Copy Paste: [[2402.18028]] OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine(https://arxiv.org/abs/2402.18028)
• Keywords: foundation model
• Abstract: The emerging trend of advancing generalist artificial intelligence, such as GPTv4 and Gemini, has reshaped the landscape of research (academia and industry) in machine learning and many other research areas. However, domain-specific applications of such foundation models (e.g., in medicine) remain untouched or often at their very early stages. It will require an individual set of transfer learning and model adaptation techniques by further expanding and injecting these models with domain knowledge and data. The development of such technologies could be largely accelerated if the bundle of data, algorithms, and pre-trained foundation models were gathered together and open-sourced in an organized manner. In this work, we present OpenMEDLab, an open-source platform for multi-modality foundation models. It encapsulates not only solutions of pioneering attempts in prompting and fine-tuning large language and vision models for frontline clinical and bioinformatic applications but also building domain-specific foundation models with large-scale multi-modal medical data. Importantly, it opens access to a group of pre-trained foundation models for various medical image modalities, clinical text, protein engineering, etc. Inspiring and competitive results are also demonstrated for each collected approach and model in a variety of benchmarks for downstream tasks. We welcome researchers in the field of medical artificial intelligence to continuously contribute cutting-edge methods and models to OpenMEDLab, which can be accessed via https://github.com/openmedlab.

Title: Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension

• Authors: Fan Yin, Jayanth Srinivasa, Kai-Wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18048
• Pdf URL: https://arxiv.org/pdf/2402.18048
• Copy Paste: [[2402.18048]] Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension(https://arxiv.org/abs/2402.18048)
• Keywords: generative
• Abstract: We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs), which serves as a crucial step in building trust between humans and LLMs. Although several approaches based on entropy or verbalized uncertainty have been proposed to calibrate model predictions, these methods are often intractable, sensitive to hyperparameters, and less reliable when applied in generative tasks with LLMs. In this paper, we suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations. Through experiments on four question answering (QA) datasets, we demonstrate the effectiveness ohttps://info.arxiv.org/help/prep#abstractsf our proposed method. Additionally, we study intrinsic dimensions in LLMs and their relations with model layers, autoregressive language modeling, and the training of LLMs, revealing that intrinsic dimensions can be a powerful approach to understanding LLMs.

Title: SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

• Authors: Bin Cao, Jianhao Yuan, Yexin Liu, Jian Li, Shuyang Sun, Jing Liu, Bo Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18068
• Pdf URL: https://arxiv.org/pdf/2402.18068
• Copy Paste: [[2402.18068]] SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model(https://arxiv.org/abs/2402.18068)
• Keywords: diffusion, generative
• Abstract: In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.

Title: Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

• Authors: Yanzuo Lu, Manlin Zhang, Andy J Ma, Xiaohua Xie, Jian-Huang Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18078
• Pdf URL: https://arxiv.org/pdf/2402.18078
• Copy Paste: [[2402.18078]] Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis(https://arxiv.org/abs/2402.18078)
• Keywords: diffusion
• Abstract: Diffusion model is a promising approach to image generation and has been employed for Pose-Guided Person Image Synthesis (PGPIS) with competitive performance. While existing methods simply align the person appearance to the target pose, they are prone to overfitting due to the lack of a high-level semantic understanding on the source person image. In this paper, we propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. In the absence of image-caption pairs and textual prompts, we develop a novel training paradigm purely based on images to control the generation process of the pre-trained text-to-image diffusion model. A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt. This allows for the decoupling of fine-grained appearance and pose information controls at different stages, and thus circumventing the potential overfitting problem. To generate more realistic texture details, a hybrid-granularity attention module is proposed to encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt. Both quantitative and qualitative experimental results on the DeepFashion benchmark demonstrate the superiority of our method over the state of the arts for PGPIS. Code is available at https://github.com/YanzuoLu/CFLD.

Title: Context-aware Talking Face Video Generation

• Authors: Meidai Xuanyuan, Yuwang Wang, Honglei Guo, Qionghai Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18092
• Pdf URL: https://arxiv.org/pdf/2402.18092
• Copy Paste: [[2402.18092]] Context-aware Talking Face Video Generation(https://arxiv.org/abs/2402.18092)
• Keywords: diffusion
• Abstract: In this paper, we consider a novel and practical case for talking face video generation. Specifically, we focus on the scenarios involving multi-people interactions, where the talking context, such as audience or surroundings, is present. In these situations, the video generation should take the context into consideration in order to generate video content naturally aligned with driving audios and spatially coherent to the context. To achieve this, we provide a two-stage and cross-modal controllable video generation pipeline, taking facial landmarks as an explicit and compact control signal to bridge the driving audio, talking context and generated videos. Inside this pipeline, we devise a 3D video diffusion model, allowing for efficient contort of both spatial conditions (landmarks and context video), as well as audio condition for temporally coherent generation. The experimental results verify the advantage of the proposed method over other baselines in terms of audio-video synchronization, video fidelity and frame consistency.

Title: No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

• Authors: June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18096
• Pdf URL: https://arxiv.org/pdf/2402.18096
• Copy Paste: [[2402.18096]] No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization(https://arxiv.org/abs/2402.18096)
• Keywords: generative
• Abstract: Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the potential ramifications of eviction on the generative process are yet to be thoroughly examined. In this paper, we examine the detrimental impact of cache eviction and observe that unforeseen risks arise as the information contained in the KV pairs is exhaustively discarded, resulting in safety breaches, hallucinations, and context loss. Surprisingly, we find that preserving even a small amount of information contained in the evicted KV pairs via reduced precision quantization substantially recovers the incurred degradation. On the other hand, we observe that the important KV pairs must be kept at a relatively higher precision to safeguard the generation quality. Motivated by these observations, we propose \textit{Mixed-precision KV cache}~(MiKV), a reliable cache compression method that simultaneously preserves the context details by retaining the evicted KV pairs in low-precision and ensure generation quality by keeping the important KV pairs in high-precision. Experiments on diverse benchmarks and LLM backbones show that our proposed method offers a state-of-the-art trade-off between compression ratio and performance, compared to other baselines.

Title: Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization

• Authors: Han Guo, Ramtin Hosseini, Ruiyi Zhang, Sai Ashish Somayajula, Ranak Roy Chowdhury, Rajesh K. Gupta, Pengtao Xie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18128
• Pdf URL: https://arxiv.org/pdf/2402.18128
• Copy Paste: [[2402.18128]] Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization(https://arxiv.org/abs/2402.18128)
• Keywords: self-supervised
• Abstract: Masked Autoencoder (MAE) is a notable method for self-supervised pretraining in visual representation learning. It operates by randomly masking image patches and reconstructing these masked patches using the unmasked ones. A key limitation of MAE lies in its disregard for the varying informativeness of different patches, as it uniformly selects patches to mask. To overcome this, some approaches propose masking based on patch informativeness. However, these methods often do not consider the specific requirements of downstream tasks, potentially leading to suboptimal representations for these tasks. In response, we introduce the Multi-level Optimized Mask Autoencoder (MLO-MAE), a novel framework that leverages end-to-end feedback from downstream tasks to learn an optimal masking strategy during pretraining. Our experimental findings highlight MLO-MAE's significant advancements in visual representation learning. Compared to existing methods, it demonstrates remarkable improvements across diverse datasets and tasks, showcasing its adaptability and efficiency. Our code is available at: https://github.com/Alexiland/MLOMAE

Title: Understanding the Role of Pathways in a Deep Neural Network

• Authors: Lei Lyu, Chen Pang, Jihua Wang
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.18132
• Pdf URL: https://arxiv.org/pdf/2402.18132
• Copy Paste: [[2402.18132]] Understanding the Role of Pathways in a Deep Neural Network(https://arxiv.org/abs/2402.18132)
• Keywords: diffusion
• Abstract: Deep neural networks have demonstrated superior performance in artificial intelligence applications, but the opaqueness of their inner working mechanism is one major drawback in their application. The prevailing unit-based interpretation is a statistical observation of stimulus-response data, which fails to show a detailed internal process of inherent mechanisms of neural networks. In this work, we analyze a convolutional neural network (CNN) trained in the classification task and present an algorithm to extract the diffusion pathways of individual pixels to identify the locations of pixels in an input image associated with object classes. The pathways allow us to test the causal components which are important for classification and the pathway-based representations are clearly distinguishable between categories. We find that the few largest pathways of an individual pixel from an image tend to cross the feature maps in each layer that is important for classification. And the large pathways of images of the same category are more consistent in their trends than those of different categories. We also apply the pathways to understanding adversarial attacks, object completion, and movement perception. Further, the total number of pathways on feature maps in all layers can clearly discriminate the original, deformed, and target samples.

Title: Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation

• Authors: Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, Jie Zhou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.18150
• Pdf URL: https://arxiv.org/pdf/2402.18150
• Copy Paste: [[2402.18150]] Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation(https://arxiv.org/abs/2402.18150)
• Keywords: in-context
• Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating additional information from retrieval. However, studies have shown that LLMs still face challenges in effectively using the retrieved information, even ignoring it or being misled by it. The key reason is that the training of LLMs does not clearly make LLMs learn how to utilize input retrieved texts with varied quality. In this paper, we propose a novel perspective that considers the role of LLMs in RAG as ``Information Refiner'', which means that regardless of correctness, completeness, or usefulness of retrieved texts, LLMs can consistently integrate knowledge within the retrieved texts and model parameters to generate the texts that are more concise, accurate, and complete than the retrieved texts. To this end, we propose an information refinement training method named InFO-RAG that optimizes LLMs for RAG in an unsupervised manner. InFO-RAG is low-cost and general across various tasks. Extensive experiments on zero-shot prediction of 11 datasets in diverse tasks including Question Answering, Slot-Filling, Language Modeling, Dialogue, and Code Generation show that InFO-RAG improves the performance of LLaMA2 by an average of 9.39\% relative points. InFO-RAG also shows advantages in in-context learning and robustness of RAG.

Title: Diffusion-based Neural Network Weights Generation

• Authors: Bedionita Soro, Bruno Andreis, Hayeon Lee, Song Chong, Frank Hutter, Sung Ju Hwang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18153
• Pdf URL: https://arxiv.org/pdf/2402.18153
• Copy Paste: [[2402.18153]] Diffusion-based Neural Network Weights Generation(https://arxiv.org/abs/2402.18153)
• Keywords: diffusion
• Abstract: Transfer learning is a topic of significant interest in recent deep learning research because it enables faster convergence and improved performance on new tasks. While the performance of transfer learning depends on the similarity of the source data to the target data, it is costly to train a model on a large number of datasets. Therefore, pretrained models are generally blindly selected with the hope that they will achieve good performance on the given task. To tackle such suboptimality of the pretrained models, we propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights, to learn the distribution of a set of pretrained weights conditioned on each dataset for transfer learning on unseen datasets. By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.

Title: Autoencoder-based General Purpose Representation Learning for Customer Embedding

• Authors: Jan Henrik Bertrand, Jacopo Pio Gargano, Laurent Mombaerts, Jonathan Taws
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18164
• Pdf URL: https://arxiv.org/pdf/2402.18164
• Copy Paste: [[2402.18164]] Autoencoder-based General Purpose Representation Learning for Customer Embedding(https://arxiv.org/abs/2402.18164)
• Keywords: generative
• Abstract: In recent years, exploiting the domain-specific underlying structure of data and its generative factors for representation learning has shown success in various use-case agnostic applications. However, the diversity and complexity of tabular data have made it challenging to represent these structures in a latent space through multi-dimensional vectors. We design an autoencoder-based framework for building general purpose embeddings, we assess the performance of different autoencoder architectures, and show simpler models outperform complex ones in embedding highly complex tabular data. We apply our framework to produce plug-and-play, rich, and anonymized embeddings representing AWS customers for usage in any model, saving up to 45% of development time, and observe significant improvements in downstream models. Moreover, we propose a significant improvement to the calculation of reconstruction loss for multi-layer contractive autoencoders (CAE) by calculating the Jacobian of the entire encoder leading to a 15% improvement in reconstruction quality when compared to a stacked CAE.

Title: Self-Supervised Spatially Variant PSF Estimation for Aberration-Aware Depth-from-Defocus

• Authors: Zhuofeng Wu, Yusuke Monno, Masatoshi Okutomi
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.18175
• Pdf URL: https://arxiv.org/pdf/2402.18175
• Copy Paste: [[2402.18175]] Self-Supervised Spatially Variant PSF Estimation for Aberration-Aware Depth-from-Defocus(https://arxiv.org/abs/2402.18175)
• Keywords: self-supervised
• Abstract: In this paper, we address the task of aberration-aware depth-from-defocus (DfD), which takes account of spatially variant point spread functions (PSFs) of a real camera. To effectively obtain the spatially variant PSFs of a real camera without requiring any ground-truth PSFs, we propose a novel self-supervised learning method that leverages the pair of real sharp and blurred images, which can be easily captured by changing the aperture setting of the camera. In our PSF estimation, we assume rotationally symmetric PSFs and introduce the polar coordinate system to more accurately learn the PSF estimation network. We also handle the focus breathing phenomenon that occurs in real DfD situations. Experimental results on synthetic and real data demonstrate the effectiveness of our method regarding both the PSF estimation and the depth estimation.

Title: Balancing Act: Distribution-Guided Debiasing in Diffusion Models

• Authors: Rishubh Parihar, Abhijnya Bhat, Saswat Mallick, Abhipsa Basu, Jogendra Nath Kundu, R. Venkatesh Babu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18206
• Pdf URL: https://arxiv.org/pdf/2402.18206
• Copy Paste: [[2402.18206]] Balancing Act: Distribution-Guided Debiasing in Diffusion Models(https://arxiv.org/abs/2402.18206)
• Keywords: diffusion, generative
• Abstract: Diffusion Models (DMs) have emerged as powerful generative models with unprecedented image generation capability. These models are widely used for data augmentation and creative applications. However, DMs reflect the biases present in the training datasets. This is especially concerning in the context of faces, where the DM prefers one demographic subgroup vs others (eg. female vs male). In this work, we present a method for debiasing DMs without relying on additional data or model retraining. Specifically, we propose Distribution Guidance, which enforces the generated images to follow the prescribed attribute distribution. To realize this, we build on the key insight that the latent features of denoising UNet hold rich demographic semantics, and the same can be leveraged to guide debiased generation. We train Attribute Distribution Predictor (ADP) - a small mlp that maps the latent features to the distribution of attributes. ADP is trained with pseudo labels generated from existing attribute classifiers. The proposed Distribution Guidance with ADP enables us to do fair generation. Our method reduces bias across single/multiple attributes and outperforms the baseline by a significant margin for unconditional and text-conditional diffusion models. Further, we present a downstream task of training a fair attribute classifier by rebalancing the training set with our generated data.

Title: Zero-Shot Aerial Object Detection with Visual Description Regularization

• Authors: Zhengqing Zang, Chenyu Lin, Chenwei Tang, Tao Wang, Jiancheng Lv
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18233
• Pdf URL: https://arxiv.org/pdf/2402.18233
• Copy Paste: [[2402.18233]] Zero-Shot Aerial Object Detection with Visual Description Regularization(https://arxiv.org/abs/2402.18233)
• Keywords: generative
• Abstract: Existing object detection models are mainly trained on large-scale labeled datasets. However, annotating data for novel aerial object classes is expensive since it is time-consuming and may require expert knowledge. Thus, it is desirable to study label-efficient object detection methods on aerial images. In this work, we propose a zero-shot method for aerial object detection named visual Description Regularization, or DescReg. Concretely, we identify the weak semantic-visual correlation of the aerial objects and aim to address the challenge with prior descriptions of their visual appearance. Instead of directly encoding the descriptions into class embedding space which suffers from the representation gap problem, we propose to infuse the prior inter-class visual similarity conveyed in the descriptions into the embedding learning. The infusion process is accomplished with a newly designed similarity-aware triplet loss which incorporates structured regularization on the representation space. We conduct extensive experiments with three challenging aerial object detection datasets, including DIOR, xView, and DOTA. The results demonstrate that DescReg significantly outperforms the state-of-the-art ZSD methods with complex projection designs and generative frameworks, e.g., DescReg outperforms best reported ZSD method on DIOR by 4.5 mAP on unseen classes and 8.1 in HM. We further show the generalizability of DescReg by integrating it into generative ZSD methods as well as varying the detection architecture.

Title: Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient

• Authors: Mingxin Li, Richong Zhang, Zhijie Nie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18281
• Pdf URL: https://arxiv.org/pdf/2402.18281
• Copy Paste: [[2402.18281]] Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient(https://arxiv.org/abs/2402.18281)
• Keywords: self-supervised
• Abstract: Sentence Representation Learning (SRL) is a crucial task in Natural Language Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently a mainstream approach. However, the reasons behind its remarkable effectiveness remain unclear. Specifically, in other research fields, contrastive SSL shares similarities in both theory and practical performance with non-contrastive SSL (e.g., alignment & uniformity, Barlow Twins, and VICReg). However, in SRL, contrastive SSL outperforms non-contrastive SSL significantly. Therefore, two questions arise: First, what commonalities enable various contrastive losses to achieve superior performance in SRL? Second, how can we make non-contrastive SSL, which is similar to contrastive SSL but ineffective in SRL, effective? To address these questions, we start from the perspective of gradients and discover that four effective contrastive losses can be integrated into a unified paradigm, which depends on three components: the Gradient Dissipation, the Weight, and the Ratio. Then, we conduct an in-depth analysis of the roles these components play in optimization and experimentally demonstrate their significance for model performance. Finally, by adjusting these components, we enable non-contrastive SSL to achieve outstanding performance in SRL.

Title: Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization

• Authors: Shuo Yang, Gjergji Kasneci
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18284
• Pdf URL: https://arxiv.org/pdf/2402.18284
• Copy Paste: [[2402.18284]] Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization(https://arxiv.org/abs/2402.18284)
• Keywords: self-supervised, generative
• Abstract: Wide usage of ChatGPT has highlighted the potential of reinforcement learning from human feedback. However, its training pipeline relies on manual ranking, a resource-intensive process. To reduce labor costs, we propose a self-supervised text ranking approach for applying Proximal-Policy-Optimization to fine-tune language models while eliminating the need for human annotators. Our method begins with probabilistic sampling to encourage a language model to generate diverse responses for each input. We then employ TextRank and ISODATA algorithms to rank and cluster these responses based on their semantics. Subsequently, we construct a reward model to learn the rank and optimize our generative policy. Our experimental results, conducted using two language models on three tasks, demonstrate that the models trained by our method considerably outperform baselines regarding BLEU, GLEU, and METEOR scores. Furthermore, our manual evaluation shows that our ranking results exhibit a remarkably high consistency with that of humans. This research significantly reduces training costs of proximal policy-guided models and demonstrates the potential for self-correction of language models.

Title: Self-Supervised Learning in Electron Microscopy: Towards a Foundation Model for Advanced Image Analysis

• Authors: Bashir Kazimi, Karina Ruzaeva, Stefan Sandfeld
• Subjects: cs.CV, cond-mat.mtrl-sci, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18286
• Pdf URL: https://arxiv.org/pdf/2402.18286
• Copy Paste: [[2402.18286]] Self-Supervised Learning in Electron Microscopy: Towards a Foundation Model for Advanced Image Analysis(https://arxiv.org/abs/2402.18286)
• Keywords: self-supervised, foundation model
• Abstract: In this work, we explore the potential of self-supervised learning from unlabeled electron microscopy datasets, taking a step toward building a foundation model in this field. We show how self-supervised pretraining facilitates efficient fine-tuning for a spectrum of downstream tasks, including semantic segmentation, denoising, noise & background removal, and super-resolution. Experimentation with varying model complexities and receptive field sizes reveals the remarkable phenomenon that fine-tuned models of lower complexity consistently outperform more complex models with random weight initialization. We demonstrate the versatility of self-supervised pretraining across various downstream tasks in the context of electron microscopy, allowing faster convergence and better performance. We conclude that self-supervised pretraining serves as a powerful catalyst, being especially advantageous when limited annotated data are available and efficient scaling of computational cost are important.

Title: Windowed-FourierMixer: Enhancing Clutter-Free Room Modeling with Fourier Transform

• Authors: Bruno Henriques, Benjamin Allaert, Jean-Philippe Vandeborre
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18287
• Pdf URL: https://arxiv.org/pdf/2402.18287
• Copy Paste: [[2402.18287]] Windowed-FourierMixer: Enhancing Clutter-Free Room Modeling with Fourier Transform(https://arxiv.org/abs/2402.18287)
• Keywords: generative
• Abstract: With the growing demand for immersive digital applications, the need to understand and reconstruct 3D scenes has significantly increased. In this context, inpainting indoor environments from a single image plays a crucial role in modeling the internal structure of interior spaces as it enables the creation of textured and clutter-free reconstructions. While recent methods have shown significant progress in room modeling, they rely on constraining layout estimators to guide the reconstruction process. These methods are highly dependent on the performance of the structure estimator and its generative ability in heavily occluded environments. In response to these issues, we propose an innovative approach based on a U-Former architecture and a new Windowed-FourierMixer block, resulting in a unified, single-phase network capable of effectively handle human-made periodic structures such as indoor spaces. This new architecture proves advantageous for tasks involving indoor scenes where symmetry is prevalent, allowing the model to effectively capture features such as horizon/ceiling height lines and cuboid-shaped rooms. Experiments show the proposed approach outperforms current state-of-the-art methods on the Structured3D dataset demonstrating superior performance in both quantitative metrics and qualitative results. Code and models will be made publicly available.

Title: Grid-Based Continuous Normal Representation for Anomaly Detection

• Authors: Joo Chan Lee, Taejune Kim, Eunbyung Park, Simon S. Woo, Jong Hwan Ko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18293
• Pdf URL: https://arxiv.org/pdf/2402.18293
• Copy Paste: [[2402.18293]] Grid-Based Continuous Normal Representation for Anomaly Detection(https://arxiv.org/abs/2402.18293)
• Keywords: anomaly
• Abstract: There have been significant advancements in anomaly detection in an unsupervised manner, where only normal images are available for training. Several recent methods aim to detect anomalies based on a memory, comparing the input and the directly stored normal features (or trained features with normal images). However, such memory-based approaches operate on a discrete feature space implemented by the nearest neighbor or attention mechanism, suffering from poor generalization or an identity shortcut issue outputting the same as input, respectively. Furthermore, the majority of existing methods are designed to detect single-class anomalies, resulting in unsatisfactory performance when presented with multiple classes of objects. To tackle all of the above challenges, we propose GRAD, a novel anomaly detection method for representing normal features within a "continuous" feature space, enabled by transforming spatial features into coordinates and mapping them to continuous grids. Furthermore, we carefully design the grids tailored for anomaly detection, representing both local and global normal features and fusing them effectively. Our extensive experiments demonstrate that GRAD successfully generalizes the normal features and mitigates the identity shortcut, furthermore, GRAD effectively handles diverse classes in a single model thanks to the high-granularity global representation. In an evaluation using the MVTec AD dataset, GRAD significantly outperforms the previous state-of-the-art method by reducing 65.0\% of the error for multi-class unified anomaly detection. The project page is available at https://tae-mo.github.io/grad/.

Title: How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning

• Authors: Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, Tanmoy Chakraborty
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18312
• Pdf URL: https://arxiv.org/pdf/2402.18312
• Copy Paste: [[2402.18312]] How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning(https://arxiv.org/abs/2402.18312)
• Keywords: in-context
• Abstract: Despite superior reasoning prowess demonstrated by Large Language Models (LLMs) with Chain-of-Thought (CoT) prompting, a lack of understanding prevails around the internal mechanisms of the models that facilitate CoT generation. This work investigates the neural sub-structures within LLMs that manifest CoT reasoning from a mechanistic point of view. From an analysis of LLaMA-2 7B applied to multistep reasoning over fictional ontologies, we demonstrate that LLMs deploy multiple parallel pathways of answer generation for step-by-step reasoning. These parallel pathways provide sequential answers from the input question context as well as the generated CoT. We observe a striking functional rift in the middle layers of the LLM. Token representations in the initial half remain strongly biased towards the pretraining prior, with the in-context taking over abruptly in the later half. This internal phase shift manifests in different functional components: attention heads that write the answer token predominantly appear in the later half, attention heads that move information along ontological relationships appear exclusively in the initial half, and so on. To the best of our knowledge, this is the first attempt towards mechanistic investigation of CoT reasoning in LLMs.

Title: FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes

• Authors: Ziying Pan, Kun Wang, Gang Li, Feihong He, Xiwang Li, Yongxuan Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18331
• Pdf URL: https://arxiv.org/pdf/2402.18331
• Copy Paste: [[2402.18331]] FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes(https://arxiv.org/abs/2402.18331)
• Keywords: diffusion
• Abstract: The class-conditional image generation based on diffusion models is renowned for generating high-quality and diverse images. However, most prior efforts focus on generating images for general categories, e.g., 1000 classes in ImageNet-1k. A more challenging task, large-scale fine-grained image generation, remains the boundary to explore. In this work, we present a parameter-efficient strategy, called FineDiffusion, to fine-tune large pre-trained diffusion models scaling to large-scale fine-grained image generation with 10,000 categories. FineDiffusion significantly accelerates training and reduces storage overhead by only fine-tuning tiered class embedder, bias terms, and normalization layers' parameters. To further improve the image generation quality of fine-grained categories, we propose a novel sampling method for fine-grained image generation, which utilizes superclass-conditioned guidance, specifically tailored for fine-grained categories, to replace the conventional classifier-free guidance sampling. Compared to full fine-tuning, FineDiffusion achieves a remarkable 1.56x training speed-up and requires storing merely 1.77% of the total model parameters, while achieving state-of-the-art FID of 9.776 on image generation of 10,000 classes. Extensive qualitative and quantitative experiments demonstrate the superiority of our method compared to other parameter-efficient fine-tuning methods. The code and more generated results are available at our project website: https://finediffusion.github.io/.

Title: Objective and Interpretable Breast Cosmesis Evaluation with Attention Guided Denoising Diffusion Anomaly Detection Model

• Authors: Sangjoon Park, Yong Bae Kim, Jee Suk Chang, Seo Hee Choi, Hyungjin Chung, Ik Jae Lee, Hwa Kyung Byun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18362
• Pdf URL: https://arxiv.org/pdf/2402.18362
• Copy Paste: [[2402.18362]] Objective and Interpretable Breast Cosmesis Evaluation with Attention Guided Denoising Diffusion Anomaly Detection Model(https://arxiv.org/abs/2402.18362)
• Keywords: diffusion, self-supervised, anomaly
• Abstract: As advancements in the field of breast cancer treatment continue to progress, the assessment of post-surgical cosmetic outcomes has gained increasing significance due to its substantial impact on patients' quality of life. However, evaluating breast cosmesis presents challenges due to the inherently subjective nature of expert labeling. In this study, we present a novel automated approach, Attention-Guided Denoising Diffusion Anomaly Detection (AG-DDAD), designed to assess breast cosmesis following surgery, addressing the limitations of conventional supervised learning and existing anomaly detection models. Our approach leverages the attention mechanism of the distillation with no label (DINO) self-supervised Vision Transformer (ViT) in combination with a diffusion model to achieve high-quality image reconstruction and precise transformation of discriminative regions. By training the diffusion model on unlabeled data predominantly with normal cosmesis, we adopt an unsupervised anomaly detection perspective to automatically score the cosmesis. Real-world data experiments demonstrate the effectiveness of our method, providing visually appealing representations and quantifiable scores for cosmesis evaluation. Compared to commonly used rule-based programs, our fully automated approach eliminates the need for manual annotations and offers objective evaluation. Moreover, our anomaly detection model exhibits state-of-the-art performance, surpassing existing models in accuracy. Going beyond the scope of breast cosmesis, our research represents a significant advancement in unsupervised anomaly detection within the medical domain, thereby paving the way for future investigations.

Title: Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation

• Authors: Yusheng Liao, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18428
• Pdf URL: https://arxiv.org/pdf/2402.18428
• Copy Paste: [[2402.18428]] Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation(https://arxiv.org/abs/2402.18428)
• Keywords: generative
• Abstract: Autoregressive (AR) and Non-autoregressive (NAR) models are two types of generative models for Neural Machine Translation (NMT). AR models predict tokens in a word-by-word manner and can effectively capture the distribution of real translations. NAR models predict tokens by extracting bidirectional contextual information which can improve the inference speed but they suffer from performance degradation. Previous works utilized AR models to enhance NAR models by reducing the training data's complexity or incorporating the global information into AR models by virtue of NAR models. However, those investigated methods only take advantage of the contextual information of a single type of model while neglecting the diversity in the contextual information that can be provided by different types of models. In this paper, we propose a novel generic collaborative learning method, DCMCL, where AR and NAR models are treated as collaborators instead of teachers and students. To hierarchically leverage the bilateral contextual information, token-level mutual learning and sequence-level contrastive learning are adopted between AR and NAR models. Extensive experiments on four widely used benchmarks show that the proposed DCMCL method can simultaneously improve both AR and NAR models with up to 1.38 and 2.98 BLEU scores respectively, and can also outperform the current best-unified model with up to 0.97 BLEU scores for both AR and NAR decoding.

Title: Dynamical Regimes of Diffusion Models

• Authors: Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, Marc Mézard
• Subjects: cs.LG, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2402.18491
• Pdf URL: https://arxiv.org/pdf/2402.18491
• Copy Paste: [[2402.18491]] Dynamical Regimes of Diffusion Models(https://arxiv.org/abs/2402.18491)
• Keywords: diffusion, generative
• Abstract: Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.

Title: Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification

• Authors: Garima Chhikara, Anurag Sharma, Kripabandhu Ghosh, Abhijnan Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18502
• Pdf URL: https://arxiv.org/pdf/2402.18502
• Copy Paste: [[2402.18502]] Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification(https://arxiv.org/abs/2402.18502)
• Keywords: in-context
• Abstract: Employing Large Language Models (LLM) in various downstream applications such as classification is crucial, especially for smaller companies lacking the expertise and resources required for fine-tuning a model. Fairness in LLMs helps ensure inclusivity, equal representation based on factors such as race, gender and promotes responsible AI deployment. As the use of LLMs has become increasingly prevalent, it is essential to assess whether LLMs can generate fair outcomes when subjected to considerations of fairness. In this study, we introduce a framework outlining fairness regulations aligned with various fairness definitions, with each definition being modulated by varying degrees of abstraction. We explore the configuration for in-context learning and the procedure for selecting in-context demonstrations using RAG, while incorporating fairness rules into the process. Experiments conducted with different LLMs indicate that GPT-4 delivers superior results in terms of both accuracy and fairness compared to other models. This work is one of the early attempts to achieve fairness in prediction tasks by utilizing LLMs through in-context learning.

Title: Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling

• Authors: Mahdi Karami, Ali Ghodsi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18508
• Pdf URL: https://arxiv.org/pdf/2402.18508
• Copy Paste: [[2402.18508]] Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling(https://arxiv.org/abs/2402.18508)
• Keywords: in-context
• Abstract: In the rapidly evolving landscape of deep learning, the quest for models that balance expressivity with computational efficiency has never been more critical. This paper introduces Orchid, a novel architecture that reimagines sequence modeling by incorporating a new data-dependent convolution mechanism. Orchid is designed to address the inherent limitations of traditional attention mechanisms, particularly their quadratic complexity, without compromising the ability to capture long-range dependencies and in-context learning. At the core of Orchid lies the data-dependent convolution layer, which dynamically adjusts its kernel conditioned on input data using a dedicated conditioning neural network. We design two simple conditioning networks that maintain shift equivariance in the adaptive convolution operation. The dynamic nature of data-dependent convolution kernel, coupled with gating operations, grants Orchid high expressivity while maintaining efficiency and quasilinear scalability for long sequences. We rigorously evaluate Orchid across multiple domains, including language modeling and image classification, to showcase its performance and generality. Our experiments demonstrate that Orchid architecture not only outperforms traditional attention-based architectures such as BERT and Vision Transformers with smaller model sizes, but also extends the feasible sequence length beyond the limitations of the dense attention layers. This achievement represents a significant step towards more efficient and scalable deep learning models for sequence modeling.

Title: RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval

• Authors: Kaiyue Wen, Xingyu Dang, Kaifeng Lyu
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18510
• Pdf URL: https://arxiv.org/pdf/2402.18510
• Copy Paste: [[2402.18510]] RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval(https://arxiv.org/abs/2402.18510)
• Keywords: in-context
• Abstract: This paper investigates the gap in representation powers of Recurrent Neural Networks (RNNs) and Transformers in the context of solving algorithmic problems. We focus on understanding whether RNNs, known for their memory efficiency in handling long sequences, can match the performance of Transformers, particularly when enhanced with Chain-of-Thought (CoT) prompting. Our theoretical analysis reveals that CoT improves RNNs but is insufficient to close the gap with Transformers. A key bottleneck lies in the inability of RNNs to perfectly retrieve information from the context, even with CoT: for several tasks that explicitly or implicitly require this capability, such as associative recall and determining if a graph is a tree, we prove that RNNs are not expressive enough to solve the tasks while Transformers can solve them with ease. Conversely, we prove that adopting techniques to enhance the in-context retrieval capability of RNNs, including Retrieval-Augmented Generation (RAG) and adding a single Transformer layer, can elevate RNNs to be capable of solving all polynomial-time solvable problems with CoT, hence closing the representation gap with Transformers.

Title: Diffusion Language Models Are Versatile Protein Learners

• Authors: Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, Quanquan Gu
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2402.18567
• Pdf URL: https://arxiv.org/pdf/2402.18567
• Copy Paste: [[2402.18567]] Diffusion Language Models Are Versatile Protein Learners(https://arxiv.org/abs/2402.18567)
• Keywords: diffusion, self-supervised, generative
• Abstract: This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework, which generalizes language modeling for proteins in a principled way. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation. We further demonstrate the proposed diffusion generative pre-training makes DPLM possess a better understanding of proteins, making it a superior representation learner, which can be fine-tuned for various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022). Moreover, DPLM can be tailored for various needs, which showcases its prowess of conditional generation in several ways: (1) conditioning on partial peptide sequences, e.g., generating scaffolds for functional motifs with high success rate; (2) incorporating other modalities as conditioner, e.g., structure-conditioned generation for inverse folding; and (3) steering sequence generation towards desired properties, e.g., satisfying specified secondary structures, through a plug-and-play classifier guidance.