2024-01-31

Title: Informal Safety Guarantees for Simulated Optimizers Through Extrapolation from Partial Simulations

• Authors: Luke Marks
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16426
• Pdf URL: https://arxiv.org/pdf/2401.16426
• Copy Paste: [[2401.16426]] Informal Safety Guarantees for Simulated Optimizers Through Extrapolation from Partial Simulations(https://arxiv.org/abs/2401.16426)
• Keywords: self-supervised
• Abstract: Self-supervised learning is the backbone of state of the art language modeling. It has been argued that training with predictive loss on a self-supervised dataset causes simulators: entities that internally represent possible configurations of real-world systems. Under this assumption, a mathematical model for simulators is built based in the Cartesian frames model of embedded agents, which is extended to multi-agent worlds through scaling a two-dimensional frame to arbitrary dimensions, where literature prior chooses to instead use operations on frames. This variant leveraging scaling dimensionality is named the Cartesian object, and is used to represent simulations (where individual simulacra are the agents and devices in that object). Around the Cartesian object, functions like token selection and simulation complexity are accounted for in formalizing the behavior of a simulator, and used to show (through the L\"obian obstacle) that a proof of alignment between simulacra by inspection of design is impossible in the simulator context. Following this, a scheme is proposed and termed Partial Simulation Extrapolation aimed at circumventing the L\"obian obstacle through the evaluation of low-complexity simulations.

Title: Hybrid Transformer and Spatial-Temporal Self-Supervised Learning for Long-term Traffic Prediction

• Authors: Wang Zhu, Doudou Zhang, Baichao Long, Jianli Xiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16453
• Pdf URL: https://arxiv.org/pdf/2401.16453
• Copy Paste: [[2401.16453]] Hybrid Transformer and Spatial-Temporal Self-Supervised Learning for Long-term Traffic Prediction(https://arxiv.org/abs/2401.16453)
• Keywords: self-supervised
• Abstract: Long-term traffic prediction has always been a challenging task due to its dynamic temporal dependencies and complex spatial dependencies. In this paper, we propose a model that combines hybrid Transformer and spatio-temporal self-supervised learning. The model enhances its robustness by applying adaptive data augmentation techniques at the sequence-level and graph-level of the traffic data. It utilizes Transformer to overcome the limitations of recurrent neural networks in capturing long-term sequences, and employs Chebyshev polynomial graph convolution to capture complex spatial dependencies. Furthermore, considering the impact of spatio-temporal heterogeneity on traffic speed, we design two self-supervised learning tasks to model the temporal and spatial heterogeneity, thereby improving the accuracy and generalization ability of the model. Experimental evaluations are conducted on two real-world datasets, PeMS04 and PeMS08, and the results are visualized and analyzed, demonstrating the superior performance of the proposed model.

Title: Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors

• Authors: Shiyin Dong, Mingrui Zhu, Kun Cheng, Nannan Wang, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16459
• Pdf URL: https://arxiv.org/pdf/2401.16459
• Copy Paste: [[2401.16459]] Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors(https://arxiv.org/abs/2401.16459)
• Keywords: diffusion, generative
• Abstract: The remarkable prowess of diffusion models in image generation has spurred efforts to extend their application beyond generative tasks. However, a persistent challenge exists in lacking a unified approach to apply diffusion models to visual perception tasks with diverse semantic granularity requirements. Our purpose is to establish a unified visual perception framework, capitalizing on the potential synergies between generative and discriminative models. In this paper, we propose Vermouth, a simple yet effective framework comprising a pre-trained Stable Diffusion (SD) model containing rich generative priors, a unified head (U-head) capable of integrating hierarchical representations, and an adapted expert providing discriminative priors. Comprehensive investigations unveil potential characteristics of Vermouth, such as varying granularity of perception concealed in latent variables at distinct time steps and various U-net stages. We emphasize that there is no necessity for incorporating a heavyweight or intricate decoder to transform diffusion models into potent representation learners. Extensive comparative evaluations against tailored discriminative models showcase the efficacy of our approach on zero-shot sketch-based image retrieval (ZS-SBIR), few-shot classification, and open-vocabulary semantic segmentation tasks. The promising results demonstrate the potential of diffusion models as formidable learners, establishing their significance in furnishing informative and robust visual representations.

Title: DressCode: Autoregressively Sewing and Generating Garments from Text Guidance

• Authors: Kai He, Kaixin Yao, Qixuan Zhang, Jingyi Yu, Lingjie Liu, Lan Xu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.16465
• Pdf URL: https://arxiv.org/pdf/2401.16465
• Copy Paste: [[2401.16465]] DressCode: Autoregressively Sewing and Generating Garments from Text Guidance(https://arxiv.org/abs/2401.16465)
• Keywords: diffusion
• Abstract: Apparel's significant role in human appearance underscores the importance of garment digitalization for digital human creation. Recent advances in 3D content creation are pivotal for digital human creation. Nonetheless, garment generation from text guidance is still nascent. We introduce a text-driven 3D garment generation framework, DressCode, which aims to democratize design for novices and offer immense potential in fashion design, virtual try-on, and digital human creation. For our framework, we first introduce SewingGPT, a GPT-based architecture integrating cross-attention with text-conditioned embedding to generate sewing patterns with text guidance. We also tailored a pre-trained Stable Diffusion for high-quality, tile-based PBR texture generation. By leveraging a large language model, our framework generates CG-friendly garments through natural language interaction. Our method also facilitates pattern completion and texture editing, simplifying the process for designers by user-friendly interaction. With comprehensive evaluations and comparisons with other state-of-the-art methods, our method showcases the best quality and alignment with input prompts. User studies further validate our high-quality rendering results, highlighting its practical utility and potential in production settings.

Title: A Discriminative Bayesian Gaussian Process Latent Variable Model for High-Dimensional Data

• Authors: Navid Ziaei, Behzad Nazari, Ali Yousefi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16497
• Pdf URL: https://arxiv.org/pdf/2401.16497
• Copy Paste: [[2401.16497]] A Discriminative Bayesian Gaussian Process Latent Variable Model for High-Dimensional Data(https://arxiv.org/abs/2401.16497)
• Keywords: generative
• Abstract: Extracting meaningful information from high-dimensional data poses a formidable modeling challenge, particularly when the data is obscured by noise or represented through different modalities. In this research, we propose a novel non-parametric modeling approach, leveraging the Gaussian Process (GP), to characterize high-dimensional data by mapping it to a latent low-dimensional manifold. This model, named the Latent Discriminative Generative Decoder (LDGD), utilizes both the data (or its features) and associated labels (such as category or stimulus) in the manifold discovery process. To infer the latent variables, we derive a Bayesian solution, allowing LDGD to effectively capture inherent uncertainties in the data while enhancing the model's predictive accuracy and robustness. We demonstrate the application of LDGD on both synthetic and benchmark datasets. Not only does LDGD infer the manifold accurately, but its prediction accuracy in anticipating labels surpasses state-of-the-art approaches. We have introduced inducing points to reduce the computational complexity of Gaussian Processes (GPs) for large datasets. This enhancement facilitates batch training, allowing for more efficient processing and scalability in handling extensive data collections. Additionally, we illustrate that LDGD achieves higher accuracy in predicting labels and operates effectively with a limited training dataset, underscoring its efficiency and effectiveness in scenarios where data availability is constrained. These attributes set the stage for the development of non-parametric modeling approaches in the analysis of high-dimensional data; especially in fields where data are both high-dimensional and complex.

Title: Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

• Authors: Qingqing Zhu, Xiuying Chen, Qiao Jin, Benjamin Hou, Tejas Sudharshan Mathai, Pritam Mukherjee, Xin Gao, Ronald M Summers, Zhiyong Lu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16578
• Pdf URL: https://arxiv.org/pdf/2401.16578
• Copy Paste: [[2401.16578]] Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports(https://arxiv.org/abs/2401.16578)
• Keywords: in-context
• Abstract: In radiology, Artificial Intelligence (AI) has significantly advanced report generation, but automatic evaluation of these AI-produced reports remains challenging. Current metrics, such as Conventional Natural Language Generation (NLG) and Clinical Efficacy (CE), often fall short in capturing the semantic intricacies of clinical contexts or overemphasize clinical details, undermining report clarity. To overcome these issues, our proposed method synergizes the expertise of professional radiologists with Large Language Models (LLMs), like GPT-3.5 and GPT-4 1. Utilizing In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning, our approach aligns LLM evaluations with radiologist standards, enabling detailed comparisons between human and AI generated reports. This is further enhanced by a Regression model that aggregates sentence evaluation scores. Experimental results show that our ''Detailed GPT-4 (5-shot)'' model achieves a 0.48 score, outperforming the METEOR metric by 0.19, while our ''Regressed GPT-4'' model shows even greater alignment with expert evaluations, exceeding the best existing metric by a 0.35 margin. Moreover, the robustness of our explanations has been validated through a thorough iterative strategy. We plan to publicly release annotations from radiology experts, setting a new standard for accuracy in future assessments. This underscores the potential of our approach in enhancing the quality assessment of AI-driven medical reports.

Title: ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks

• Authors: Bolei Ma, Ercong Nie, Shuzhou Yuan, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16589
• Pdf URL: https://arxiv.org/pdf/2401.16589
• Copy Paste: [[2401.16589]] ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks(https://arxiv.org/abs/2401.16589)
• Keywords: in-context
• Abstract: Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.

Title: Depth Anything in Medical Images: A Comparative Study

• Authors: John J. Han, Ayberk Acar, Callahan Henry, Jie Ying Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16600
• Pdf URL: https://arxiv.org/pdf/2401.16600
• Copy Paste: [[2401.16600]] Depth Anything in Medical Images: A Comparative Study(https://arxiv.org/abs/2401.16600)
• Keywords: self-supervised, foundation model
• Abstract: Monocular depth estimation (MDE) is a critical component of many medical tracking and mapping algorithms, particularly from endoscopic or laparoscopic video. However, because ground truth depth maps cannot be acquired from real patient data, supervised learning is not a viable approach to predict depth maps for medical scenes. Although self-supervised learning for MDE has recently gained attention, the outputs are difficult to evaluate reliably and each MDE's generalizability to other patients and anatomies is limited. This work evaluates the zero-shot performance of the newly released Depth Anything Model on medical endoscopic and laparoscopic scenes. We compare the accuracy and inference speeds of Depth Anything with other MDE models trained on general scenes as well as in-domain models trained on endoscopic data. Our findings show that although the zero-shot capability of Depth Anything is quite impressive, it is not necessarily better than other models in both speed and performance. We hope that this study can spark further research in employing foundation models for MDE in medical scenes.

Title: TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese

• Authors: Nicholas Kluge Corrêa, Sophia Falk, Shiza Fatimah, Aniket Sen, Nythamar de Oliveira
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16640
• Pdf URL: https://arxiv.org/pdf/2401.16640
• Copy Paste: [[2401.16640]] TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese(https://arxiv.org/abs/2401.16640)
• Keywords: foundation model
• Abstract: Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development. See https://github.com/Nkluge-correa/TeenyTinyLlama

Title: Gradient-Based Language Model Red Teaming

• Authors: Nevan Wichers, Carson Denison, Ahmad Beirami
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.16656
• Pdf URL: https://arxiv.org/pdf/2401.16656
• Copy Paste: [[2401.16656]] Gradient-Based Language Model Red Teaming(https://arxiv.org/abs/2401.16656)
• Keywords: generative
• Abstract: Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

Title: OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer

• Authors: Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee-weon Jung, Shinji Watanabe
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.16658
• Pdf URL: https://arxiv.org/pdf/2401.16658
• Copy Paste: [[2401.16658]] OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer(https://arxiv.org/abs/2401.16658)
• Keywords: foundation model
• Abstract: Recent studies have advocated for fully open foundation models to promote transparency and open science. As an initial step, the Open Whisper-style Speech Model (OWSM) reproduced OpenAI's Whisper using publicly available data and open-source toolkits. With the aim of reproducing Whisper, the previous OWSM v1 through v3 models were still based on Transformer, which might lead to inferior performance compared to other state-of-the-art speech encoders. In this work, we aim to improve the performance and efficiency of OWSM without extra training data. We present E-Branchformer based OWSM v3.1 models at two scales, i.e., 100M and 1B. The 1B model is the largest E-Branchformer based speech model that has been made publicly available. It outperforms the previous OWSM v3 in a vast majority of evaluation benchmarks, while demonstrating up to 25% faster inference speed. We publicly release the data preparation scripts, pre-trained models and training logs.

Title: Towards Generating Informative Textual Description for Neurons in Language Models

• Authors: Shrayani Mondal, Rishabh Garodia, Arbaaz Qureshi, Taesung Lee, Youngja Park
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16731
• Pdf URL: https://arxiv.org/pdf/2401.16731
• Copy Paste: [[2401.16731]] Towards Generating Informative Textual Description for Neurons in Language Models(https://arxiv.org/abs/2401.16731)
• Keywords: generative
• Abstract: Recent developments in transformer-based language models have allowed them to capture a wide variety of world knowledge that can be adapted to downstream tasks with limited resources. However, what pieces of information are understood in these models is unclear, and neuron-level contributions in identifying them are largely unknown. Conventional approaches in neuron explainability either depend on a finite set of pre-defined descriptors or require manual annotations for training a secondary model that can then explain the neurons of the primary model. In this paper, we take BERT as an example and we try to remove these constraints and propose a novel and scalable framework that ties textual descriptions to neurons. We leverage the potential of generative language models to discover human-interpretable descriptors present in a dataset and use an unsupervised approach to explain neurons with these descriptors. Through various qualitative and quantitative analyses, we demonstrate the effectiveness of this framework in generating useful data-specific descriptors with little human involvement in identifying the neurons that encode these descriptors. In particular, our experiment shows that the proposed approach achieves 75% precision@2, and 50% recall@2

Title: MESA: Matching Everything by Segmenting Anything

• Authors: Yesheng Zhang, Xu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16741
• Pdf URL: https://arxiv.org/pdf/2401.16741
• Copy Paste: [[2401.16741]] MESA: Matching Everything by Segmenting Anything(https://arxiv.org/abs/2401.16741)
• Keywords: foundation model
• Abstract: Feature matching is a crucial task in the field of computer vision, which involves finding correspondences between images. Previous studies achieve remarkable performance using learning-based feature comparison. However, the pervasive presence of matching redundancy between images gives rise to unnecessary and error-prone computations in these methods, imposing limitations on their accuracy. To address this issue, we propose MESA, a novel approach to establish precise area (or region) matches for efficient matching redundancy reduction. MESA first leverages the advanced image understanding capability of SAM, a state-of-the-art foundation model for image segmentation, to obtain image areas with implicit semantic. Then, a multi-relational graph is proposed to model the spatial structure of these areas and construct their scale hierarchy. Based on graphical models derived from the graph, the area matching is reformulated as an energy minimization task and effectively resolved. Extensive experiments demonstrate that MESA yields substantial precision improvement for multiple point matchers in indoor and outdoor downstream tasks, e.g. +13.61% for DKM in indoor pose estimation.

Title: MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images

• Authors: Xurui Li, Ziming Huang, Feng Xue, Yu Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16753
• Pdf URL: https://arxiv.org/pdf/2401.16753
• Copy Paste: [[2401.16753]] MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images(https://arxiv.org/abs/2401.16753)
• Keywords: anomaly
• Abstract: This paper studies zero-shot anomaly classification (AC) and segmentation (AS) in industrial vision. We reveal that the abundant normal and abnormal cues implicit in unlabeled test images can be exploited for anomaly determination, which is ignored by prior methods. Our key observation is that for the industrial product images, the normal image patches could find a relatively large number of similar patches in other unlabeled images, while the abnormal ones only have a few similar patches. We leverage such a discriminative characteristic to design a novel zero-shot AC/AS method by Mutual Scoring (MuSc) of the unlabeled images, which does not need any training or prompts. Specifically, we perform Local Neighborhood Aggregation with Multiple Degrees (LNAMD) to obtain the patch features that are capable of representing anomalies in varying sizes. Then we propose the Mutual Scoring Mechanism (MSM) to leverage the unlabeled test images to assign the anomaly score to each other. Furthermore, we present an optimization approach named Re-scoring with Constrained Image-level Neighborhood (RsCIN) for image-level anomaly classification to suppress the false positives caused by noises in normal images. The superior performance on the challenging MVTec AD and VisA datasets demonstrates the effectiveness of our approach. Compared with the state-of-the-art zero-shot approaches, MuSc achieves a $\textbf{21.1%}$ PRO absolute gain (from 72.7% to 93.8%) on MVTec AD, a $\textbf{19.4%}$ pixel-AP gain and a $\textbf{14.7%}$ pixel-AUROC gain on VisA. In addition, our zero-shot approach outperforms most of the few-shot approaches and is comparable to some one-class methods. Code is available at https://github.com/xrli-U/MuSc.

Title: Diffusion model for relational inference

• Authors: Shuhan Zheng, Ziqiang Li, Kantaro Fujiwara, Gouhei Tanaka
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16755
• Pdf URL: https://arxiv.org/pdf/2401.16755
• Copy Paste: [[2401.16755]] Diffusion model for relational inference(https://arxiv.org/abs/2401.16755)
• Keywords: diffusion, self-supervised
• Abstract: Dynamical behaviors of complex interacting systems, including brain activities, financial price movements, and physical collective phenomena, are associated with underlying interactions between the system's components. The issue of uncovering interaction relations in such systems using observable dynamics is called relational inference. In this study, we propose a Diffusion model for Relational Inference (DiffRI), inspired by a self-supervised method for probabilistic time series imputation. DiffRI learns to infer the probability of the presence of connections between components through conditional diffusion modeling. Experiments on both simulated and quasi-real datasets show that DiffRI is highly competent compared with other state-of-the-art models in discovering ground truth interactions in an unsupervised manner. Our code will be made public soon.

Title: Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization

• Authors: Henglei Lv, Jiayu Xiao, Liang Li, Qingming Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16762
• Pdf URL: https://arxiv.org/pdf/2401.16762
• Copy Paste: [[2401.16762]] Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization(https://arxiv.org/abs/2401.16762)
• Keywords: diffusion, generative
• Abstract: Diffusion-based text-to-image personalization have achieved great success in generating subjects specified by users among various contexts. Even though, existing finetuning-based methods still suffer from model overfitting, which greatly harms the generative diversity, especially when given subject images are few. To this end, we propose Pick-and-Draw, a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. Our approach consists of two components: appearance picking guidance and layout drawing guidance. As for the former, we construct an appearance palette with visual features from the reference image, where we pick local patterns for generating the specified subject with consistent identity. As for layout drawing, we outline the subject's contour by referring to a generative template from the vanilla diffusion model, and inherit the strong image prior to synthesize diverse contexts according to different text conditions. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image. Qualitative and quantitative experiments show that Pick-and-Draw consistently improves identity consistency and generative diversity, pushing the trade-off between subject fidelity and image-text fidelity to a new Pareto frontier.

Title: BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion

• Authors: Yonghao Yu, Shunan Zhu, Huai Qin, Haorui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16764
• Pdf URL: https://arxiv.org/pdf/2401.16764
• Copy Paste: [[2401.16764]] BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion(https://arxiv.org/abs/2401.16764)
• Keywords: diffusion
• Abstract: Witnessing the evolution of text-to-image diffusion models, significant strides have been made in text-to-3D generation. Currently, two primary paradigms dominate the field of text-to-3D: the feed-forward generation solutions, capable of swiftly producing 3D assets but often yielding coarse results, and the Score Distillation Sampling (SDS) based solutions, known for generating high-fidelity 3D assets albeit at a slower pace. The synergistic integration of these methods holds substantial promise for advancing 3D generation techniques. In this paper, we present BoostDream, a highly efficient plug-and-play 3D refining method designed to transform coarse 3D assets into high-quality. The BoostDream framework comprises three distinct processes: (1) We introduce 3D model distillation that fits differentiable representations from the 3D assets obtained through feed-forward generation. (2) A novel multi-view SDS loss is designed, which utilizes a multi-view aware 2D diffusion model to refine the 3D assets. (3) We propose to use prompt and multi-view consistent normal maps as guidance in refinement.Our extensive experiment is conducted on different differentiable 3D representations, revealing that BoostDream excels in generating high-quality 3D assets rapidly, overcoming the Janus problem compared to conventional SDS-based methods. This breakthrough signifies a substantial advancement in both the efficiency and quality of 3D generation processes.

Title: Detection and Recovery Against Deep Neural Network Fault Injection Attacks Based on Contrastive Learning

• Authors: Chenan Wang, Pu Zhao, Siyue Wang, Xue Lin
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16766
• Pdf URL: https://arxiv.org/pdf/2401.16766
• Copy Paste: [[2401.16766]] Detection and Recovery Against Deep Neural Network Fault Injection Attacks Based on Contrastive Learning(https://arxiv.org/abs/2401.16766)
• Keywords: self-supervised
• Abstract: Deep Neural Network (DNN) models when implemented on executing devices as the inference engines are susceptible to Fault Injection Attacks (FIAs) that manipulate model parameters to disrupt inference execution with disastrous performance. This work introduces Contrastive Learning (CL) of visual representations i.e., a self-supervised learning approach into the deep learning training and inference pipeline to implement DNN inference engines with self-resilience under FIAs. Our proposed CL based FIA Detection and Recovery (CFDR) framework features (i) real-time detection with only a single batch of testing data and (ii) fast recovery effective even with only a small amount of unlabeled testing data. Evaluated with the CIFAR-10 dataset on multiple types of FIAs, our CFDR shows promising detection and recovery effectiveness.

Title: Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator

• Authors: Ryoma Furuyama, Daiki Kuyoshi, Satoshi Yamane
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16772
• Pdf URL: https://arxiv.org/pdf/2401.16772
• Copy Paste: [[2401.16772]] Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator(https://arxiv.org/abs/2401.16772)
• Keywords: generative
• Abstract: Imitation learning is often used in addition to reinforcement learning in environments where reward design is difficult or where the reward is sparse, but it is difficult to be able to imitate well in unknown states from a small amount of expert data and sampling data. Supervised learning methods such as Behavioral Cloning do not require sampling data, but usually suffer from distribution shift. The methods based on reinforcement learning, such as inverse reinforcement learning and Generative Adversarial imitation learning (GAIL), can learn from only a few expert data. However, they often need to interact with the environment. Soft Q imitation learning (SQIL) addressed the problems, and it was shown that it could learn efficiently by combining Behavioral Cloning and soft Q-learning with constant rewards. In order to make this algorithm more robust to distribution shift, we propose more efficient and robust algorithm by adding to this method a reward function based on adversarial inverse reinforcement learning that rewards the agent for performing actions in status similar to the demo. We call this algorithm Discriminator Soft Q Imitation Learning (DSQIL). We evaluated it on MuJoCo environments.

Title: An Embeddable Implicit IUVD Representation for Part-based 3D Human Surface Reconstruction

• Authors: Baoxing Li, Yong Deng, Yehui Yang, Xu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16810
• Pdf URL: https://arxiv.org/pdf/2401.16810
• Copy Paste: [[2401.16810]] An Embeddable Implicit IUVD Representation for Part-based 3D Human Surface Reconstruction(https://arxiv.org/abs/2401.16810)
• Keywords: generative
• Abstract: To reconstruct a 3D human surface from a single image, it is important to consider human pose, shape and clothing details simultaneously. In recent years, a combination of parametric body models (such as SMPL) that capture body pose and shape prior, and neural implicit functions that learn flexible clothing details, has been used to integrate the advantages of both approaches. However, the combined representation introduces additional computation, e.g. signed distance calculation, in 3D body feature extraction, which exacerbates the redundancy of the implicit query-and-infer process and fails to preserve the underlying body shape prior. To address these issues, we propose a novel IUVD-Feedback representation, which consists of an IUVD occupancy function and a feedback query algorithm. With this representation, the time-consuming signed distance calculation is replaced by a simple linear transformation in the IUVD space, leveraging the SMPL UV maps. Additionally, the redundant query points in the query-and-infer process are reduced through a feedback mechanism. This leads to more reasonable 3D body features and more effective query points, successfully preserving the parametric body prior. Moreover, the IUVD-Feedback representation can be embedded into any existing implicit human reconstruction pipelines without modifying the trained neural networks. Experiments on THuman2.0 dataset demonstrate that the proposed IUVD-Feedback representation improves result robustness and achieves three times faster acceleration in the query-and-infer process. Furthermore, this representation has the potential to be used in generative applications by leveraging its inherited semantic information from the parametric body model.

Title: Evaluating ML-Based Anomaly Detection Across Datasets of Varied Integrity: A Case Study

• Authors: Adrian Pekar, Richard Jozsa
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2401.16843
• Pdf URL: https://arxiv.org/pdf/2401.16843
• Copy Paste: [[2401.16843]] Evaluating ML-Based Anomaly Detection Across Datasets of Varied Integrity: A Case Study(https://arxiv.org/abs/2401.16843)
• Keywords: anomaly
• Abstract: Cybersecurity remains a critical challenge in the digital age, with network traffic flow anomaly detection being a key pivotal instrument in the fight against cyber threats. In this study, we address the prevalent issue of data integrity in network traffic datasets, which are instrumental in developing machine learning (ML) models for anomaly detection. We introduce two refined versions of the CICIDS-2017 dataset, NFS-2023-nTE and NFS-2023-TE, processed using NFStream to ensure methodologically sound flow expiration and labeling. Our research contrasts the performance of the Random Forest (RF) algorithm across the original CICIDS-2017, its refined counterparts WTMC-2021 and CRiSIS-2022, and our NFStream-generated datasets, in both binary and multi-class classification contexts. We observe that the RF model exhibits exceptional robustness, achieving consistent high-performance metrics irrespective of the underlying dataset quality, which prompts a critical discussion on the actual impact of data integrity on ML efficacy. Our study underscores the importance of continual refinement and methodological rigor in dataset generation for network security research. As the landscape of network threats evolves, so must the tools and techniques used to detect and analyze them.

Title: Repositioning the Subject within Image

• Authors: Yikai Wang, Chenjie Cao, Qiaole Dong, Yifan Li, Yanwei Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16861
• Pdf URL: https://arxiv.org/pdf/2401.16861
• Copy Paste: [[2401.16861]] Repositioning the Subject within Image(https://arxiv.org/abs/2401.16861)
• Keywords: diffusion, generative
• Abstract: Current image manipulation primarily centers on static manipulation, such as replacing specific regions within an image or altering its overall style. In this paper, we introduce an innovative dynamic manipulation task, subject repositioning. This task involves relocating a user-specified subject to a desired position while preserving the image's fidelity. Our research reveals that the fundamental sub-tasks of subject repositioning, which include filling the void left by the repositioned subject, reconstructing obscured portions of the subject and blending the subject to be consistent with surrounding areas, can be effectively reformulated as a unified, prompt-guided inpainting task. Consequently, we can employ a single diffusion generative model to address these sub-tasks using various task prompts learned through our proposed task inversion technique. Additionally, we integrate pre-processing and post-processing techniques to further enhance the quality of subject repositioning. These elements together form our SEgment-gEnerate-and-bLEnd (SEELE) framework. To assess SEELE's effectiveness in subject repositioning, we assemble a real-world subject repositioning dataset called ReS. Our results on ReS demonstrate the quality of repositioned image generation.

Title: Zero-shot Classification using Hyperdimensional Computing

• Authors: Samuele Ruffino, Geethan Karunaratne, Michael Hersche, Luca Benini, Abu Sebastian, Abbas Rahimi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.16876
• Pdf URL: https://arxiv.org/pdf/2401.16876
• Copy Paste: [[2401.16876]] Zero-shot Classification using Hyperdimensional Computing(https://arxiv.org/abs/2401.16876)
• Keywords: generative
• Abstract: Classification based on Zero-shot Learning (ZSL) is the ability of a model to classify inputs into novel classes on which the model has not previously seen any training examples. Providing an auxiliary descriptor in the form of a set of attributes describing the new classes involved in the ZSL-based classification is one of the favored approaches to solving this challenging task. In this work, inspired by Hyperdimensional Computing (HDC), we propose the use of stationary binary codebooks of symbol-like distributed representations inside an attribute encoder to compactly represent a computationally simple end-to-end trainable model, which we name Hyperdimensional Computing Zero-shot Classifier~(HDC-ZSC). It consists of a trainable image encoder, an attribute encoder based on HDC, and a similarity kernel. We show that HDC-ZSC can be used to first perform zero-shot attribute extraction tasks and, can later be repurposed for Zero-shot Classification tasks with minimal architectural changes and minimal model retraining. HDC-ZSC achieves Pareto optimal results with a 63.8% top-1 classification accuracy on the CUB-200 dataset by having only 26.6 million trainable parameters. Compared to two other state-of-the-art non-generative approaches, HDC-ZSC achieves 4.3% and 9.9% better accuracy, while they require more than 1.85x and 1.72x parameters compared to HDC-ZSC, respectively.

Title: WGAN-AFL: Seed Generation Augmented Fuzzer with Wasserstein-GAN

• Authors: Liqun Yang, Chunan Li, Yongxin Qiu, Chaoren Wei, Jian Yang, Hongcheng Guo, Jinxin Ma, Zhoujun Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.16947
• Pdf URL: https://arxiv.org/pdf/2401.16947
• Copy Paste: [[2401.16947]] WGAN-AFL: Seed Generation Augmented Fuzzer with Wasserstein-GAN(https://arxiv.org/abs/2401.16947)
• Keywords: generative
• Abstract: The importance of addressing security vulnerabilities is indisputable, with software becoming crucial in sectors such as national defense and finance. Consequently, The security issues caused by software vulnerabilities cannot be ignored. Fuzz testing is an automated software testing technology that can detect vulnerabilities in the software. However, most previous fuzzers encounter challenges that fuzzing performance is sensitive to initial input seeds. In the absence of high-quality initial input seeds, fuzzers may expend significant resources on program path exploration, leading to a substantial decrease in the efficiency of vulnerability detection. To address this issue, we propose WGAN-AFL. By collecting high-quality testcases, we train a generative adversarial network (GAN) to learn their features, thereby obtaining high-quality initial input seeds. To overcome drawbacks like mode collapse and training instability inherent in GANs, we utilize the Wasserstein GAN (WGAN) architecture for training, further enhancing the quality of the generated seeds. Experimental results demonstrate that WGAN-AFL significantly outperforms the original AFL in terms of code coverage, new paths, and vulnerability discovery, demonstrating the effective enhancement of seed quality by WGAN-AFL.

Title: Making Parametric Anomaly Detection on Tabular Data Non-Parametric Again

• Authors: Hugo Thimonier, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17052
• Pdf URL: https://arxiv.org/pdf/2401.17052
• Copy Paste: [[2401.17052]] Making Parametric Anomaly Detection on Tabular Data Non-Parametric Again(https://arxiv.org/abs/2401.17052)
• Keywords: anomaly
• Abstract: Deep learning for tabular data has garnered increasing attention in recent years, yet employing deep models for structured data remains challenging. While these models excel with unstructured data, their efficacy with structured data has been limited. Recent research has introduced retrieval-augmented models to address this gap, demonstrating promising results in supervised tasks such as classification and regression. In this work, we investigate using retrieval-augmented models for anomaly detection on tabular data. We propose a reconstruction-based approach in which a transformer model learns to reconstruct masked features of \textit{normal} samples. We test the effectiveness of KNN-based and attention-based modules to select relevant samples to help in the reconstruction process of the target sample. Our experiments on a benchmark of 31 tabular datasets reveal that augmenting this reconstruction-based anomaly detection (AD) method with non-parametric relationships via retrieval modules may significantly boost performance.

Title: BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

• Authors: Zhennan Wu, Yang Li, Han Yan, Taizhang Shang, Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, Pan Ji
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.17053
• Pdf URL: https://arxiv.org/pdf/2401.17053
• Copy Paste: [[2401.17053]] BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation(https://arxiv.org/abs/2401.17053)
• Keywords: diffusion
• Abstract: We present BlockFusion, a diffusion-based model that generates 3D scenes as unit blocks and seamlessly incorporates new blocks to extend the scene. BlockFusion is trained using datasets of 3D blocks that are randomly cropped from complete 3D scene meshes. Through per-block fitting, all training blocks are converted into the hybrid neural fields: with a tri-plane containing the geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the signed distance values. A variational auto-encoder is employed to compress the tri-planes into the latent tri-plane space, on which the denoising diffusion process is performed. Diffusion applied to the latent representations allows for high-quality and diverse 3D scene generation. To expand a scene during generation, one needs only to append empty blocks to overlap with the current scene and extrapolate existing latent tri-planes to populate new blocks. The extrapolation is done by conditioning the generation process with the feature samples from the overlapping tri-planes during the denoising iterations. Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that harmoniously blend with the existing scene. A 2D layout conditioning mechanism is used to control the placement and arrangement of scene elements. Experimental results indicate that BlockFusion is capable of generating diverse, geometrically consistent and unbounded large 3D scenes with unprecedented high-quality shapes in both indoor and outdoor scenarios.

Title: Active Generation Network of Human Skeleton for Action Recognition

• Authors: Long Liu, Xin Wang, Fangming Li, Jiayu Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17086
• Pdf URL: https://arxiv.org/pdf/2401.17086
• Copy Paste: [[2401.17086]] Active Generation Network of Human Skeleton for Action Recognition(https://arxiv.org/abs/2401.17086)
• Keywords: generative
• Abstract: Data generation is a data augmentation technique for enhancing the generalization ability for skeleton-based human action recognition. Most existing data generation methods face challenges to ensure the temporal consistency of the dynamic information for action. In addition, the data generated by these methods lack diversity when only a few training samples are available. To solve those problems, We propose a novel active generative network (AGN), which can adaptively learn various action categories by motion style transfer to generate new actions when the data for a particular action is only a single sample or few samples. The AGN consists of an action generation network and an uncertainty metric network. The former, with ST-GCN as the Backbone, can implicitly learn the morphological features of the target action while preserving the category features of the source action. The latter guides generating actions. Specifically, an action recognition model generates prediction vectors for each action, which is then scored using an uncertainty metric. Finally, UMN provides the uncertainty sampling basis for the generated actions.

Title: Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled

• Authors: Shengchao Liu, Chengpeng Wang, Jiarui Lu, Weili Nie, Hanchen Wang, Zhuoxinran Li, Bolei Zhou, Jian Tang
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2401.17123
• Pdf URL: https://arxiv.org/pdf/2401.17123
• Copy Paste: [[2401.17123]] Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled(https://arxiv.org/abs/2401.17123)
• Keywords: generative
• Abstract: Deep generative models (DGMs) have been widely developed for graph data. However, much less investigation has been carried out on understanding the latent space of such pretrained graph DGMs. These understandings possess the potential to provide constructive guidelines for crucial tasks, such as graph controllable generation. Thus in this work, we are interested in studying this problem and propose GraphCG, a method for the unsupervised discovery of steerable factors in the latent space of pretrained graph DGMs. We first examine the representation space of three pretrained graph DGMs with six disentanglement metrics, and we observe that the pretrained representation space is entangled. Motivated by this observation, GraphCG learns the steerable factors via maximizing the mutual information between semantic-rich directions, where the controlled graph moving along the same direction will share the same steerable factors. We quantitatively verify that GraphCG outperforms four competitive baselines on two graph DGMs pretrained on two molecule datasets. Additionally, we qualitatively illustrate seven steerable factors learned by GraphCG on five pretrained DGMs over five graph datasets, including two for molecules and three for point clouds.

Title: Transfer Learning for Text Diffusion Models

• Authors: Kehang Han, Kathleen Kenealy, Aditya Barua, Noah Fiedel, Noah Constant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.17181
• Pdf URL: https://arxiv.org/pdf/2401.17181
• Copy Paste: [[2401.17181]] Transfer Learning for Text Diffusion Models(https://arxiv.org/abs/2401.17181)
• Keywords: diffusion
• Abstract: In this report, we explore the potential for text diffusion to replace autoregressive (AR) decoding for the training and deployment of large language models (LLMs). We are particularly interested to see whether pretrained AR models can be transformed into text diffusion models through a lightweight adaptation procedure we call ``AR2Diff''. We begin by establishing a strong baseline setup for training text diffusion models. Comparing across multiple architectures and pretraining objectives, we find that training a decoder-only model with a prefix LM objective is best or near-best across several tasks. Building on this finding, we test various transfer learning setups for text diffusion models. On machine translation, we find that text diffusion underperforms the standard AR approach. However, on code synthesis and extractive QA, we find diffusion models trained from scratch outperform AR models in many cases. We also observe quality gains from AR2Diff -- adapting AR models to use diffusion decoding. These results are promising given that text diffusion is relatively underexplored and can be significantly faster than AR decoding for long text generation.

Title: Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI

• Authors: Alexander Oberstrass, Sascha E. A. Muenzing, Meiqi Niu, Nicola Palomero-Gallagher, Christian Schiffer, Markus Axer, Katrin Amunts, Timo Dickscheid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17207
• Pdf URL: https://arxiv.org/pdf/2401.17207
• Copy Paste: [[2401.17207]] Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI(https://arxiv.org/abs/2401.17207)
• Keywords: self-supervised
• Abstract: A comprehensive understanding of the organizational principles in the human brain requires, among other factors, well-quantifiable descriptors of nerve fiber architecture. Three-dimensional polarized light imaging (3D-PLI) is a microscopic imaging technique that enables insights into the fine-grained organization of myelinated nerve fibers with high resolution. Descriptors characterizing the fiber architecture observed in 3D-PLI would enable downstream analysis tasks such as multimodal correlation studies, clustering, and mapping. However, best practices for observer-independent characterization of fiber architecture in 3D-PLI are not yet available. To this end, we propose the application of a fully data-driven approach to characterize nerve fiber architecture in 3D-PLI images using self-supervised representation learning. We introduce a 3D-Context Contrastive Learning (CL-3D) objective that utilizes the spatial neighborhood of texture examples across histological brain sections of a 3D reconstructed volume to sample positive pairs for contrastive learning. We combine this sampling strategy with specifically designed image augmentations to gain robustness to typical variations in 3D-PLI parameter maps. The approach is demonstrated for the 3D reconstructed occipital lobe of a vervet monkey brain. We show that extracted features are highly sensitive to different configurations of nerve fibers, yet robust to variations between consecutive brain sections arising from histological processing. We demonstrate their practical applicability for retrieving clusters of homogeneous fiber architecture and performing data mining for interactively selected templates of specific components of fiber architecture such as U-fibers.

Title: ContactGen: Contact-Guided Interactive 3D Human Generation for Partners

• Authors: Dongjun Gu, Jaehyeok Shim, Jaehoon Jang, Changwoo Kang, Kyungdon Joo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17212
• Pdf URL: https://arxiv.org/pdf/2401.17212
• Copy Paste: [[2401.17212]] ContactGen: Contact-Guided Interactive 3D Human Generation for Partners(https://arxiv.org/abs/2401.17212)
• Keywords: diffusion
• Abstract: Among various interactions between humans, such as eye contact and gestures, physical interactions by contact can act as an essential moment in understanding human behaviors. Inspired by this fact, given a 3D partner human with the desired interaction label, we introduce a new task of 3D human generation in terms of physical contact. Unlike previous works of interacting with static objects or scenes, a given partner human can have diverse poses and different contact regions according to the type of interaction. To handle this challenge, we propose a novel method of generating interactive 3D humans for a given partner human based on a guided diffusion framework. Specifically, we newly present a contact prediction module that adaptively estimates potential contact regions between two input humans according to the interaction label. Using the estimated potential contact regions as complementary guidances, we dynamically enforce ContactGen to generate interactive 3D humans for a given partner human within a guided diffusion model. We demonstrate ContactGen on the CHI3D dataset, where our method generates physically plausible and diverse poses compared to comparison methods.

Title: You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation

• Authors: Mehdi Noroozi, Isma Hadji, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.17258
• Pdf URL: https://arxiv.org/pdf/2401.17258
• Copy Paste: [[2401.17258]] You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation(https://arxiv.org/abs/2401.17258)
• Keywords: diffusion
• Abstract: In this paper, we introduce YONOS-SR, a novel stable diffusion-based approach for image super-resolution that yields state-of-the-art results using only a single DDIM step. We propose a novel scale distillation approach to train our SR model. Instead of directly training our SR model on the scale factor of interest, we start by training a teacher model on a smaller magnification scale, thereby making the SR problem simpler for the teacher. We then train a student model for a higher magnification scale, using the predictions of the teacher as a target during the training. This process is repeated iteratively until we reach the target scale factor of the final model. The rationale behind our scale distillation is that the teacher aids the student diffusion model training by i) providing a target adapted to the current noise level rather than using the same target coming from ground truth data for all noise levels and ii) providing an accurate target as the teacher has a simpler task to solve. We empirically show that the distilled model significantly outperforms the model trained for high scales directly, specifically with few steps during inference. Having a strong diffusion model that requires only one step allows us to freeze the U-Net and fine-tune the decoder on top of it. We show that the combination of spatially distilled U-Net and fine-tuned decoder outperforms state-of-the-art methods requiring 200 steps with only one single step.

Title: Weaver: Foundation Models for Creative Writing

• Authors: Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, Yibin Liu, Jialong Wu, Shengwei Ding, Long Li, Zhiwei Huang, Xinle Deng, Teng Yu, Gangan Ma, Han Xiao, Zixin Chen, Danjun Xiang, Yunxia Wang, Yuanyuan Zhu, Yi Xiao, Jing Wang, Yiru Wang, Siran Ding, Jiayang Huang, Jiayi Xu, Yilihamu Tayier, Zhenyu Hu, Yuan Gao, Chengfeng Zheng, Yueshu Ye, Yihang Li, Lei Wan, Xinyue Jiang, Yujie Wang, Siyu Cheng, Zhule Song, Xiangru Tang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, Yuchen Eleanor Jiang, Wangchunshu Zhou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.17268
• Pdf URL: https://arxiv.org/pdf/2401.17268
• Copy Paste: [[2401.17268]] Weaver: Foundation Models for Creative Writing(https://arxiv.org/abs/2401.17268)
• Keywords: foundation model
• Abstract: This work introduces Weaver, our first family of large language models (LLMs) dedicated to content creation. Weaver is pre-trained on a carefully selected corpus that focuses on improving the writing capabilities of large language models. We then fine-tune Weaver for creative and professional writing purposes and align it to the preference of professional writers using a suit of novel methods for instruction data synthesis and LLM alignment, making it able to produce more human-like texts and follow more diverse instructions for content creation. The Weaver family consists of models of Weaver Mini (1.8B), Weaver Base (6B), Weaver Pro (14B), and Weaver Ultra (34B) sizes, suitable for different applications and can be dynamically dispatched by a routing agent according to query complexity to balance response quality and computation cost. Evaluation on a carefully curated benchmark for assessing the writing capabilities of LLMs shows Weaver models of all sizes outperform generalist LLMs several times larger than them. Notably, our most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes. Moreover, Weaver natively supports retrieval-augmented generation (RAG) and function calling (tool usage). We present various use cases of these abilities for improving AI-assisted writing systems, including integration of external knowledge bases, tools, or APIs, and providing personalized writing assistance. Furthermore, we discuss and summarize a guideline and best practices for pre-training and fine-tuning domain-specific LLMs.