2024-02-14

Title: Enhancing Amharic-LLaMA: Integrating Task Specific and Generative Datasets

• Authors: Israel Abebe Azime, Mitiku Yohannes Fuge, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.08015
• Pdf URL: https://arxiv.org/pdf/2402.08015
• Copy Paste: [[2402.08015]] Enhancing Amharic-LLaMA: Integrating Task Specific and Generative Datasets(https://arxiv.org/abs/2402.08015)
• Keywords: generative
• Abstract: Large language models (LLMs) have received a lot of attention in natural language processing (NLP) research because of their exceptional performance in understanding and generating human languages. However, low-resource languages are left behind due to the unavailability of resources. In this work, we focus on enhancing the LLaMA-2-Amharic model by integrating task-specific and generative datasets to improve language model performance for Amharic. We compile an Amharic instruction fine-tuning dataset and fine-tuned LLaMA-2-Amharic model. The fine-tuned model shows promising results in different NLP tasks. We open-source our dataset creation pipeline, instruction datasets, trained models, and evaluation outputs to promote language-specific studies on these models.

Title: Nearest Neighbour Score Estimators for Diffusion Generative Models

• Authors: Matthew Niedoba, Dylan Green, Saeid Naderiparizi, Vasileios Lioutas, Jonathan Wilder Lavington, Xiaoxuan Liang, Yunpeng Liu, Ke Zhang, Setareh Dabiri, Adam Ścibior, Berend Zwartsenberg, Frank Wood
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08018
• Pdf URL: https://arxiv.org/pdf/2402.08018
• Copy Paste: [[2402.08018]] Nearest Neighbour Score Estimators for Diffusion Generative Models(https://arxiv.org/abs/2402.08018)
• Keywords: diffusion, generative
• Abstract: Score function estimation is the cornerstone of both training and sampling from diffusion generative models. Despite this fact, the most commonly used estimators are either biased neural network approximations or high variance Monte Carlo estimators based on the conditional score. We introduce a novel nearest neighbour score function estimator which utilizes multiple samples from the training set to dramatically decrease estimator variance. We leverage our low variance estimator in two compelling applications. Training consistency models with our estimator, we report a significant increase in both convergence speed and sample quality. In diffusion models, we show that our estimator can replace a learned network for probability-flow ODE integration, opening promising new avenues of future research.

Title: UGMAE: A Unified Framework for Graph Masked Autoencoders

• Authors: Yijun Tian, Chuxu Zhang, Ziyi Kou, Zheyuan Liu, Xiangliang Zhang, Nitesh V. Chawla
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08023
• Pdf URL: https://arxiv.org/pdf/2402.08023
• Copy Paste: [[2402.08023]] UGMAE: A Unified Framework for Graph Masked Autoencoders(https://arxiv.org/abs/2402.08023)
• Keywords: self-supervised, generative
• Abstract: Generative self-supervised learning on graphs, particularly graph masked autoencoders, has emerged as a popular learning paradigm and demonstrated its efficacy in handling non-Euclidean data. However, several remaining issues limit the capability of existing methods: 1) the disregard of uneven node significance in masking, 2) the underutilization of holistic graph information, 3) the ignorance of semantic knowledge in the representation space due to the exclusive use of reconstruction loss in the output space, and 4) the unstable reconstructions caused by the large volume of masked contents. In light of this, we propose UGMAE, a unified framework for graph masked autoencoders to address these issues from the perspectives of adaptivity, integrity, complementarity, and consistency. Specifically, we first develop an adaptive feature mask generator to account for the unique significance of nodes and sample informative masks (adaptivity). We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information and emphasize the topological proximity between neighbors (integrity). After that, we present a bootstrapping-based similarity module to encode the high-level semantic knowledge in the representation space, complementary to the low-level reconstruction in the output space (complementarity). Finally, we build a consistency assurance module to provide reconstruction objectives with extra stabilized consistency targets (consistency). Extensive experiments demonstrate that UGMAE outperforms both contrastive and generative state-of-the-art baselines on several tasks across multiple datasets.

Title: Text-centric Alignment for Multi-Modality Learning

• Authors: Yun-Da Tsai, Ting-Yu Yen, Pei-Fu Guo, Zhe-Yan Li, Shou-De Lin
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08086
• Pdf URL: https://arxiv.org/pdf/2402.08086
• Copy Paste: [[2402.08086]] Text-centric Alignment for Multi-Modality Learning(https://arxiv.org/abs/2402.08086)
• Keywords: foundation model, in-context
• Abstract: This research paper addresses the challenge of modality mismatch in multimodal learning, where the modalities available during inference differ from those available at training. We propose the Text-centric Alignment for Multi-Modality Learning (TAMML) approach, an innovative method that utilizes Large Language Models (LLMs) with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, TAMML demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. TAMML not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.

Title: Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation

• Authors: AprilPyone MaungMaung, Huy H. Nguyen, Hitoshi Kiya, Isao Echizen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08200
• Pdf URL: https://arxiv.org/pdf/2402.08200
• Copy Paste: [[2402.08200]] Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation(https://arxiv.org/abs/2402.08200)
• Keywords: diffusion
• Abstract: We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Although the previous work detects spurious features in a large-scale dataset like ImageNet and introduces Spurious ImageNet, we found that not all spurious images are spurious across different classifiers. Although spurious images help measure the reliance of a classifier, filtering many images from the Internet to find more spurious features is time-consuming. To this end, we utilize an existing approach of personalizing large-scale text-to-image diffusion models with available discovered spurious images and propose a new spurious feature similarity loss based on neural features of an adversarially robust model. Precisely, we fine-tune Stable Diffusion with several reference images from Spurious ImageNet with a modified objective incorporating the proposed spurious-feature similarity loss. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers. Moreover, the generated spurious images are visually similar to reference images from Spurious ImageNet.

Title: Improving Black-box Robustness with In-Context Rewriting

• Authors: Kyle O'Brien, Nathan Ng, Isha Puri, Jorge Mendez, Hamid Palangi, Yoon Kim, Marzyeh Ghassemi, Thomas Hartvigsen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08225
• Pdf URL: https://arxiv.org/pdf/2402.08225
• Copy Paste: [[2402.08225]] Improving Black-box Robustness with In-Context Rewriting(https://arxiv.org/abs/2402.08225)
• Keywords: in-context
• Abstract: Machine learning models often excel on in-distribution (ID) data but struggle with unseen out-of-distribution (OOD) inputs. Most techniques for improving OOD robustness are not applicable to settings where the model is effectively a black box, such as when the weights are frozen, retraining is costly, or the model is leveraged via an API. Test-time augmentation (TTA) is a simple post-hoc technique for improving robustness that sidesteps black-box constraints by aggregating predictions across multiple augmentations of the test input. TTA has seen limited use in NLP due to the challenge of generating effective natural language augmentations. In this work, we propose LLM-TTA, which uses LLM-generated augmentations as TTA's augmentation function. LLM-TTA outperforms conventional augmentation functions across sentiment, toxicity, and news classification tasks for BERT and T5 models, with BERT's OOD robustness improving by an average of 4.30 percentage points without regressing average ID performance. We explore selectively augmenting inputs based on prediction entropy to reduce the rate of expensive LLM augmentations, allowing us to maintain performance gains while reducing the average number of generated augmentations by 57.76%. LLM-TTA is agnostic to the task model architecture, does not require OOD labels, and is effective across low and high-resource settings. We share our data, models, and code for reproducibility.

Title: APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

• Authors: Barathi Subramanian, Rathinaraja Jeyaraj, Rakhmonov Akhrorjon Akhmadjon Ugli, Jeonghong Kim
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.08244
• Pdf URL: https://arxiv.org/pdf/2402.08244
• Copy Paste: [[2402.08244]] APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks(https://arxiv.org/abs/2402.08244)
• Keywords: anomaly
• Abstract: Activation function is a pivotal component of deep learning, facilitating the extraction of intricate data patterns. While classical activation functions like ReLU and its variants are extensively utilized, their static nature and simplicity, despite being advantageous, often limit their effectiveness in specialized tasks. The trainable activation functions also struggle sometimes to adapt to the unique characteristics of the data. Addressing these limitations, we introduce a novel trainable activation function, adaptive piecewise approximated activation linear unit (APALU), to enhance the learning performance of deep learning across a broad range of tasks. It presents a unique set of features that enable it to maintain stability and efficiency in the learning process while adapting to complex data representations. Experiments reveal significant improvements over widely used activation functions for different tasks. In image classification, APALU increases MobileNet and GoogleNet accuracy by 0.37% and 0.04%, respectively, on the CIFAR10 dataset. In anomaly detection, it improves the average area under the curve of One-CLASS Deep SVDD by 0.8% on the MNIST dataset, 1.81% and 1.11% improvements with DifferNet, and knowledge distillation, respectively, on the MVTech dataset. Notably, APALU achieves 100% accuracy on a sign language recognition task with a limited dataset. For regression tasks, APALU enhances the performance of deep neural networks and recurrent neural networks on different datasets. These improvements highlight the robustness and adaptability of APALU across diverse deep-learning applications.

Title: A Dense Reward View on Aligning Text-to-Image Diffusion with Preference

• Authors: Shentao Yang, Tianqi Chen, Mingyuan Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08265
• Pdf URL: https://arxiv.org/pdf/2402.08265
• Copy Paste: [[2402.08265]] A Dense Reward View on Aligning Text-to-Image Diffusion with Preference(https://arxiv.org/abs/2402.08265)
• Keywords: diffusion
• Abstract: Aligning text-to-image diffusion model (T2I) with preference has been gaining increasing research attention. While prior works exist on directly optimizing T2I by preference data, these methods are developed under the bandit assumption of a latent reward on the entire diffusion reverse chain, while ignoring the sequential nature of the generation process. From literature, this may harm the efficacy and efficiency of alignment. In this paper, we take on a finer dense reward perspective and derive a tractable alignment objective that emphasizes the initial steps of the T2I reverse chain. In particular, we introduce temporal discounting into the DPO-style explicit-reward-free loss, to break the temporal symmetry therein and suit the T2I generation hierarchy. In experiments on single and multiple prompt generation, our method is competitive with strong relevant baselines, both quantitatively and qualitatively. Further studies are conducted to illustrate the insight of our approach.

Title: One-to-many Reconstruction of 3D Geometry of cultural Artifacts using a synthetically trained Generative Model

• Authors: Thomas Pöllabauer, Julius Kühn, Jiayi Li, Arjan Kuijper
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08310
• Pdf URL: https://arxiv.org/pdf/2402.08310
• Copy Paste: [[2402.08310]] One-to-many Reconstruction of 3D Geometry of cultural Artifacts using a synthetically trained Generative Model(https://arxiv.org/abs/2402.08310)
• Keywords: generative
• Abstract: Estimating the 3D shape of an object using a single image is a difficult problem. Modern approaches achieve good results for general objects, based on real photographs, but worse results on less expressive representations such as historic sketches. Our automated approach generates a variety of detailed 3D representation from a single sketch, depicting a medieval statue, and can be guided by multi-modal inputs, such as text prompts. It relies solely on synthetic data for training, making it adoptable even in cases of only small numbers of training examples. Our solution allows domain experts such as a curators to interactively reconstruct potential appearances of lost artifacts.

Title: Approximating Families of Sharp Solutions to Fisher's Equation with Physics-Informed Neural Networks

• Authors: Franz M. Rohrhofer, Stefan Posch, Clemens Gößnitzer, Bernhard C. Geiger
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08313
• Pdf URL: https://arxiv.org/pdf/2402.08313
• Copy Paste: [[2402.08313]] Approximating Families of Sharp Solutions to Fisher's Equation with Physics-Informed Neural Networks(https://arxiv.org/abs/2402.08313)
• Keywords: diffusion
• Abstract: This paper employs physics-informed neural networks (PINNs) to solve Fisher's equation, a fundamental representation of a reaction-diffusion system with both simplicity and significance. The focus lies specifically in investigating Fisher's equation under conditions of large reaction rate coefficients, wherein solutions manifest as traveling waves, posing a challenge for numerical methods due to the occurring steepness of the wave front. To address optimization challenges associated with the standard PINN approach, a residual weighting scheme is introduced. This scheme is designed to enhance the tracking of propagating wave fronts by considering the reaction term in the reaction-diffusion equation. Furthermore, a specific network architecture is studied which is tailored for solutions in the form of traveling waves. Lastly, the capacity of PINNs to approximate an entire family of solutions is assessed by incorporating the reaction rate coefficient as an additional input to the network architecture. This modification enables the approximation of the solution across a broad and continuous range of reaction rate coefficients, thus solving a class of reaction-diffusion systems using a single PINN instance.

Title: Visually Dehallucinative Instruction Generation

• Authors: Sungguk Cha, Jusung Lee, Younghyun Lee, Cheoljong Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08348
• Pdf URL: https://arxiv.org/pdf/2402.08348
• Copy Paste: [[2402.08348]] Visually Dehallucinative Instruction Generation(https://arxiv.org/abs/2402.08348)
• Keywords: generative
• Abstract: In recent years, synthetic visual instructions by generative language model have demonstrated plausible text generation performance on the visual question-answering tasks. However, challenges persist in the hallucination of generative language models, i.e., the generated image-text data contains unintended contents. This paper presents a novel and scalable method for generating visually dehallucinative instructions, dubbed CAP2QA, that constrains the scope to only image contents. Our key contributions lie in introducing image-aligned instructive QA dataset CAP2QA-COCO and its scalable recipe. In our experiments, we compare synthetic visual instruction datasets that share the same source data by visual instruction tuning and conduct general visual recognition tasks. It shows that our proposed method significantly reduces visual hallucination while consistently improving visual recognition ability and expressiveness.

Title: Leveraging Self-Supervised Instance Contrastive Learning for Radar Object Detection

• Authors: Colin Decourt, Rufin VanRullen, Didier Salle, Thomas Oberlin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08427
• Pdf URL: https://arxiv.org/pdf/2402.08427
• Copy Paste: [[2402.08427]] Leveraging Self-Supervised Instance Contrastive Learning for Radar Object Detection(https://arxiv.org/abs/2402.08427)
• Keywords: self-supervised
• Abstract: In recent years, driven by the need for safer and more autonomous transport systems, the automotive industry has shifted toward integrating a growing number of Advanced Driver Assistance Systems (ADAS). Among the array of sensors employed for object recognition tasks, radar sensors have emerged as a formidable contender due to their abilities in adverse weather conditions or low-light scenarios and their robustness in maintaining consistent performance across diverse environments. However, the small size of radar datasets and the complexity of the labelling of those data limit the performance of radar object detectors. Driven by the promising results of self-supervised learning in computer vision, this paper presents RiCL, an instance contrastive learning framework to pre-train radar object detectors. We propose to exploit the detection from the radar and the temporal information to pre-train the radar object detection model in a self-supervised way using contrastive learning. We aim to pre-train an object detector's backbone, head and neck to learn with fewer data. Experiments on the CARRADA and the RADDet datasets show the effectiveness of our approach in learning generic representations of objects in range-Doppler maps. Notably, our pre-training strategy allows us to use only 20% of the labelled data to reach a similar mAP@0.5 than a supervised approach using the whole training set.

Title: P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentation

• Authors: Zi Ye, Tianxiang Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08506
• Pdf URL: https://arxiv.org/pdf/2402.08506
• Copy Paste: [[2402.08506]] P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentation(https://arxiv.org/abs/2402.08506)
• Keywords: diffusion
• Abstract: In pediatric cardiology, the accurate and immediate assessment of cardiac function through echocardiography is important since it can determine whether urgent intervention is required in many emergencies. However, echocardiography is characterized by ambiguity and heavy background noise interference, bringing more difficulty to accurate segmentation. Present methods lack efficiency and are also prone to mistakenly segmenting some background noise areas as the left ventricular area due to noise disturbance. To relieve the two issues, we introduce P-Mamba for efficient pediatric echocardiographic left ventricular segmentation. Specifically, we turn to the recently proposed vision mamba layers in our vision mamba encoder branch to improve the computing and memory efficiency of our model while modeling global dependencies. In the other DWT-based PMD encoder branch, we devise DWT-based Perona-Malik Diffusion (PMD) Blocks that utilize PMD for noise suppression, while simultaneously preserving the local shape cues of the left ventricle. Leveraging the strengths of both the two encoder branches, P-Mamba achieves superior accuracy and efficiency to established models, such as vision transformers with quadratic and linear computational complexity. This innovative approach promises significant advancements in pediatric cardiac imaging and beyond.

Title: A Distributional Analogue to the Successor Representation

• Authors: Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Marc G. Bellemare, Mark Rowland
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08530
• Pdf URL: https://arxiv.org/pdf/2402.08530
• Copy Paste: [[2402.08530]] A Distributional Analogue to the Successor Representation(https://arxiv.org/abs/2402.08530)
• Keywords: generative
• Abstract: This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representation (SR) describes the expected consequences of behaving according to a given policy, our distributional successor measure (SM) describes the distributional consequences of this behaviour. We formulate the distributional SM as a distribution over distributions and provide theory connecting it with distributional and model-based reinforcement learning. Moreover, we propose an algorithm that learns the distributional SM from data by minimizing a two-level maximum mean discrepancy. Key to our method are a number of algorithmic techniques that are independently valuable for learning generative models of state. As an illustration of the usefulness of the distributional SM, we show that it enables zero-shot risk-sensitive policy evaluation in a way that was not previously possible.

Title: Generative VS non-Generative Models in Engineering Shape Optimization

• Authors: Muhammad Usama, Zahid Masood, Shahroz Khan, Konstantinos Kostas, Panagiotis Kaklis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08540
• Pdf URL: https://arxiv.org/pdf/2402.08540
• Copy Paste: [[2402.08540]] Generative VS non-Generative Models in Engineering Shape Optimization(https://arxiv.org/abs/2402.08540)
• Keywords: generative
• Abstract: In this work, we perform a systematic comparison of the effectiveness and efficiency of generative and non-generative models in constructing design spaces for novel and efficient design exploration and shape optimization. We apply these models in the case of airfoil/hydrofoil design and conduct the comparison on the resulting design spaces. A conventional Generative Adversarial Network (GAN) and a state-of-the-art generative model, the Performance-Augmented Diverse Generative Adversarial Network (PaDGAN), are juxtaposed with a linear non-generative model based on the coupling of the Karhunen-Lo\`eve Expansion and a physics-informed Shape Signature Vector (SSV-KLE). The comparison demonstrates that, with an appropriate shape encoding and a physics-augmented design space, non-generative models have the potential to cost-effectively generate high-performing valid designs with enhanced coverage of the design space. In this work, both approaches are applied to two large foil profile datasets comprising real-world and artificial designs generated through either a profile-generating parametric model or deep-learning approach. These datasets are further enriched with integral properties of their members' shapes as well as physics-informed parameters. Our results illustrate that the design spaces constructed by the non-generative model outperform the generative model in terms of design validity, generating robust latent spaces with none or significantly fewer invalid designs when compared to generative models. We aspire that these findings will aid the engineering design community in making informed decisions when constructing designs spaces for shape optimization, as we have show that under certain conditions computationally inexpensive approaches can closely match or even outperform state-of-the art generative models.

Title: Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases

• Authors: Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, Dacheng Tao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08552
• Pdf URL: https://arxiv.org/pdf/2402.08552
• Copy Paste: [[2402.08552]] Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases(https://arxiv.org/abs/2402.08552)
• Keywords: diffusion, generative
• Abstract: Bridging the gap between diffusion models and human preferences is crucial for their integration into practical generative workflows. While optimizing downstream reward models has emerged as a promising alignment strategy, concerns arise regarding the risk of excessive optimization with learned reward models, which potentially compromises ground-truth performance. In this work, we confront the reward overoptimization problem in diffusion model alignment through the lenses of both inductive and primacy biases. We first identify the divergence of current methods from the temporal inductive bias inherent in the multi-step denoising process of diffusion models as a potential source of overoptimization. Then, we surprisingly discover that dormant neurons in our critic model act as a regularization against overoptimization, while active neurons reflect primacy bias in this setting. Motivated by these observations, we propose Temporal Diffusion Policy Optimization with critic active neuron Reset (TDPO-R), a policy gradient algorithm that exploits the temporal inductive bias of intermediate timesteps, along with a novel reset strategy that targets active neurons to counteract the primacy bias. Empirical results demonstrate the superior efficacy of our algorithms in mitigating reward overoptimization.

Title: Denoising Diffusion Restoration Tackles Forward and Inverse Problems for the Laplace Operator

• Authors: Amartya Mukherjee, Melissa M. Stadt, Lena Podina, Mohammad Kohandel, Jun Liu
• Subjects: cs.LG, cs.CV, math.AP
• Abstract URL: https://arxiv.org/abs/2402.08563
• Pdf URL: https://arxiv.org/pdf/2402.08563
• Copy Paste: [[2402.08563]] Denoising Diffusion Restoration Tackles Forward and Inverse Problems for the Laplace Operator(https://arxiv.org/abs/2402.08563)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as a promising class of generative models that map noisy inputs to realistic images. More recently, they have been employed to generate solutions to partial differential equations (PDEs). However, they still struggle with inverse problems in the Laplacian operator, for instance, the Poisson equation, because the eigenvalues that are large in magnitude amplify the measurement noise. This paper presents a novel approach for the inverse and forward solution of PDEs through the use of denoising diffusion restoration models (DDRM). DDRMs were used in linear inverse problems to restore original clean signals by exploiting the singular value decomposition (SVD) of the linear operator. Equivalently, we present an approach to restore the solution and the parameters in the Poisson equation by exploiting the eigenvalues and the eigenfunctions of the Laplacian operator. Our results show that using denoising diffusion restoration significantly improves the estimation of the solution and parameters. Our research, as a result, pioneers the integration of diffusion models with the principles of underlying physics to solve PDEs.

Title: Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing

• Authors: Yunji Jung, Seokju Lee, Tair Djanibekov, Hyunjung Shim, Jongchul Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08601
• Pdf URL: https://arxiv.org/pdf/2402.08601
• Copy Paste: [[2402.08601]] Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing(https://arxiv.org/abs/2402.08601)
• Keywords: diffusion
• Abstract: Text-guided non-rigid editing involves complex edits for input images, such as changing motion or compositions within their surroundings. Since it requires manipulating the input structure, existing methods often struggle with preserving object identity and background, particularly when combined with Stable Diffusion. In this work, we propose a training-free approach for non-rigid editing with Stable Diffusion, aimed at improving the identity preservation quality without compromising editability. Our approach comprises three stages: text optimization, latent inversion, and timestep-aware text injection sampling. Inspired by the recent success of Imagic, we employ their text optimization for smooth editing. Then, we introduce latent inversion to preserve the input image's identity without additional model fine-tuning. To fully utilize the input reconstruction ability of latent inversion, we suggest timestep-aware text inject sampling. This effectively retains the structure of the input image by injecting the source text prompt in early sampling steps and then transitioning to the target prompt in subsequent sampling steps. This strategic approach seamlessly harmonizes with text optimization, facilitating complex non-rigid edits to the input without losing the original identity. We demonstrate the effectiveness of our method in terms of identity preservation, editability, and aesthetic quality through extensive experiments.

Title: CaPS: Collaborative and Private Synthetic Data Generation from Distributed Sources

• Authors: Sikha Pentyala, Mayana Pereira, Martine De Cock
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.08614
• Pdf URL: https://arxiv.org/pdf/2402.08614
• Copy Paste: [[2402.08614]] CaPS: Collaborative and Private Synthetic Data Generation from Distributed Sources(https://arxiv.org/abs/2402.08614)
• Keywords: generative
• Abstract: Data is the lifeblood of the modern world, forming a fundamental part of AI, decision-making, and research advances. With increase in interest in data, governments have taken important steps towards a regulated data world, drastically impacting data sharing and data usability and resulting in massive amounts of data confined within the walls of organizations. While synthetic data generation (SDG) is an appealing solution to break down these walls and enable data sharing, the main drawback of existing solutions is the assumption of a trusted aggregator for generative model training. Given that many data holders may not want to, or be legally allowed to, entrust a central entity with their raw data, we propose a framework for the collaborative and private generation of synthetic tabular data from distributed data holders. Our solution is general, applicable to any marginal-based SDG, and provides input privacy by replacing the trusted aggregator with secure multi-party computation (MPC) protocols and output privacy via differential privacy (DP). We demonstrate the applicability and scalability of our approach for the state-of-the-art select-measure-generate SDG algorithms MWEM+PGM and AIM.

Title: Generating Universal Adversarial Perturbations for Quantum Classifiers

• Authors: Gautham Anil, Vishnu Vinod, Apurva Narayan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.08648
• Pdf URL: https://arxiv.org/pdf/2402.08648
• Copy Paste: [[2402.08648]] Generating Universal Adversarial Perturbations for Quantum Classifiers(https://arxiv.org/abs/2402.08648)
• Keywords: generative
• Abstract: Quantum Machine Learning (QML) has emerged as a promising field of research, aiming to leverage the capabilities of quantum computing to enhance existing machine learning methodologies. Recent studies have revealed that, like their classical counterparts, QML models based on Parametrized Quantum Circuits (PQCs) are also vulnerable to adversarial attacks. Moreover, the existence of Universal Adversarial Perturbations (UAPs) in the quantum domain has been demonstrated theoretically in the context of quantum classifiers. In this work, we introduce QuGAP: a novel framework for generating UAPs for quantum classifiers. We conceptualize the notion of additive UAPs for PQC-based classifiers and theoretically demonstrate their existence. We then utilize generative models (QuGAP-A) to craft additive UAPs and experimentally show that quantum classifiers are susceptible to such attacks. Moreover, we formulate a new method for generating unitary UAPs (QuGAP-U) using quantum generative models and a novel loss function based on fidelity constraints. We evaluate the performance of the proposed framework and show that our method achieves state-of-the-art misclassification rates, while maintaining high fidelity between legitimate and adversarial samples.

Title: Learning Continuous 3D Words for Text-to-Image Generation

• Authors: Ta-Ying Cheng, Matheus Gadelha, Thibault Groueix, Matthew Fisher, Radomir Mech, Andrew Markham, Niki Trigoni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.08654
• Pdf URL: https://arxiv.org/pdf/2402.08654
• Copy Paste: [[2402.08654]] Learning Continuous 3D Words for Text-to-Image Generation(https://arxiv.org/abs/2402.08654)
• Keywords: diffusion, generative
• Abstract: Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words

Title: Target Score Matching

• Authors: Valentin De Bortoli, Michael Hutchinson, Peter Wirnsberger, Arnaud Doucet
• Subjects: cs.LG, stat.CO, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.08667
• Pdf URL: https://arxiv.org/pdf/2402.08667
• Copy Paste: [[2402.08667]] Target Score Matching(https://arxiv.org/abs/2402.08667)
• Keywords: diffusion
• Abstract: Denoising Score Matching estimates the score of a noised version of a target distribution by minimizing a regression loss and is widely used to train the popular class of Denoising Diffusion Models. A well known limitation of Denoising Score Matching, however, is that it yields poor estimates of the score at low noise levels. This issue is particularly unfavourable for problems in the physical sciences and for Monte Carlo sampling tasks for which the score of the clean original target is known. Intuitively, estimating the score of a slightly noised version of the target should be a simple task in such cases. In this paper, we address this shortcoming and show that it is indeed possible to leverage knowledge of the target score. We present a Target Score Identity and corresponding Target Score Matching regression loss which allows us to obtain score estimates admitting favourable properties at low noise levels.

Title: IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation

• Authors: Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, Filippos Kokkinos
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.08682
• Pdf URL: https://arxiv.org/pdf/2402.08682
• Copy Paste: [[2402.08682]] IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation(https://arxiv.org/abs/2402.08682)
• Keywords: diffusion
• Abstract: Most text-to-3D generators build upon off-the-shelf text-to-image models trained on billions of images. They use variants of Score Distillation Sampling (SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation is to fine-tune the 2D generator to be multi-view aware, which can help distillation or can be combined with reconstruction networks to output 3D objects directly. In this paper, we further explore the design space of text-to-3D models. We significantly improve multi-view generation by considering video instead of image generators. Combined with a 3D reconstruction algorithm which, by using Gaussian splatting, can optimize a robust image-based loss, we directly produce high-quality 3D outputs from the generated views. Our new method, IM-3D, reduces the number of evaluations of the 2D generator network 10-100x, resulting in a much more efficient pipeline, better quality, fewer geometric inconsistencies, and higher yield of usable 3D assets.