secure

Title: Pudding: Private User Discovery in Anonymity Networks. (arXiv:2311.10825v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10825
• Code URL: null
• Copy Paste: [[2311.10825]] Pudding: Private User Discovery in Anonymity Networks(http://arxiv.org/abs/2311.10825)
• Summary:

  Anonymity networks allow messaging with metadata privacy, providing better privacy than popular encrypted messaging applications. However, contacting a user on an anonymity network currently requires knowing their public key or similar high-entropy information, as these systems lack a privacy-preserving mechanism for contacting a user via a short, human-readable username. Previous research suggests that this is a barrier to widespread adoption.

  In this paper we propose Pudding, a novel private user discovery protocol that allows a user to be contacted on an anonymity network knowing only their email address. Our protocol hides contact relationships between users, prevents impersonation, and conceals which usernames are registered on the network. Pudding is Byzantine fault tolerant, remaining available and secure as long as less than one third of servers are crashed, unavailable, or malicious. It can be deployed on Loopix and Nym without changes to the underlying anonymity network protocol, and it supports mobile devices with intermittent network connectivity. We demonstrate the practicality of Pudding with a prototype using the Nym anonymity network. We also formally define the security and privacy goals of our protocol and conduct a thorough analysis to assess its compliance with these definitions.

Title: Secure Software Development: Issues and Challenges. (arXiv:2311.11021v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.11021
• Code URL: null
• Copy Paste: [[2311.11021]] Secure Software Development: Issues and Challenges(http://arxiv.org/abs/2311.11021)
• Summary:

  In recent years, technology has advanced considerably with the introduction of many systems including advanced robotics, big data analytics, cloud computing, machine learning and many more. The opportunities to exploit the yet to come security that comes with these systems are going toe to toe with new releases of security protocols to combat this exploitation to provide a secure system. The digitization of our lives proves to solve our human problems as well as improve quality of life but because it is digitalized, information and technology could be misused for other malicious gains. Hackers aim to steal the data of innocent people to use it for other causes such as identity fraud, scams and many more. This issue can be corrected during the software development life cycle, integrating security across the development phases, and testing of the software is done early to reduce the number of vulnerabilities that might or might not heavily impact an organisation depending on the range of the attack. The goal of a secured system software is to prevent such exploitations from ever happening by conducting a system life cycle where through planning and testing is done to maximise security while maintaining functionality of the system. In this paper, we are going to discuss the recent trends in security for system development as well as our predictions and suggestions to improve the current security practices in this industry.

security

Title: Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks. (arXiv:2311.10944v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10944
• Code URL: null
• Copy Paste: [[2311.10944]] Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks(http://arxiv.org/abs/2311.10944)
• Summary:

  Deception detection is gaining increasing interest due to ethical and security concerns. This paper explores the application of convolutional neural networks for the purpose of multimodal deception detection. We use a dataset built by interviewing 104 subjects about two topics, with one truthful and one falsified response from each subject about each topic. In particular, we make three main contributions. First, we extract linguistic and physiological features from this data to train and construct the neural network models. Second, we propose a fused convolutional neural network model using both modalities in order to achieve an improved overall performance. Third, we compare our new approach with earlier methods designed for multimodal deception detection. We find that our system outperforms regular classification methods; our results indicate the feasibility of using neural networks for deception detection even in the presence of limited amounts of data.

Title: Dazed & Confused: A Large-Scale Real-World User Study of reCAPTCHAv2. (arXiv:2311.10911v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10911
• Code URL: null
• Copy Paste: [[2311.10911]] Dazed & Confused: A Large-Scale Real-World User Study of reCAPTCHAv2(http://arxiv.org/abs/2311.10911)
• Summary:

  Since about 2003, captchas have been widely used as a barrier against bots, while simultaneously annoying great multitudes of users worldwide. As their use grew, techniques to defeat or bypass captchas kept improving, while captchas themselves evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots and humans. Given this long-standing and still-ongoing arms race, it is important to investigate usability, solving performance, and user perceptions of modern captchas. In this work, we do so via a large-scale (over 3, 600 distinct users) 13-month unbiased user study and post-study survey. The study, conducted at a large public university, was based on a live account creation and password recovery service with currently prevalent captcha type: reCAPTCHAv2.

  Results show that, with more attempts, users improve in solving checkbox challenges. For website developers and user study designers, results indicate that the website context directly influences (with statistically significant differences) solving time between password recovery and account creation. We consider the impact of participants' major and education level, showing that certain majors exhibit better performance, while, in general, education level has a direct impact on solving time. Unsurprisingly, we discover that participants find image challenges to be annoying, while checkbox challenges are perceived as easy. We also show that, rated via System Usability Scale (SUS), image tasks are viewed as "OK", while checkbox tasks are viewed as "good". We explore the cost and security of reCAPTCHAv2 and conclude that it has an immense cost and no security. Overall, we believe that this study's results prompt a natural conclusion: reCAPTCHAv2 and similar reCAPTCHA technology should be deprecated.

Title: Reveal the Mathematical Structures of Honeyword Security Metrics. (arXiv:2311.10960v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10960
• Code URL: null
• Copy Paste: [[2311.10960]] Reveal the Mathematical Structures of Honeyword Security Metrics(http://arxiv.org/abs/2311.10960)
• Summary:

  Honeyword is a representative ``honey" technique to detect intruders by luring them with decoy data. This kind of honey technique blends a primary object (from distribution $P$) with decoy samples (from distribution $Q$). In this research, we focus on two key Honeyword security metrics: the flatness function and the success-number function. Previous researchers are engaged in designing experimental methods to estimate their values. We've derived theoretical formulas on both metrics of the strongest $\mathcal{A}$ using the optimal guessing strategy, marking a first in the field.

  The mathematical structures of these metrics are intriguing: the flatness function has an expression as $\epsilon(i)=\sum_{j=1}^{i}\int_{0}^{+\infty}\tbinom{k-1}{j-1} f(x)G^{k-j}(x)(1-G(x))^{j-1}dx$. In particular, the most important one, $\epsilon(1)$ is $\frac{1}{k}(M-\int_{0}^{M}G^k(x)dx)+b$, where $M=\max_{x: Q(x)\neq 0}\frac{P(x)}{Q(x)}$, $b=\sum_{x: Q(x)=0}P(x)$, and $G$ is a cumulative distribution function derived from $P$ and $Q$. This formula provides a criterion to compare different honey distributions: the one with smaller $M$ and $b$ is more satisfactory. The mathematical structure of the success-number function is a series of convolutions with beta distribution kernels: $\lambda_U(i)=U\sum_{j=1}^{i}\int_{\frac{1}{k}}^{1} \frac{\phi(x)}{1-\phi(x)} \tbinom{U-1}{j-1} x^{U-j}(1-x)^{j-1}dx$, where $U$ is the number of users in the system and $\phi(x)$ is a monotonically increasing function. For further elaboration, we made some representative calculations. Our findings offer insights into security assessments for Honeyword and similar honey techniques, contributing to enhanced security measures in these systems.

privacy

protect

defense

attack

Title: PACOL: Poisoning Attacks Against Continual Learners. (arXiv:2311.10919v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10919
• Code URL: null
• Copy Paste: [[2311.10919]] PACOL: Poisoning Attacks Against Continual Learners(http://arxiv.org/abs/2311.10919)
• Summary:

  Continual learning algorithms are typically exposed to untrusted sources that contain training data inserted by adversaries and bad actors. An adversary can insert a small number of poisoned samples, such as mislabeled samples from previously learned tasks, or intentional adversarial perturbed samples, into the training datasets, which can drastically reduce the model's performance. In this work, we demonstrate that continual learning systems can be manipulated by malicious misinformation and present a new category of data poisoning attacks specific for continual learners, which we refer to as {\em Poisoning Attacks Against Continual Learners} (PACOL). The effectiveness of labeling flipping attacks inspires PACOL; however, PACOL produces attack samples that do not change the sample's label and produce an attack that causes catastrophic forgetting. A comprehensive set of experiments shows the vulnerability of commonly used generative replay and regularization-based continual learning approaches against attack methods. We evaluate the ability of label-flipping and a new adversarial poison attack, namely PACOL proposed in this work, to force the continual learning system to forget the knowledge of a learned task(s). More specifically, we compared the performance degradation of continual learning systems trained on benchmark data streams with and without poisoning attacks. Moreover, we discuss the stealthiness of the attacks in which we test the success rate of data sanitization defense and other outlier detection-based defenses for filtering out adversarial samples.

robust

Title: A Language Agent for Autonomous Driving. (arXiv:2311.10813v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10813
• Code URL: https://github.com/usc-gvl/agent-driver
• Copy Paste: [[2311.10813]] A Language Agent for Autonomous Driving(http://arxiv.org/abs/2311.10813)
• Summary:

  Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a perception-prediction-planning framework, yet their systems do not capitalize on the inherent reasoning ability and experiential knowledge of humans. In this paper, we propose a fundamental paradigm shift from current pipelines, exploiting Large Language Models (LLMs) as a cognitive agent to integrate human-like intelligence into autonomous driving systems. Our approach, termed Agent-Driver, transforms the traditional autonomous driving pipeline by introducing a versatile tool library accessible via function calls, a cognitive memory of common sense and experiential knowledge for decision-making, and a reasoning engine capable of chain-of-thought reasoning, task planning, motion planning, and self-reflection. Powered by LLMs, our Agent-Driver is endowed with intuitive common sense and robust reasoning capabilities, thus enabling a more nuanced, human-like approach to autonomous driving. We evaluate our approach on the large-scale nuScenes benchmark, and extensive experiments substantiate that our Agent-Driver significantly outperforms the state-of-the-art driving methods by a large margin. Our approach also demonstrates superior interpretability and few-shot learning ability to these methods. Project page: \href{https://github.com/USC-GVL/Agent-Driver/blob/main/index.html}{here}.

Title: Domain Generalization of 3D Object Detection by Density-Resampling. (arXiv:2311.10845v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10845
• Code URL: null
• Copy Paste: [[2311.10845]] Domain Generalization of 3D Object Detection by Density-Resampling(http://arxiv.org/abs/2311.10845)
• Summary:

  Point-cloud-based 3D object detection suffers from performance degradation when encountering data with novel domain gaps. To tackle it, the single-domain generalization (SDG) aims to generalize the detection model trained in a limited single source domain to perform robustly on unexplored domains. In this paper, we propose an SDG method to improve the generalizability of 3D object detection to unseen target domains. Unlike prior SDG works for 3D object detection solely focusing on data augmentation, our work introduces a novel data augmentation method and contributes a new multi-task learning strategy in the methodology. Specifically, from the perspective of data augmentation, we design a universal physical-aware density-based data augmentation (PDDA) method to mitigate the performance loss stemming from diverse point densities. From the learning methodology viewpoint, we develop a multi-task learning for 3D object detection: during source training, besides the main standard detection task, we leverage an auxiliary self-supervised 3D scene restoration task to enhance the comprehension of the encoder on background and foreground details for better recognition and detection of objects. Furthermore, based on the auxiliary self-supervised task, we propose the first test-time adaptation method for domain generalization of 3D object detection, which efficiently adjusts the encoder's parameters to adapt to unseen target domains during testing time, to further bridge domain gaps. Extensive cross-dataset experiments covering "Car", "Pedestrian", and "Cyclist" detections, demonstrate our method outperforms state-of-the-art SDG methods and even overpass unsupervised domain adaptation methods under some circumstances. The code will be made publicly available.

Title: Towards Robust and Accurate Visual Prompting. (arXiv:2311.10992v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10992
• Code URL: null
• Copy Paste: [[2311.10992]] Towards Robust and Accurate Visual Prompting(http://arxiv.org/abs/2311.10992)
• Summary:

  Visual prompting, an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Whether a visual prompt derived from a robust model can inherit the robustness while suffering from the generalization performance decline, albeit for a downstream dataset that is different from the source dataset? In this work, we get an affirmative answer of the above question and give an explanation on the visual representation level. Moreover, we introduce a novel technique named Prompt Boundary Loose (PBL) to effectively mitigates the suboptimal results of visual prompt on standard accuracy without losing (or even significantly improving) its adversarial robustness when using a robust model as source model. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of our proposed method.

Title: Implicit Event-RGBD Neural SLAM. (arXiv:2311.11013v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.11013
• Code URL: null
• Copy Paste: [[2311.11013]] Implicit Event-RGBD Neural SLAM(http://arxiv.org/abs/2311.11013)
• Summary:

  Implicit neural SLAM has achieved remarkable progress recently. Nevertheless, existing methods face significant challenges in non-ideal scenarios, such as motion blur or lighting variation, which often leads to issues like convergence failures, localization drifts, and distorted mapping. To address these challenges, we propose $\textbf{EN-SLAM}$, the first event-RGBD implicit neural SLAM framework, which effectively leverages the high rate and high dynamic range advantages of event data for tracking and mapping. Specifically, EN-SLAM proposes a differentiable CRF (Camera Response Function) rendering technique to generate distinct RGB and event camera data via a shared radiance field, which is optimized by learning a unified implicit representation with the captured event and RGBD supervision. Moreover, based on the temporal difference property of events, we propose a temporal aggregating optimization strategy for the event joint tracking and global bundle adjustment, capitalizing on the consecutive difference constraints of events, significantly enhancing tracking accuracy and robustness. Finally, we construct the simulated dataset $\textbf{DEV-Indoors}$ and real captured dataset $\textbf{DEV-Reals}$ containing 6 scenes, 17 sequences with practical motion blur and lighting changes for evaluations. Experimental results show that our method outperforms the SOTA methods in both tracking ATE and mapping ACC with a real-time $17$ FPS in various challenging environments. The code and dataset will be released upon the paper publication.

Title: How False Data Affects Machine Learning Models in Electrochemistry?. (arXiv:2311.10795v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10795
• Code URL: https://github.com/Demodesu/How-False-Data-Affects-Machine-Learning-Models-in-Electrochemistry
• Copy Paste: [[2311.10795]] How False Data Affects Machine Learning Models in Electrochemistry?(http://arxiv.org/abs/2311.10795)
• Summary:

  Recently, the selection of machine learning model based on only the data distribution without concerning the noise of the data. This study aims to distinguish, which models perform well under noisy data, and establish whether stacking machine learning models actually provide robustness to otherwise weak-to-noise models. The electrochemical data were tested with 12 standalone models and stacking model. This includes XGB, LGBM, RF, GB, ADA, NN, ELAS, LASS, RIDGE, SVM, KNN, DT, and the stacking model. It is found that linear models handle noise well with the average error of (slope) to 1.75 F g-1 up to error per 100% percent noise added; but it suffers from prediction accuracy due to having an average of 60.19 F g-1 estimated at minimal error at 0% noise added. Tree-based models fail in terms of noise handling (average slope is 55.24 F g-1 at 100% percent noise), but it can provide higher prediction accuracy (lowest error of 23.9 F g-1) than that of linear. To address the controversial between prediction accuracy and error handling, the stacking model was constructed, which is not only show high accuracy (intercept of 25.03 F g-1), but it also exhibits good noise handling (slope of 43.58 F g-1), making stacking models a relatively low risk and viable choice for beginner and experienced machine learning research in electrochemistry. Even though neural networks (NN) are gaining popularity in the electrochemistry field. However, this study presents that NN is not suitable for electrochemical data, and improper tuning resulting in a model that is susceptible to noise. Thus, STACK models should provide better benefits in that even with untuned base models, they can achieve an accurate and noise-tolerant model. Overall, this work provides insight into machine learning model selection for electrochemical data, which should aid the understanding of data science in chemistry context.

Title: Robustness Enhancement in Neural Networks with Alpha-Stable Training Noise. (arXiv:2311.10803v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10803
• Code URL: null
• Copy Paste: [[2311.10803]] Robustness Enhancement in Neural Networks with Alpha-Stable Training Noise(http://arxiv.org/abs/2311.10803)
• Summary:

  With the increasing use of deep learning on data collected by non-perfect sensors and in non-perfect environments, the robustness of deep learning systems has become an important issue. A common approach for obtaining robustness to noise has been to train deep learning systems with data augmented with Gaussian noise. In this work, we challenge the common choice of Gaussian noise and explore the possibility of stronger robustness for non-Gaussian impulsive noise, specifically alpha-stable noise. Justified by the Generalized Central Limit Theorem and evidenced by observations in various application areas, alpha-stable noise is widely present in nature. By comparing the testing accuracy of models trained with Gaussian noise and alpha-stable noise on data corrupted by different noise, we find that training with alpha-stable noise is more effective than Gaussian noise, especially when the dataset is corrupted by impulsive noise, thus improving the robustness of the model. The generality of this conclusion is validated through experiments conducted on various deep learning models with image and time series datasets, and other benchmark corrupted datasets. Consequently, we propose a novel data augmentation method that replaces Gaussian noise, which is typically added to the training data, with alpha-stable noise.

biometric

steal

extraction

Title: Extraction and Summarization of Explicit Video Content using Multi-Modal Deep Learning. (arXiv:2311.10899v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10899
• Code URL: null
• Copy Paste: [[2311.10899]] Extraction and Summarization of Explicit Video Content using Multi-Modal Deep Learning(http://arxiv.org/abs/2311.10899)
• Summary:

  With the increase in video-sharing platforms across the internet, it is difficult for humans to moderate the data for explicit content. Hence, an automated pipeline to scan through video data for explicit content has become the need of the hour. We propose a novel pipeline that uses multi-modal deep learning to first extract the explicit segments of input videos and then summarize their content using text to determine its age appropriateness and age rating. We also evaluate our pipeline's effectiveness in the end using standard metrics.

Title: Use GPT-J Prompt Generation with RoBERTa for NER Models on Diagnosis Extraction of Periodontal Diagnosis from Electronic Dental Records. (arXiv:2311.10810v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10810
• Code URL: null
• Copy Paste: [[2311.10810]] Use GPT-J Prompt Generation with RoBERTa for NER Models on Diagnosis Extraction of Periodontal Diagnosis from Electronic Dental Records(http://arxiv.org/abs/2311.10810)
• Summary:

  This study explored the usability of prompt generation on named entity recognition (NER) tasks and the performance in different settings of the prompt. The prompt generation by GPT-J models was utilized to directly test the gold standard as well as to generate the seed and further fed to the RoBERTa model with the spaCy package. In the direct test, a lower ratio of negative examples with higher numbers of examples in prompt achieved the best results with a F1 score of 0.72. The performance revealed consistency, 0.92-0.97 in the F1 score, in all settings after training with the RoBERTa model. The study highlighted the importance of seed quality rather than quantity in feeding NER models. This research reports on an efficient and accurate way to mine clinical notes for periodontal diagnoses, allowing researchers to easily and quickly build a NER model with the prompt generation approach.

membership infer

federate

Title: Exploring Machine Learning Models for Federated Learning: A Review of Approaches, Performance, and Limitations. (arXiv:2311.10832v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10832
• Code URL: null
• Copy Paste: [[2311.10832]] Exploring Machine Learning Models for Federated Learning: A Review of Approaches, Performance, and Limitations(http://arxiv.org/abs/2311.10832)
• Summary:

  In the growing world of artificial intelligence, federated learning is a distributed learning framework enhanced to preserve the privacy of individuals' data. Federated learning lays the groundwork for collaborative research in areas where the data is sensitive. Federated learning has several implications for real-world problems. In times of crisis, when real-time decision-making is critical, federated learning allows multiple entities to work collectively without sharing sensitive data. This distributed approach enables us to leverage information from multiple sources and gain more diverse insights. This paper is a systematic review of the literature on privacy-preserving machine learning in the last few years based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Specifically, we have presented an extensive review of supervised/unsupervised machine learning algorithms, ensemble methods, meta-heuristic approaches, blockchain technology, and reinforcement learning used in the framework of federated learning, in addition to an overview of federated learning applications. This paper reviews the literature on the components of federated learning and its applications in the last few years. The main purpose of this work is to provide researchers and practitioners with a comprehensive overview of federated learning from the machine learning point of view. A discussion of some open problems and future research directions in federated learning is also provided.

fair

Title: Can Language Model Moderators Improve the Health of Online Discourse?. (arXiv:2311.10781v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10781
• Code URL: null
• Copy Paste: [[2311.10781]] Can Language Model Moderators Improve the Health of Online Discourse?(http://arxiv.org/abs/2311.10781)
• Summary:

  Human moderation of online conversation is essential to maintaining civility and focus in a dialogue, but is challenging to scale and harmful to moderators. The inclusion of sophisticated natural language generation modules as a force multiplier aid moderators is a tantalizing prospect, but adequate evaluation approaches have so far been elusive. In this paper, we establish a systematic definition of conversational moderation effectiveness through a multidisciplinary lens that incorporates insights from social science. We then propose a comprehensive evaluation framework that uses this definition to asses models' moderation capabilities independently of human intervention. With our framework, we conduct the first known study of conversational dialogue models as moderators, finding that appropriately prompted models can provide specific and fair feedback on toxic behavior but struggle to influence users to increase their levels of respect and cooperation.

interpretability

Title: Flexible Model Interpretability through Natural Language Model Editing. (arXiv:2311.10905v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10905
• Code URL: null
• Copy Paste: [[2311.10905]] Flexible Model Interpretability through Natural Language Model Editing(http://arxiv.org/abs/2311.10905)
• Summary:

  Model interpretability and model editing are crucial goals in the age of large language models. Interestingly, there exists a link between these two goals: if a method is able to systematically edit model behavior with regard to a human concept of interest, this editor method can help make internal representations more interpretable by pointing towards relevant representations and systematically manipulating them.

Title: Bit Cipher -- A Simple yet Powerful Word Representation System that Integrates Efficiently with Language Models. (arXiv:2311.11012v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.11012
• Code URL: null
• Copy Paste: [[2311.11012]] Bit Cipher -- A Simple yet Powerful Word Representation System that Integrates Efficiently with Language Models(http://arxiv.org/abs/2311.11012)
• Summary:

  While Large Language Models (LLMs) become ever more dominant, classic pre-trained word embeddings sustain their relevance through computational efficiency and nuanced linguistic interpretation. Drawing from recent studies demonstrating that the convergence of GloVe and word2vec optimizations all tend towards log-co-occurrence matrix variants, we construct a novel word representation system called Bit-cipher that eliminates the need of backpropagation while leveraging contextual information and hyper-efficient dimensionality reduction techniques based on unigram frequency, providing strong interpretability, alongside efficiency. We use the bit-cipher algorithm to train word vectors via a two-step process that critically relies on a hyperparameter -- bits -- that controls the vector dimension. While the first step trains the bit-cipher, the second utilizes it under two different aggregation modes -- summation or concatenation -- to produce contextually rich representations from word co-occurrences. We extend our investigation into bit-cipher's efficacy, performing probing experiments on part-of-speech (POS) tagging and named entity recognition (NER) to assess its competitiveness with classic embeddings like word2vec and GloVe. Additionally, we explore its applicability in LM training and fine-tuning. By replacing embedding layers with cipher embeddings, our experiments illustrate the notable efficiency of cipher in accelerating the training process and attaining better optima compared to conventional training paradigms. Experiments on the integration of bit-cipher embedding layers with Roberta, T5, and OPT, prior to or as a substitute for fine-tuning, showcase a promising enhancement to transfer learning, allowing rapid model convergence while preserving competitive performance.

explainability

watermark

diffusion

Title: Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression. (arXiv:2311.10794v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10794
• Code URL: null
• Copy Paste: [[2311.10794]] Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression(http://arxiv.org/abs/2311.10794)
• Summary:

  We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models (LDMs) in a distinct domain with high visual quality, prompt alignment and scene diversity. We choose sticker image generation as the target domain, as the images significantly differ from photorealistic samples typically generated by large-scale LDMs. We start with a competent text-to-image model, like Emu, and show that relying on prompt engineering with a photorealistic model to generate stickers leads to poor prompt alignment and scene diversity. To overcome these drawbacks, we first finetune Emu on millions of sticker-like images collected using weak supervision to elicit diversity. Next, we curate human-in-the-loop (HITL) Alignment and Style datasets from model generations, and finetune to improve prompt alignment and style alignment respectively. Sequential finetuning on these datasets poses a tradeoff between better style alignment and prompt alignment gains. To address this tradeoff, we propose a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff. Evaluation results show our method improves visual quality by 14%, prompt alignment by 16.2% and scene diversity by 15.3%, compared to prompt engineering the base Emu model for stickers generation.

Title: Make Pixels Dance: High-Dynamic Video Generation. (arXiv:2311.10982v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10982
• Code URL: null
• Copy Paste: [[2311.10982]] Make Pixels Dance: High-Dynamic Video Generation(http://arxiv.org/abs/2311.10982)
• Summary:

  Creating high-dynamic videos such as motion-rich actions and sophisticated visual effects poses a significant challenge in the field of artificial intelligence. Unfortunately, current state-of-the-art video generation methods, primarily focusing on text-to-video generation, tend to produce video clips with minimal motions despite maintaining high fidelity. We argue that relying solely on text instructions is insufficient and suboptimal for video generation. In this paper, we introduce PixelDance, a novel approach based on diffusion models that incorporates image instructions for both the first and last frames in conjunction with text instructions for video generation. Comprehensive experimental results demonstrate that PixelDance trained with public data exhibits significantly better proficiency in synthesizing videos with complex scenes and intricate motions, setting a new standard for video generation.

Title: Behavior Optimized Image Generation. (arXiv:2311.10995v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10995
• Code URL: null
• Copy Paste: [[2311.10995]] Behavior Optimized Image Generation(http://arxiv.org/abs/2311.10995)
• Summary:

  The last few years have witnessed great success on image generation, which has crossed the acceptance thresholds of aesthetics, making it directly applicable to personal and commercial applications. However, images, especially in marketing and advertising applications, are often created as a means to an end as opposed to just aesthetic concerns. The goal can be increasing sales, getting more clicks, likes, or image sales (in the case of stock businesses). Therefore, the generated images need to perform well on these key performance indicators (KPIs), in addition to being aesthetically good. In this paper, we make the first endeavor to answer the question of "How can one infuse the knowledge of the end-goal within the image generation process itself to create not just better-looking images but also "better-performing'' images?''. We propose BoigLLM, an LLM that understands both image content and user behavior. BoigLLM knows how an image should look to get a certain required KPI. We show that BoigLLM outperforms 13x larger models such as GPT-3.5 and GPT-4 in this task, demonstrating that while these state-of-the-art models can understand images, they lack information on how these images perform in the real world. To generate actual pixels of behavior-conditioned images, we train a diffusion-based model (BoigSD) to align with a proposed BoigLLM-defined reward. We show the performance of the overall pipeline on two datasets covering two different behaviors: a stock dataset with the number of forward actions as the KPI and a dataset containing tweets with the total likes as the KPI, denoted as BoigBench. To advance research in the direction of utility-driven image generation and understanding, we release BoigBench, a benchmark dataset containing 168 million enterprise tweets with their media, brand account names, time of post, and total likes.

Title: Improving Adversarial Transferability by Stable Diffusion. (arXiv:2311.11017v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.11017
• Code URL: null
• Copy Paste: [[2311.11017]] Improving Adversarial Transferability by Stable Diffusion(http://arxiv.org/abs/2311.11017)
• Summary:

  Deep neural networks (DNNs) are susceptible to adversarial examples, which introduce imperceptible perturbations to benign samples, deceiving DNN predictions. While some attack methods excel in the white-box setting, they often struggle in the black-box scenario, particularly against models fortified with defense mechanisms. Various techniques have emerged to enhance the transferability of adversarial attacks for the black-box scenario. Among these, input transformation-based attacks have demonstrated their effectiveness. In this paper, we explore the potential of leveraging data generated by Stable Diffusion to boost adversarial transferability. This approach draws inspiration from recent research that harnessed synthetic data generated by Stable Diffusion to enhance model generalization. In particular, previous work has highlighted the correlation between the presence of both real and synthetic data and improved model generalization. Building upon this insight, we introduce a novel attack method called Stable Diffusion Attack Method (SDAM), which incorporates samples generated by Stable Diffusion to augment input images. Furthermore, we propose a fast variant of SDAM to reduce computational overhead while preserving high adversarial transferability. Our extensive experimental results demonstrate that our method outperforms state-of-the-art baselines by a substantial margin. Moreover, our approach is compatible with existing transfer-based attacks to further enhance adversarial transferability.

Title: A Study on Altering the Latent Space of Pretrained Text to Speech Models for Improved Expressiveness. (arXiv:2311.10804v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10804
• Code URL: null
• Copy Paste: [[2311.10804]] A Study on Altering the Latent Space of Pretrained Text to Speech Models for Improved Expressiveness(http://arxiv.org/abs/2311.10804)
• Summary:

  This report explores the challenge of enhancing expressiveness control in Text-to-Speech (TTS) models by augmenting a frozen pretrained model with a Diffusion Model that is conditioned on joint semantic audio/text embeddings. The paper identifies the challenges encountered when working with a VAE-based TTS model and evaluates different image-to-image methods for altering latent speech features. Our results offer valuable insights into the complexities of adding expressiveness control to TTS systems and open avenues for future research in this direction.

noise learning

data-free

transformer

Title: Multi-entity Video Transformers for Fine-Grained Video Representation Learning. (arXiv:2311.10873v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10873
• Code URL: https://github.com/facebookresearch/video_rep_learning
• Copy Paste: [[2311.10873]] Multi-entity Video Transformers for Fine-Grained Video Representation Learning(http://arxiv.org/abs/2311.10873)
• Summary:

  The area of temporally fine-grained video representation learning aims to generate frame-by-frame representations for temporally dense tasks. In this work, we advance the state-of-the-art for this area by re-examining the design of transformer architectures for video representation learning. A salient aspect of our self-supervised method is the improved integration of spatial information in the temporal pipeline by representing multiple entities per frame. Prior works use late fusion architectures that reduce frames to a single dimensional vector before any cross-frame information is shared, while our method represents each frame as a group of entities or tokens. Our Multi-entity Video Transformer (MV-Former) architecture achieves state-of-the-art results on multiple fine-grained video benchmarks. MV-Former leverages image features from self-supervised ViTs, and employs several strategies to maximize the utility of the extracted features while also avoiding the need to fine-tune the complex ViT backbone. This includes a Learnable Spatial Token Pooling strategy, which is used to identify and extract features for multiple salient regions per frame. Our experiments show that MV-Former not only outperforms previous self-supervised methods, but also surpasses some prior works that use additional supervision or training data. When combined with additional pre-training data from Kinetics-400, MV-Former achieves a further performance boost. The code for MV-Former is available at https://github.com/facebookresearch/video_rep_learning.

Title: Multiple View Geometry Transformers for 3D Human Pose Estimation. (arXiv:2311.10983v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10983
• Code URL: null
• Copy Paste: [[2311.10983]] Multiple View Geometry Transformers for 3D Human Pose Estimation(http://arxiv.org/abs/2311.10983)
• Summary:

  In this work, we aim to improve the 3D reasoning ability of Transformers in multi-view 3D human pose estimation. Recent works have focused on end-to-end learning-based transformer designs, which struggle to resolve geometric information accurately, particularly during occlusion. Instead, we propose a novel hybrid model, MVGFormer, which has a series of geometric and appearance modules organized in an iterative manner. The geometry modules are learning-free and handle all viewpoint-dependent 3D tasks geometrically which notably improves the model's generalization ability. The appearance modules are learnable and are dedicated to estimating 2D poses from image signals end-to-end which enables them to achieve accurate estimates even when occlusion occurs, leading to a model that is both accurate and generalizable to new cameras and geometries. We evaluate our approach for both in-domain and out-of-domain settings, where our model consistently outperforms state-of-the-art methods, and especially does so by a significant margin in the out-of-domain setting. We will release the code and models: https://github.com/XunshanMan/MVGFormer.

Title: Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention. (arXiv:2311.10988v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10988
• Code URL: null
• Copy Paste: [[2311.10988]] Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention(http://arxiv.org/abs/2311.10988)
• Summary:

  Scene Graph Generation (SGG) offers a structured representation critical in many computer vision applications. Traditional SGG approaches, however, are limited by a closed-set assumption, restricting their ability to recognize only predefined object and relation categories. To overcome this, we categorize SGG scenarios into four distinct settings based on the node and edge: Closed-set SGG, Open Vocabulary (object) Detection-based SGG (OvD-SGG), Open Vocabulary Relation-based SGG (OvR-SGG), and Open Vocabulary Detection + Relation-based SGG (OvD+R-SGG). While object-centric open vocabulary SGG has been studied recently, the more challenging problem of relation-involved open-vocabulary SGG remains relatively unexplored. To fill this gap, we propose a unified framework named OvSGTR towards fully open vocabulary SGG from a holistic view. The proposed framework is an end-toend transformer architecture, which learns a visual-concept alignment for both nodes and edges, enabling the model to recognize unseen categories. For the more challenging settings of relation-involved open vocabulary SGG, the proposed approach integrates relation-aware pre-training utilizing image-caption data and retains visual-concept alignment through knowledge distillation. Comprehensive experimental results on the Visual Genome benchmark demonstrate the effectiveness and superiority of the proposed framework.

Title: Learning Scene Context Without Images. (arXiv:2311.10998v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10998
• Code URL: null
• Copy Paste: [[2311.10998]] Learning Scene Context Without Images(http://arxiv.org/abs/2311.10998)
• Summary:

  Teaching machines of scene contextual knowledge would enable them to interact more effectively with the environment and to anticipate or predict objects that may not be immediately apparent in their perceptual field. In this paper, we introduce a novel transformer-based approach called $LMOD$ ( Label-based Missing Object Detection) to teach scene contextual knowledge to machines using an attention mechanism. A distinctive aspect of the proposed approach is its reliance solely on labels from image datasets to teach scene context, entirely eliminating the need for the actual image itself. We show how scene-wide relationships among different objects can be learned using a self-attention mechanism. We further show that the contextual knowledge gained from label based learning can enhance performance of other visual based object detection algorithm.

Title: Comparing Generalization in Learning with Limited Numbers of Exemplars: Transformer vs. RNN in Attractor Dynamics. (arXiv:2311.10763v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10763
• Code URL: null
• Copy Paste: [[2311.10763]] Comparing Generalization in Learning with Limited Numbers of Exemplars: Transformer vs(http://arxiv.org/abs/2311.10763)
• Summary:

  ChatGPT, a widely-recognized large language model (LLM), has recently gained substantial attention for its performance scaling, attributed to the billions of web-sourced natural language sentences used for training. Its underlying architecture, Transformer, has found applications across diverse fields, including video, audio signals, and robotic movement. %The crucial question this raises concerns the Transformer's generalization-in-learning (GIL) capacity. However, this raises a crucial question about Transformer's generalization in learning (GIL) capacity. Is ChatGPT's success chiefly due to the vast dataset used for training, or is there more to the story? To investigate this, we compared Transformer's GIL capabilities with those of a traditional Recurrent Neural Network (RNN) in tasks involving attractor dynamics learning. For performance evaluation, the Dynamic Time Warping (DTW) method has been employed. Our simulation results suggest that under conditions of limited data availability, Transformer's GIL abilities are markedly inferior to those of RNN.

Title: Automatic Restoration of Diacritics for Speech Data Sets. (arXiv:2311.10771v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10771
• Code URL: null
• Copy Paste: [[2311.10771]] Automatic Restoration of Diacritics for Speech Data Sets(http://arxiv.org/abs/2311.10771)
• Summary:

  Automatic text-based diacritic restoration models generally have high diacritic error rates when applied to speech transcripts as a result of domain and style shifts in spoken language. In this work, we explore the possibility of improving the performance of automatic diacritic restoration when applied to speech data by utilizing the parallel spoken utterances. In particular, we use the pre-trained Whisper ASR model fine-tuned on relatively small amounts of diacritized Arabic speech data to produce rough diacritized transcripts for the speech utterances, which we then use as an additional input for a transformer-based diacritic restoration model. The proposed model consistently improve diacritic restoration performance compared to an equivalent text-only model, with at least 5\% absolute reduction in diacritic error rate within the same domain and on two out-of-domain test sets. Our results underscore the inadequacy of current text-based diacritic restoration models for speech data sets and provide a new baseline for speech-based diacritic restoration.

Title: Partially Randomizing Transformer Weights for Dialogue Response Diversity. (arXiv:2311.10943v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10943
• Code URL: null
• Copy Paste: [[2311.10943]] Partially Randomizing Transformer Weights for Dialogue Response Diversity(http://arxiv.org/abs/2311.10943)
• Summary:

  Despite recent progress in generative open-domain dialogue, the issue of low response diversity persists. Prior works have addressed this issue via either novel objective functions, alternative learning approaches such as variational frameworks, or architectural extensions such as the Randomized Link (RL) Transformer. However, these approaches typically entail either additional difficulties during training/inference, or a significant increase in model size and complexity. Hence, we propose the \underline{Pa}rtially \underline{Ra}ndomized trans\underline{Former} (PaRaFormer), a simple extension of the transformer which involves freezing the weights of selected layers after random initialization. Experimental results reveal that the performance of the PaRaformer is comparable to that of the aforementioned approaches, despite not entailing any additional training difficulty or increase in model complexity.

generative

Title: Journey of Hallucination-minimized Generative AI Solutions for Financial Decision Makers. (arXiv:2311.10961v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10961
• Code URL: null
• Copy Paste: [[2311.10961]] Journey of Hallucination-minimized Generative AI Solutions for Financial Decision Makers(http://arxiv.org/abs/2311.10961)
• Summary:

  Generative AI has significantly reduced the entry barrier to the domain of AI owing to the ease of use and core capabilities of automation, translation, and intelligent actions in our day to day lives. Currently, Large language models (LLMs) that power such chatbots are being utilized primarily for their automation capabilities for software monitoring, report generation etc. and for specific personalized question answering capabilities, on a limited scope and scale. One major limitation of the currently evolving family of LLMs is 'hallucinations', wherein inaccurate responses are reported as factual. Hallucinations are primarily caused by biased training data, ambiguous prompts and inaccurate LLM parameters, and they majorly occur while combining mathematical facts with language-based context. Thus, monitoring and controlling for hallucinations becomes necessary when designing solutions that are meant for decision makers. In this work we present the three major stages in the journey of designing hallucination-minimized LLM-based solutions that are specialized for the decision makers of the financial domain, namely: prototyping, scaling and LLM evolution using human feedback. These three stages and the novel data to answer generation modules presented in this work are necessary to ensure that the Generative AI chatbots, autonomous reports and alerts are reliable and high-quality to aid key decision-making processes.

Title: Compact and Intuitive Airfoil Parameterization Method through Physics-aware Variational Autoencoder. (arXiv:2311.10921v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10921
• Code URL: null
• Copy Paste: [[2311.10921]] Compact and Intuitive Airfoil Parameterization Method through Physics-aware Variational Autoencoder(http://arxiv.org/abs/2311.10921)
• Summary:

  Airfoil shape optimization plays a critical role in the design of high-performance aircraft. However, the high-dimensional nature of airfoil representation causes the challenging problem known as the "curse of dimensionality". To overcome this problem, numerous airfoil parameterization methods have been developed, which can be broadly classified as polynomial-based and data-driven approaches. Each of these methods has desirable characteristics such as flexibility, parsimony, feasibility, and intuitiveness, but a single approach that encompasses all of these attributes has yet to be found. For example, polynomial-based methods struggle to balance parsimony and flexibility, while data-driven methods lack in feasibility and intuitiveness. In recent years, generative models, such as generative adversarial networks and variational autoencoders, have shown promising potential in airfoil parameterization. However, these models still face challenges related to intuitiveness due to their black-box nature. To address this issue, we developed a novel airfoil parameterization method using physics-aware variational autoencoder. The proposed method not only explicitly separates the generation of thickness and camber distributions to produce smooth and non-intersecting airfoils, thereby improving feasibility, but it also directly aligns its latent dimensions with geometric features of the airfoil, significantly enhancing intuitiveness. Finally, extensive comparative studies were performed to demonstrate the effectiveness of our approach.

Title: Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models. (arXiv:2311.11003v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.11003
• Code URL: null
• Copy Paste: [[2311.11003]] Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models(http://arxiv.org/abs/2311.11003)
• Summary:

  Score-based generative models (SGMs) is a recent class of deep generative models with state-of-the-art performance in many applications. In this paper, we establish convergence guarantees for a general class of SGMs in 2-Wasserstein distance, assuming accurate score estimates and smooth log-concave data distribution. We specialize our result to several concrete SGMs with specific choices of forward processes modelled by stochastic differential equations, and obtain an upper bound on the iteration complexity for each model, which demonstrates the impacts of different choices of the forward processes. We also provide a lower bound when the data distribution is Gaussian. Numerically, we experiment SGMs with different forward processes, some of which are newly proposed in this paper, for unconditional image generation on CIFAR-10. We find that the experimental results are in good agreement with our theoretical predictions on the iteration complexity, and the models with our newly proposed forward processes can outperform existing models.

large language model

Title: Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Values. (arXiv:2311.10766v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10766
• Code URL: https://github.com/valuecompass/valuecompass.github.io
• Copy Paste: [[2311.10766]] Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Values(http://arxiv.org/abs/2311.10766)
• Summary:

  The rapid advancement of Large Language Models (LLMs) has attracted much attention to value alignment for their responsible development. However, how to define values in this context remains a largely unexplored question. Existing work mainly follows the Helpful, Honest, Harmless principle and specifies values as risk criteria formulated in the AI community, e.g., fairness and privacy protection, suffering from poor clarity, adaptability and transparency. Inspired by basic values in humanity and social science across cultures, this work proposes a novel basic value alignment paradigm and introduces a value space spanned by basic value dimensions. All LLMs' behaviors can be mapped into the space by identifying the underlying values, possessing the potential to address the three challenges. To foster future research, we apply the representative Schwartz's Theory of Basic Values as an initialized example and construct FULCRA, a dataset consisting of 5k (LLM output, value vector) pairs. Our extensive analysis of FULCRA reveals the underlying relation between basic values and LLMs' behaviors, demonstrating that our approach not only covers existing mainstream risks but also anticipates possibly unidentified ones. Additionally, we present an initial implementation of the basic value evaluation and alignment, paving the way for future research in this line.

Title: MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning. (arXiv:2311.10774v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10774
• Code URL: https://github.com/fuxiaoliu/mmc
• Copy Paste: [[2311.10774]] MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning(http://arxiv.org/abs/2311.10774)
• Summary:

  With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chart image understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal Chart Instruction (MMC-Instruction) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we develop MultiModal Chart Assistant (MMCA), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the need for a comprehensive evaluation of LMM chart understanding, we also propose a MultiModal Chart Benchmark (MMC-Benchmark), a comprehensive human-annotated benchmark with 9 distinct tasks evaluating reasoning capabilities over charts. Extensive experiments on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the most recent GPT-4V model. Our work provides an instruction-tuning methodology and benchmark to advance multimodal understanding of charts.

Title: ToolTalk: Evaluating Tool-Usage in a Conversational Setting. (arXiv:2311.10775v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10775
• Code URL: null
• Copy Paste: [[2311.10775]] ToolTalk: Evaluating Tool-Usage in a Conversational Setting(http://arxiv.org/abs/2311.10775)
• Summary:

  Large language models (LLMs) have displayed massive improvements in reasoning and decision-making skills and can hold natural conversations with users. Many recent works seek to augment LLM-based assistants with external tools so they can access private or up-to-date information and carry out actions on behalf of users. To better measure the performance of these assistants, this paper introduces ToolTalk, a benchmark consisting of complex user intents requiring multi-step tool usage specified through dialogue. ToolTalk contains 28 tools grouped into 7 plugins, and includes a complete simulated implementation of each tool, allowing for fully automated evaluation of assistants that rely on execution feedback. ToolTalk also emphasizes tools that externally affect the world rather than only tools for referencing or searching information. We evaluate GPT-3.5 and GPT-4 on ToolTalk resulting in success rates of 26% and 50% respectively. Our analysis of the errors reveals three major categories and suggests some future directions for improvement. We release ToolTalk at https://github.com/microsoft/ToolTalk.

Title: Text Sanitization Beyond Specific Domains: Zero-Shot Redaction & Substitution with Large Language Models. (arXiv:2311.10785v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10785
• Code URL: null
• Copy Paste: [[2311.10785]] Text Sanitization Beyond Specific Domains: Zero-Shot Redaction & Substitution with Large Language Models(http://arxiv.org/abs/2311.10785)
• Summary:

  In the context of information systems, text sanitization techniques are used to identify and remove sensitive data to comply with security and regulatory requirements. Even though many methods for privacy preservation have been proposed, most of them are focused on the detection of entities from specific domains (e.g., credit card numbers, social security numbers), lacking generality and requiring customization for each desirable domain. Moreover, removing words is, in general, a drastic measure, as it can degrade text coherence and contextual information. Less severe measures include substituting a word for a safe alternative, yet it can be challenging to automatically find meaningful substitutions. We present a zero-shot text sanitization technique that detects and substitutes potentially sensitive information using Large Language Models. Our evaluation shows that our method excels at protecting privacy while maintaining text coherence and contextual information, preserving data utility for downstream tasks.

segmentation

Title: Zero-Shot Digital Rock Image Segmentation with a Fine-Tuned Segment Anything Model. (arXiv:2311.10865v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10865
• Code URL: null
• Copy Paste: [[2311.10865]] Zero-Shot Digital Rock Image Segmentation with a Fine-Tuned Segment Anything Model(http://arxiv.org/abs/2311.10865)
• Summary:

  Accurate image segmentation is crucial in reservoir modelling and material characterization, enhancing oil and gas extraction efficiency through detailed reservoir models. This precision offers insights into rock properties, advancing digital rock physics understanding. However, creating pixel-level annotations for complex CT and SEM rock images is challenging due to their size and low contrast, lengthening analysis time. This has spurred interest in advanced semi-supervised and unsupervised segmentation techniques in digital rock image analysis, promising more efficient, accurate, and less labour-intensive methods. Meta AI's Segment Anything Model (SAM) revolutionized image segmentation in 2023, offering interactive and automated segmentation with zero-shot capabilities, essential for digital rock physics with limited training data and complex image features. Despite its advanced features, SAM struggles with rock CT/SEM images due to their absence in its training set and the low-contrast nature of grayscale images. Our research fine-tunes SAM for rock CT/SEM image segmentation, optimizing parameters and handling large-scale images to improve accuracy. Experiments on rock CT and SEM images show that fine-tuning significantly enhances SAM's performance, enabling high-quality mask generation in digital rock image analysis. Our results demonstrate the feasibility and effectiveness of the fine-tuned SAM model (RockSAM) for rock images, offering segmentation without extensive training or complex labelling.

Title: Labeling Indoor Scenes with Fusion of Out-of-the-Box Perception Models. (arXiv:2311.10883v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10883
• Code URL: null
• Copy Paste: [[2311.10883]] Labeling Indoor Scenes with Fusion of Out-of-the-Box Perception Models(http://arxiv.org/abs/2311.10883)
• Summary:

  The image annotation stage is a critical and often the most time-consuming part required for training and evaluating object detection and semantic segmentation models. Deployment of the existing models in novel environments often requires detecting novel semantic classes not present in the training data. Furthermore, indoor scenes contain significant viewpoint variations, which need to be handled properly by trained perception models. We propose to leverage the recent advancements in state-of-the-art models for bottom-up segmentation (SAM), object detection (Detic), and semantic segmentation (MaskFormer), all trained on large-scale datasets. We aim to develop a cost-effective labeling approach to obtain pseudo-labels for semantic segmentation and object instance detection in indoor environments, with the ultimate goal of facilitating the training of lightweight models for various downstream tasks. We also propose a multi-view labeling fusion stage, which considers the setting where multiple views of the scenes are available and can be used to identify and rectify single-view inconsistencies. We demonstrate the effectiveness of the proposed approach on the Active Vision dataset and the ADE20K dataset. We evaluate the quality of our labeling process by comparing it with human annotations. Also, we demonstrate the effectiveness of the obtained labels in downstream tasks such as object goal navigation and part discovery. In the context of object goal navigation, we depict enhanced performance using this fusion approach compared to a zero-shot baseline that utilizes large monolithic vision-language pre-trained models.

Title: Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder. (arXiv:2311.10887v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10887
• Code URL: null
• Copy Paste: [[2311.10887]] Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder(http://arxiv.org/abs/2311.10887)
• Summary:

  In recent years, the field of 3D self-supervised learning has witnessed significant progress, resulting in the emergence of Multi-Modality Masked AutoEncoders (MAE) methods that leverage both 2D images and 3D point clouds for pre-training. However, a notable limitation of these approaches is that they do not fully utilize the multi-view attributes inherent in 3D point clouds, which is crucial for a deeper understanding of 3D structures. Building upon this insight, we introduce a novel approach employing a 3D to multi-view masked autoencoder to fully harness the multi-modal attributes of 3D point clouds. To be specific, our method uses the encoded tokens from 3D masked point clouds to generate original point clouds and multi-view depth images across various poses. This approach not only enriches the model's comprehension of geometric structures but also leverages the inherent multi-modal properties of point clouds. Our experiments illustrate the effectiveness of the proposed method for different tasks and under different settings. Remarkably, our method outperforms state-of-the-art counterparts by a large margin in a variety of downstream tasks, including 3D object classification, few-shot learning, part segmentation, and 3D object detection. Code will be available at: https://github.com/Zhimin-C/Multiview-MAE

Title: Synthetic Data Generation for Bridging Sim2Real Gap in a Production Environment. (arXiv:2311.11039v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.11039
• Code URL: null
• Copy Paste: [[2311.11039]] Synthetic Data Generation for Bridging Sim2Real Gap in a Production Environment(http://arxiv.org/abs/2311.11039)
• Summary:

  Synthetic data is being used lately for training deep neural networks in computer vision applications such as object detection, object segmentation and 6D object pose estimation. Domain randomization hereby plays an important role in reducing the simulation to reality gap. However, this generalization might not be effective in specialized domains like a production environment involving complex assemblies. Either the individual parts, trained with synthetic images, are integrated in much larger assemblies making them indistinguishable from their counterparts and result in false positives or are partially occluded just enough to give rise to false negatives. Domain knowledge is vital in these cases and if conceived effectively while generating synthetic data, can show a considerable improvement in bridging the simulation to reality gap. This paper focuses on synthetic data generation procedures for parts and assemblies used in a production environment. The basic procedures for synthetic data generation and their various combinations are evaluated and compared on images captured in a production environment, where results show up to 15% improvement using combinations of basic procedures. Reducing the simulation to reality gap in this way can aid to utilize the true potential of robot assisted production using artificial intelligence.