2024-04-10

Title: Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations

• Authors: Georgy Tyukin
• Subjects: cs.LG, cs.AI, cs.CL, cs.PF
• Abstract URL: https://arxiv.org/abs/2404.05741
• Pdf URL: https://arxiv.org/pdf/2404.05741
• Copy Paste: [[2404.05741]] Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations(https://arxiv.org/abs/2404.05741)
• Keywords: transformer, large language model
• Abstract: Large Language Models are growing in size, and we expect them to continue to do so, as larger models train quicker. However, this increase in size will severely impact inference costs. Therefore model compression is important, to retain the performance of larger models, but with a reduced cost of running them. In this thesis we explore the methods of model compression, and we empirically demonstrate that the simple method of skipping latter attention sublayers in Transformer LLMs is an effective method of model compression, as these layers prove to be redundant, whilst also being incredibly computationally expensive. We observed a 21% speed increase in one-token generation for Llama 2 7B, whilst surprisingly and unexpectedly improving performance over several common benchmarks.

Title: Exploiting CPU Clock Modulation for Covert Communication Channel

• Authors: Shariful Alam, Jidong Xiao, Nasir U. Eisty
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05823
• Pdf URL: https://arxiv.org/pdf/2404.05823
• Copy Paste: [[2404.05823]] Exploiting CPU Clock Modulation for Covert Communication Channel(https://arxiv.org/abs/2404.05823)
• Keywords: secure, security, attack
• Abstract: Covert channel attacks represent a significant threat to system security, leveraging shared resources to clandestinely transmit information from highly secure systems, thereby violating the system's security policies. These attacks exploit shared resources as communication channels, necessitating resource partitioning and isolation techniques as countermeasures. However, mitigating attacks exploiting modern processors' hardware features to leak information is challenging because successful attacks can conceal the channel's existence. In this paper, we unveil a novel covert channel exploiting the duty cycle modulation feature of modern x86 processors. Specifically, we illustrate how two collaborating processes, a sender and a receiver can manipulate this feature to transmit sensitive information surreptitiously. Our live system implementation demonstrates that this covert channel can achieve a data transfer rate of up to 55.24 bits per second.

Title: Privacy-Preserving Deep Learning Using Deformable Operators for Secure Task Learning

• Authors: Fabian Perez, Jhon Lopez, Henry Arguello
• Subjects: cs.CV, cs.CR, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.05828
• Pdf URL: https://arxiv.org/pdf/2404.05828
• Copy Paste: [[2404.05828]] Privacy-Preserving Deep Learning Using Deformable Operators for Secure Task Learning(https://arxiv.org/abs/2404.05828)
• Keywords: secure, privacy, protect
• Abstract: In the era of cloud computing and data-driven applications, it is crucial to protect sensitive information to maintain data privacy, ensuring truly reliable systems. As a result, preserving privacy in deep learning systems has become a critical concern. Existing methods for privacy preservation rely on image encryption or perceptual transformation approaches. However, they often suffer from reduced task performance and high computational costs. To address these challenges, we propose a novel Privacy-Preserving framework that uses a set of deformable operators for secure task learning. Our method involves shuffling pixels during the analog-to-digital conversion process to generate visually protected data. Those are then fed into a well-known network enhanced with deformable operators. Using our approach, users can achieve equivalent performance to original images without additional training using a secret key. Moreover, our method enables access control against unauthorized users. Experimental results demonstrate the efficacy of our approach, showcasing its potential in cloud-based scenarios and privacy-sensitive applications.

Title: SambaLingo: Teaching Large Language Models New Languages

• Authors: Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, Urmish Thakker
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05829
• Pdf URL: https://arxiv.org/pdf/2404.05829
• Copy Paste: [[2404.05829]] SambaLingo: Teaching Large Language Models New Languages(https://arxiv.org/abs/2404.05829)
• Keywords: large language model
• Abstract: Despite the widespread availability of LLMs, there remains a substantial gap in their capabilities and availability across diverse languages. One approach to address these issues has been to take an existing pre-trained LLM and continue to train it on new languages. While prior works have experimented with language adaptation, many questions around best practices and methodology have not been covered. In this paper, we present a comprehensive investigation into the adaptation of LLMs to new languages. Our study covers the key components in this process, including vocabulary extension, direct preference optimization and the data scarcity problem for human alignment in low-resource languages. We scale these experiments across 9 languages and 2 parameter scales (7B and 70B). We compare our models against Llama 2, Aya-101, XGLM, BLOOM and existing language experts, outperforming all prior published baselines. Additionally, all evaluation code and checkpoints are made public to facilitate future research.

Title: ÚFAL LatinPipe at EvaLatin 2024: Morphosyntactic Analysis of Latin

• Authors: Milan Straka, Jana Straková, Federica Gamba
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05839
• Pdf URL: https://arxiv.org/pdf/2404.05839
• Copy Paste: [[2404.05839]] ÚFAL LatinPipe at EvaLatin 2024: Morphosyntactic Analysis of Latin(https://arxiv.org/abs/2404.05839)
• Keywords: transformer
• Abstract: We present LatinPipe, the winning submission to the EvaLatin 2024 Dependency Parsing shared task. Our system consists of a fine-tuned concatenation of base and large pre-trained LMs, with a dot-product attention head for parsing and softmax classification heads for morphology to jointly learn both dependency parsing and morphological analysis. It is trained by sampling from seven publicly available Latin corpora, utilizing additional harmonization of annotations to achieve a more unified annotation style. Before fine-tuning, we train the system for a few initial epochs with frozen weights. We also add additional local relative contextualization by stacking the BiLSTM layers on top of the Transformer(s). Finally, we ensemble output probability distributions from seven randomly instantiated networks for the final submission. The code is available at https://github.com/ufal/evalatin2024-latinpipe.

Title: Softmax Attention with Constant Cost per Token

• Authors: Franz A. Heinsen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05843
• Pdf URL: https://arxiv.org/pdf/2404.05843
• Copy Paste: [[2404.05843]] Softmax Attention with Constant Cost per Token(https://arxiv.org/abs/2404.05843)
• Keywords: transformer
• Abstract: We propose a simple modification to the conventional attention mechanism applied by Transformers: Instead of quantifying pairwise query-key similarity with scaled dot-products, we quantify it with the logarithms of scaled dot-products of exponentials. Attention becomes expressible as a composition of log-sums of exponentials that is linearizable, with a latent space of constant size, enabling sequential application with constant time and space complexity per token. We implement our modification, verify that it works in practice, and conclude that it is a promising alternative to conventional attention.

Title: Towards Improved Semiconductor Defect Inspection for high-NA EUVL based on SEMI-SuperYOLO-NAS

• Authors: Ying-Lin Chen, Jacob Deforce, Vic De Ridder, Bappaditya Dey, Victor Blanco, Sandip Halder, Philippe Leray
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05862
• Pdf URL: https://arxiv.org/pdf/2404.05862
• Copy Paste: [[2404.05862]] Towards Improved Semiconductor Defect Inspection for high-NA EUVL based on SEMI-SuperYOLO-NAS(https://arxiv.org/abs/2404.05862)
• Keywords: robust
• Abstract: Due to potential pitch reduction, the semiconductor industry is adopting High-NA EUVL technology. However, its low depth of focus presents challenges for High Volume Manufacturing. To address this, suppliers are exploring thinner photoresists and new underlayers/hardmasks. These may suffer from poor SNR, complicating defect detection. Vision-based ML algorithms offer a promising solution for semiconductor defect inspection. However, developing a robust ML model across various image resolutions without explicit training remains a challenge for nano-scale defect inspection. This research's goal is to propose a scale-invariant ADCD framework capable to upscale images, addressing this issue. We propose an improvised ADCD framework as SEMI-SuperYOLO-NAS, which builds upon the baseline YOLO-NAS architecture. This framework integrates a SR assisted branch to aid in learning HR features by the defect detection backbone, particularly for detecting nano-scale defect instances from LR images. Additionally, the SR-assisted branch can recursively generate upscaled images from their corresponding downscaled counterparts, enabling defect detection inference across various image resolutions without requiring explicit training. Moreover, we investigate improved data augmentation strategy aimed at generating diverse and realistic training datasets to enhance model performance. We have evaluated our proposed approach using two original FAB datasets obtained from two distinct processes and captured using two different imaging tools. Finally, we demonstrate zero-shot inference for our model on a new, originating from a process condition distinct from the training dataset and possessing different Pitch characteristics. Experimental validation demonstrates that our proposed ADCD framework aids in increasing the throughput of imaging tools for defect inspection by reducing the required image pixel resolutions.

Title: Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

• Authors: Ruiqi Zhang, Licong Lin, Yu Bai, Song Mei
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.05868
• Pdf URL: https://arxiv.org/pdf/2404.05868
• Copy Paste: [[2404.05868]] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning(https://arxiv.org/abs/2404.05868)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the pre-trained model while preserving the model's utilities on other tasks. Several practical methods have recently been proposed for LLM unlearning, mostly based on gradient ascent (GA) on the loss of undesirable data. However, on certain unlearning tasks, these methods either fail to effectively unlearn the target data or suffer from catastrophic collapse -- a drastic degradation of the model's utilities. In this paper, we propose Negative Preference Optimization (NPO), a simple alignment-inspired method that could efficiently and effectively unlearn a target dataset. We theoretically show that the progression toward catastrophic collapse by minimizing the NPO loss is exponentially slower than GA. Through experiments on synthetic data and the benchmark TOFU dataset, we demonstrate that NPO-based methods achieve a better balance between unlearning the undesirable data and maintaining the model's utilities. We also observe that NPO-based methods generate more sensible outputs than GA-based methods, whose outputs are often gibberish. Remarkably, on TOFU, NPO-based methods are the first to achieve reasonable unlearning results in forgetting 50% (or more) of the training data, whereas existing methods already struggle with forgetting 10% of training data.

Title: CodecLM: Aligning Language Models with Tailored Synthetic Data

• Authors: Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T. Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05875
• Pdf URL: https://arxiv.org/pdf/2404.05875
• Copy Paste: [[2404.05875]] CodecLM: Aligning Language Models with Tailored Synthetic Data(https://arxiv.org/abs/2404.05875)
• Keywords: large language model
• Abstract: Instruction tuning has emerged as the key in aligning large language models (LLMs) with specific task instructions, thereby mitigating the discrepancy between the next-token prediction objective and users' actual goals. To reduce the labor and time cost to collect or annotate data by humans, researchers start to explore the use of LLMs to generate instruction-aligned synthetic data. Recent works focus on generating diverse instructions and applying LLM to increase instruction complexity, often neglecting downstream use cases. It remains unclear how to tailor high-quality data to elicit better instruction-following abilities in different target instruction distributions and LLMs. To this end, we introduce CodecLM, a general framework for adaptively generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. Drawing on the Encode-Decode principles, we use LLMs as codecs to guide the data generation process. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution, and then decode metadata to create tailored instructions. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples. Extensive experiments on four open-domain instruction following benchmarks validate the effectiveness of CodecLM over the current state-of-the-arts.

Title: Privacy and Security of Women's Reproductive Health Apps in a Changing Legal Landscape

• Authors: Shalini Saini, Nitesh Saxena
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2404.05876
• Pdf URL: https://arxiv.org/pdf/2404.05876
• Copy Paste: [[2404.05876]] Privacy and Security of Women's Reproductive Health Apps in a Changing Legal Landscape(https://arxiv.org/abs/2404.05876)
• Keywords: secure, security, privacy
• Abstract: FemTech, a rising trend in mobile apps, empowers women to digitally manage their health and family planning. However, privacy and security vulnerabilities in period-tracking and fertility-monitoring apps present significant risks, such as unintended pregnancies and legal consequences. Our approach involves manual observations of privacy policies and app permissions, along with dynamic and static analysis using multiple evaluation frameworks. Our research reveals that many of these apps gather personally identifiable information (PII) and sensitive healthcare data. Furthermore, our analysis identifies that 61% of the code vulnerabilities found in the apps are classified under the top-ten Open Web Application Security Project (OWASP) vulnerabilities. Our research emphasizes the significance of tackling the privacy and security vulnerabilities present in period-tracking and fertility-monitoring mobile apps. By highlighting these crucial risks, we aim to initiate a vital discussion and advocate for increased accountability and transparency of digital tools for women's health. We encourage the industry to prioritize user privacy and security, ultimately promoting a safer and more secure environment for women's health management.

Title: Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge

• Authors: Weikai Lu, Ziqian Zeng, Jianwei Wang, Zhengdong Lu, Zelin Chen, Huiping Zhuang, Cen Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05880
• Pdf URL: https://arxiv.org/pdf/2404.05880
• Copy Paste: [[2404.05880]] Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge(https://arxiv.org/abs/2404.05880)
• Keywords: defense, attack, large language model
• Abstract: Jailbreaking attacks can enable Large Language Models (LLMs) to bypass the safeguard and generate harmful content. Existing jailbreaking defense methods have failed to address the fundamental issue that harmful knowledge resides within the model, leading to potential jailbreak risks for LLMs. In this paper, we propose a novel defense method called Eraser, which mainly includes three goals: unlearning harmful knowledge, retaining general knowledge, and maintaining safety alignment. The intuition is that if an LLM forgets the specific knowledge required to answer a harmful question, it will no longer have the ability to answer harmful questions. The training of Erase does not actually require the model's own harmful knowledge, and it can benefit from unlearning general answers related to harmful queries, which means it does not need assistance from the red team. The experimental results show that Eraser can significantly reduce the jailbreaking success rate for various attacks without compromising the general capabilities of the model.

Title: WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents

• Authors: Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, Giovanni Campagna
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05902
• Pdf URL: https://arxiv.org/pdf/2404.05902
• Copy Paste: [[2404.05902]] WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents(https://arxiv.org/abs/2404.05902)
• Keywords: robust, generative, large language model
• Abstract: In the realm of web agent research, achieving both generalization and accuracy remains a challenging problem. Due to high variance in website structure, existing approaches often fail. Moreover, existing fine-tuning and in-context learning techniques fail to generalize across multiple websites. We introduce Wilbur, an approach that uses a differentiable ranking model and a novel instruction synthesis technique to optimally populate a black-box large language model's prompt with task demonstrations from previous runs. To maximize end-to-end success rates, we also propose an intelligent backtracking mechanism that learns and recovers from its mistakes. Finally, we show that our ranking model can be trained on data from a generative auto-curriculum which samples representative goals from an LLM, runs the agent, and automatically evaluates it, with no manual annotation. Wilbur achieves state-of-the-art results on the WebVoyager benchmark, beating text-only models by 8% overall, and up to 36% on certain websites. On the same benchmark, Wilbur is within 5% of a strong multi-modal model despite only receiving textual inputs, and further analysis reveals a substantial number of failures are due to engineering challenges of operating the web.

Title: Natural Learning

• Authors: Hadi Fanaee-T
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05903
• Pdf URL: https://arxiv.org/pdf/2404.05903
• Copy Paste: [[2404.05903]] Natural Learning(https://arxiv.org/abs/2404.05903)
• Keywords: interpretability, explainability
• Abstract: We introduce Natural Learning (NL), a novel algorithm that elevates the explainability and interpretability of machine learning to an extreme level. NL simplifies decisions into intuitive rules, like "We rejected your loan because your income, employment status, and age collectively resemble a rejected prototype more than an accepted prototype." When applied to real-life datasets, NL produces impressive results. For example, in a colon cancer dataset with 1545 patients and 10935 genes, NL achieves 98.1% accuracy, comparable to DNNs and RF, by analyzing just 3 genes of test samples against 2 discovered prototypes. Similarly, in the UCI's WDBC dataset, NL achieves 98.3% accuracy using only 7 features and 2 prototypes. Even on the MNIST dataset (0 vs. 1), NL achieves 99.5% accuracy with only 3 pixels from 2 prototype images. NL is inspired by prototype theory, an old concept in cognitive psychology suggesting that people learn single sparse prototypes to categorize objects. Leveraging this relaxed assumption, we redesign Support Vector Machines (SVM), replacing its mathematical formulation with a fully nearest-neighbor-based solution, and to address the curse of dimensionality, we utilize locality-sensitive hashing. Following theory's generalizability principle, we propose a recursive method to prune non-core features. As a result, NL efficiently discovers the sparsest prototypes in O(n^2pL) with high parallelization capacity in terms of n. Evaluation of NL with 17 benchmark datasets shows its significant outperformance compared to decision trees and logistic regression, two methods widely favored in healthcare for their interpretability. Moreover, NL achieves performance comparable to finetuned black-box models such as deep neural networks and random forests in 40% of cases, with only a 1-2% lower average accuracy. The code is available via this http URL

Title: The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models

• Authors: Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-Beltrachini, Max Ryabinin, Xuanli He, Pasquale Minervini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05904
• Pdf URL: https://arxiv.org/pdf/2404.05904
• Copy Paste: [[2404.05904]] The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models(https://arxiv.org/abs/2404.05904)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have transformed the Natural Language Processing (NLP) landscape with their remarkable ability to understand and generate human-like text. However, these models are prone to ``hallucinations'' -- outputs that do not align with factual reality or the input context. This paper introduces the Hallucinations Leaderboard, an open initiative to quantitatively measure and compare the tendency of each model to produce hallucinations. The leaderboard uses a comprehensive set of benchmarks focusing on different aspects of hallucinations, such as factuality and faithfulness, across various tasks, including question-answering, summarisation, and reading comprehension. Our analysis provides insights into the performance of different models, guiding researchers and practitioners in choosing the most reliable models for their applications.

Title: Interpretability in Symbolic Regression: a benchmark of Explanatory Methods using the Feynman data set

• Authors: Guilherme Seidyo Imai Aldeia, Fabricio Olivetti de Franca (Federal University of ABC)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05908
• Pdf URL: https://arxiv.org/pdf/2404.05908
• Copy Paste: [[2404.05908]] Interpretability in Symbolic Regression: a benchmark of Explanatory Methods using the Feynman data set(https://arxiv.org/abs/2404.05908)
• Keywords: robust, fair, interpretability
• Abstract: In some situations, the interpretability of the machine learning models plays a role as important as the model accuracy. Interpretability comes from the need to trust the prediction model, verify some of its properties, or even enforce them to improve fairness. Many model-agnostic explanatory methods exists to provide explanations for black-box models. In the regression task, the practitioner can use white-boxes or gray-boxes models to achieve more interpretable results, which is the case of symbolic regression. When using an explanatory method, and since interpretability lacks a rigorous definition, there is a need to evaluate and compare the quality and different explainers. This paper proposes a benchmark scheme to evaluate explanatory methods to explain regression models, mainly symbolic regression models. Experiments were performed using 100 physics equations with different interpretable and non-interpretable regression methods and popular explanation methods, evaluating the performance of the explainers performance with several explanation measures. In addition, we further analyzed four benchmarks from the GP community. The results have shown that Symbolic Regression models can be an interesting alternative to white-box and black-box models that is capable of returning accurate models with appropriate explanations. Regarding the explainers, we observed that Partial Effects and SHAP were the most robust explanation models, with Integrated Gradients being unstable only with tree-based models. This benchmark is publicly available for further experiments.

Title: Deep Reinforcement Learning for Personalized Diagnostic Decision Pathways Using Electronic Health Records: A Comparative Study on Anemia and Systemic Lupus Erythematosus

• Authors: Lillian Muyama, Antoine Neuraz, Adrien Coulet
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05913
• Pdf URL: https://arxiv.org/pdf/2404.05913
• Copy Paste: [[2404.05913]] Deep Reinforcement Learning for Personalized Diagnostic Decision Pathways Using Electronic Health Records: A Comparative Study on Anemia and Systemic Lupus Erythematosus(https://arxiv.org/abs/2404.05913)
• Keywords: robust
• Abstract: Background: Clinical diagnosis is typically reached by following a series of steps recommended by guidelines authored by colleges of experts. Accordingly, guidelines play a crucial role in rationalizing clinical decisions but suffer from limitations as they are built to cover the majority of the population and fail at covering patients with uncommon conditions. Moreover, their updates are long and expensive, making them unsuitable for emerging diseases and practices. Methods: Inspired by guidelines, we formulate the task of diagnosis as a sequential decision-making problem and study the use of Deep Reinforcement Learning (DRL) algorithms to learn the optimal sequence of actions to perform in order to obtain a correct diagnosis from Electronic Health Records (EHRs). We apply DRL on synthetic, but realistic EHRs and develop two clinical use cases: Anemia diagnosis, where the decision pathways follow the schema of a decision tree; and Systemic Lupus Erythematosus (SLE) diagnosis, which follows a weighted criteria score. We particularly evaluate the robustness of our approaches to noisy and missing data since these frequently occur in EHRs. Results: In both use cases, and in the presence of imperfect data, our best DRL algorithms exhibit competitive performance when compared to the traditional classifiers, with the added advantage that they enable the progressive generation of a pathway to the suggested diagnosis which can both guide and explain the decision-making process. Conclusion: DRL offers the opportunity to learn personalized decision pathways to diagnosis. We illustrate with our two use cases their advantages: they generate step-by-step pathways that are self-explanatory; and their correctness is competitive when compared to state-of-the-art approaches.

Title: Prompt-driven Universal Model for View-Agnostic Echocardiography Analysis

• Authors: Sekeun Kim, Hui Ren, Peng Guo, Abder-Rahman Ali, Patrick Zhang, Kyungsang Kim, Xiang Li, Quanzheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05916
• Pdf URL: https://arxiv.org/pdf/2404.05916
• Copy Paste: [[2404.05916]] Prompt-driven Universal Model for View-Agnostic Echocardiography Analysis(https://arxiv.org/abs/2404.05916)
• Keywords: segmentation
• Abstract: Echocardiography segmentation for cardiac analysis is time-consuming and resource-intensive due to the variability in image quality and the necessity to process scans from various standard views. While current automated segmentation methods in echocardiography show promising performance, they are trained on specific scan views to analyze corresponding data. However, this solution has a limitation as the number of required models increases with the number of standard views. To address this, in this paper, we present a prompt-driven universal method for view-agnostic echocardiography analysis. Considering the domain shift between standard views, we first introduce a method called prompt matching, aimed at learning prompts specific to different views by matching prompts and querying input embeddings using a pre-trained vision model. Then, we utilized a pre-trained medical language model to align textual information with pixel data for accurate segmentation. Extensive experiments on three standard views showed that our approach significantly outperforms the state-of-the-art universal methods and achieves comparable or even better performances over the segmentation model trained and tested on same views.

Title: VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

• Authors: Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05955
• Pdf URL: https://arxiv.org/pdf/2404.05955
• Copy Paste: [[2404.05955]] VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?(https://arxiv.org/abs/2404.05955)
• Keywords: large language model
• Abstract: Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce \bench{}, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. \bench{} consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites, covering 87 sub-domains. We evaluate 14 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on \bench{}, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe \bench{} will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.

Title: EasyTrack: Efficient and Compact One-stream 3D Point Clouds Tracker

• Authors: Baojie Fan, Wuyang Zhou, Kai Wang, Shijun Zhou, Fengyu Xu, Jiandong Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05960
• Pdf URL: https://arxiv.org/pdf/2404.05960
• Copy Paste: [[2404.05960]] EasyTrack: Efficient and Compact One-stream 3D Point Clouds Tracker(https://arxiv.org/abs/2404.05960)
• Keywords: transformer
• Abstract: Most of 3D single object trackers (SOT) in point clouds follow the two-stream multi-stage 3D Siamese or motion tracking paradigms, which process the template and search area point clouds with two parallel branches, built on supervised point cloud backbones. In this work, beyond typical 3D Siamese or motion tracking, we propose a neat and compact one-stream transformer 3D SOT paradigm from the novel perspective, termed as \textbf{EasyTrack}, which consists of three special designs: 1) A 3D point clouds tracking feature pre-training module is developed to exploit the masked autoencoding for learning 3D point clouds tracking representations. 2) A unified 3D tracking feature learning and fusion network is proposed to simultaneously learns target-aware 3D features, and extensively captures mutual correlation through the flexible self-attention mechanism. 3) A target location network in the dense bird's eye view (BEV) feature space is constructed for target classification and regression. Moreover, we develop an enhanced version named EasyTrack++, which designs the center points interaction (CPI) strategy to reduce the ambiguous targets caused by the noise point cloud background information. The proposed EasyTrack and EasyTrack++ set a new state-of-the-art performance ($\textbf{18\%}$, $\textbf{40\%}$ and $\textbf{3\%}$ success gains) in KITTI, NuScenes, and Waymo while runing at \textbf{52.6fps} with few parameters (\textbf{1.3M}). The code will be available at https://github.com/KnightApple427/Easytrack.

Title: LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

• Authors: Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, Siva Reddy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05961
• Pdf URL: https://arxiv.org/pdf/2404.05961
• Copy Paste: [[2404.05961]] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders(https://arxiv.org/abs/2404.05961)
• Keywords: large language model
• Abstract: Large decoder-only language models (LLMs) are the state-of-the-art models on most of today's NLP tasks and benchmarks. Yet, the community is only slowly adopting these models for text embedding tasks, which require rich contextualized representations. In this work, we introduce LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3 popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed models on English word- and sequence-level tasks. We outperform encoder-only models by a large margin on word-level tasks and reach a new unsupervised state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB). Moreover, when combining LLM2Vec with supervised contrastive learning, we achieve state-of-the-art performance on MTEB among models that train only on publicly available data. Our strong empirical results and extensive analysis demonstrate that LLMs can be effectively transformed into universal text encoders in a parameter-efficient manner without the need for expensive adaptation or synthetic GPT-4 generated data.

Title: Deep Learning-Based Out-of-distribution Source Code Data Identification: How Far We Have Gone?

• Authors: Van Nguyen, Xingliang Yuan, Tingmin Wu, Surya Nepal, Marthie Grobler, Carsten Rudolph
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05964
• Pdf URL: https://arxiv.org/pdf/2404.05964
• Copy Paste: [[2404.05964]] Deep Learning-Based Out-of-distribution Source Code Data Identification: How Far We Have Gone?(https://arxiv.org/abs/2404.05964)
• Keywords: security
• Abstract: Software vulnerabilities (SVs) have become a common, serious, and crucial concern to safety-critical security systems. That leads to significant progress in the use of AI-based methods for software vulnerability detection (SVD). In practice, although AI-based methods have been achieving promising performances in SVD and other domain applications (e.g., computer vision), they are well-known to fail in detecting the ground-truth label of input data (referred to as out-of-distribution, OOD, data) lying far away from the training data distribution (i.e., in-distribution, ID). This drawback leads to serious issues where the models fail to indicate when they are likely mistaken. To address this problem, OOD detectors (i.e., determining whether an input is ID or OOD) have been applied before feeding the input data to the downstream AI-based modules. While OOD detection has been widely designed for computer vision and medical diagnosis applications, automated AI-based techniques for OOD source code data detection have not yet been well-studied and explored. To this end, in this paper, we propose an innovative deep learning-based approach addressing the OOD source code data identification problem. Our method is derived from an information-theoretic perspective with the use of innovative cluster-contrastive learning to effectively learn and leverage source code characteristics, enhancing data representation learning for solving the problem. The rigorous and comprehensive experiments on real-world source code datasets show the effectiveness and advancement of our approach compared to state-of-the-art baselines by a wide margin. In short, on average, our method achieves a significantly higher performance from around 15.27%, 7.39%, and 4.93% on the FPR, AUROC, and AUPR measures, respectively, in comparison with the baselines.

Title: Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation

• Authors: Alireza Salemi, Surya Kallumadi, Hamed Zamani
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.05970
• Pdf URL: https://arxiv.org/pdf/2404.05970
• Copy Paste: [[2404.05970]] Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation(https://arxiv.org/abs/2404.05970)
• Keywords: large language model
• Abstract: This paper studies retrieval-augmented approaches for personalizing large language models (LLMs), which potentially have a substantial impact on various applications and domains. We propose the first attempt to optimize the retrieval models that deliver a limited number of personal documents to large language models for the purpose of personalized generation. We develop two optimization algorithms that solicit feedback from the downstream personalized generation tasks for retrieval optimization--one based on reinforcement learning whose reward function is defined using any arbitrary metric for personalized generation and another based on knowledge distillation from the downstream LLM to the retrieval model. This paper also introduces a pre- and post-generation retriever selection model that decides what retriever to choose for each LLM input. Extensive experiments on diverse tasks from the language model personalization (LaMP) benchmark reveal statistically significant improvements in six out of seven datasets.

Title: Does Transformer Interpretability Transfer to RNNs?

• Authors: Gonçalo Paulo, Thomas Marshall, Nora Belrose
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05971
• Pdf URL: https://arxiv.org/pdf/2404.05971
• Copy Paste: [[2404.05971]] Does Transformer Interpretability Transfer to RNNs?(https://arxiv.org/abs/2404.05971)
• Keywords: interpretability, transformer
• Abstract: Recent advances in recurrent neural network architectures, such as Mamba and RWKV, have enabled RNNs to match or exceed the performance of equal-size transformers in terms of language modeling perplexity and downstream evaluations, suggesting that future systems may be built on completely new architectures. In this paper, we examine if selected interpretability methods originally designed for transformer language models will transfer to these up-and-coming recurrent architectures. Specifically, we focus on steering model outputs via contrastive activation addition, on eliciting latent predictions via the tuned lens, and eliciting latent knowledge from models fine-tuned to produce false outputs under certain conditions. Our results show that most of these techniques are effective when applied to RNNs, and we show that it is possible to improve some of them by taking advantage of RNNs' compressed state.

Title: StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion

• Authors: Ming Tao, Bing-Kun Bao, Hao Tang, Yaowei Wang, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05979
• Pdf URL: https://arxiv.org/pdf/2404.05979
• Copy Paste: [[2404.05979]] StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion(https://arxiv.org/abs/2404.05979)
• Keywords: generative
• Abstract: Story visualization aims to generate a series of realistic and coherent images based on a storyline. Current models adopt a frame-by-frame architecture by transforming the pre-trained text-to-image model into an auto-regressive manner. Although these models have shown notable progress, there are still three flaws. 1) The unidirectional generation of auto-regressive manner restricts the usability in many scenarios. 2) The additional introduced story history encoders bring an extremely high computational cost. 3) The story visualization and continuation models are trained and inferred independently, which is not user-friendly. To these ends, we propose a bidirectional, unified, and efficient framework, namely StoryImager. The StoryImager enhances the storyboard generative ability inherited from the pre-trained text-to-image model for a bidirectional generation. Specifically, we introduce a Target Frame Masking Strategy to extend and unify different story image generation tasks. Furthermore, we propose a Frame-Story Cross Attention Module that decomposes the cross attention for local fidelity and global coherence. Moreover, we design a Contextual Feature Extractor to extract contextual information from the whole storyline. The extensive experimental results demonstrate the excellent performance of our StoryImager. The code is available at https://github.com/tobran/StoryImager.

Title: Tackling Structural Hallucination in Image Translation with Local Diffusion

• Authors: Seunghoi Kim, Chen Jin, Tom Diethe, Matteo Figini, Henry F. J. Tregidgo, Asher Mullokandov, Philip Teare, Daniel C. Alexander
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05980
• Pdf URL: https://arxiv.org/pdf/2404.05980
• Copy Paste: [[2404.05980]] Tackling Structural Hallucination in Image Translation with Local Diffusion(https://arxiv.org/abs/2404.05980)
• Keywords: diffusion
• Abstract: Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing ``image hallucination'' and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a ``branching'' module generates locally both within and outside OOD regions, and a ``fusion'' module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models.

Title: Boosting Digital Safeguards: Blending Cryptography and Steganography

• Authors: Anamitra Maiti, Subham Laha, Rishav Upadhaya, Soumyajit Biswas, Vikas Choudhary, Biplab Kar, Nikhil Kumar, Jaydip Sen
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05985
• Pdf URL: https://arxiv.org/pdf/2404.05985
• Copy Paste: [[2404.05985]] Boosting Digital Safeguards: Blending Cryptography and Steganography(https://arxiv.org/abs/2404.05985)
• Keywords: secure, security, privacy, protect, generative
• Abstract: In today's digital age, the internet is essential for communication and the sharing of information, creating a critical need for sophisticated data security measures to prevent unauthorized access and exploitation. Cryptography encrypts messages into a cipher text that is incomprehensible to unauthorized readers, thus safeguarding data during its transmission. Steganography, on the other hand, originates from the Greek term for "covered writing" and involves the art of hiding data within another medium, thereby facilitating covert communication by making the message invisible. This proposed approach takes advantage of the latest advancements in Artificial Intelligence (AI) and Deep Learning (DL), especially through the application of Generative Adversarial Networks (GANs), to improve upon traditional steganographic methods. By embedding encrypted data within another medium, our method ensures that the communication remains hidden from prying eyes. The application of GANs enables a smart, secure system that utilizes the inherent sensitivity of neural networks to slight alterations in data, enhancing the protection against detection. By merging the encryption techniques of cryptography with the hiding capabilities of steganography, and augmenting these with the strengths of AI, we introduce a comprehensive security system designed to maintain both the privacy and integrity of information. This system is crafted not just to prevent unauthorized access or modification of data, but also to keep the existence of the data hidden. This fusion of technologies tackles the core challenges of data security in the current era of open digital communication, presenting an advanced solution with the potential to transform the landscape of information security.

Title: Event-enhanced Retrieval in Real-time Search

• Authors: Yanan Zhang, Xiaoling Bai, Tianhua Zhou
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.05989
• Pdf URL: https://arxiv.org/pdf/2404.05989
• Copy Paste: [[2404.05989]] Event-enhanced Retrieval in Real-time Search(https://arxiv.org/abs/2404.05989)
• Keywords: extraction, generative
• Abstract: The embedding-based retrieval (EBR) approach is widely used in mainstream search engine retrieval systems and is crucial in recent retrieval-augmented methods for eliminating LLM illusions. However, existing EBR models often face the "semantic drift" problem and insufficient focus on key information, leading to a low adoption rate of retrieval results in subsequent steps. This issue is especially noticeable in real-time search scenarios, where the various expressions of popular events on the Internet make real-time retrieval heavily reliant on crucial event information. To tackle this problem, this paper proposes a novel approach called EER, which enhances real-time retrieval performance by improving the dual-encoder model of traditional EBR. We incorporate contrastive learning to accompany pairwise learning for encoder optimization. Furthermore, to strengthen the focus on critical event information in events, we include a decoder module after the document encoder, introduce a generative event triplet extraction scheme based on prompt-tuning, and correlate the events with query encoder optimization through comparative learning. This decoder module can be removed during inference. Extensive experiments demonstrate that EER can significantly improve the real-time search retrieval performance. We believe that this approach will provide new perspectives in the field of information retrieval. The codes and dataset are available at https://github.com/open-event-hub/Event-enhanced_Retrieval .

Title: AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts

• Authors: Shaona Ghosh, Prasoon Varshney, Erick Galinkin, Christopher Parisien
• Subjects: cs.LG, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.05993
• Pdf URL: https://arxiv.org/pdf/2404.05993
• Copy Paste: [[2404.05993]] AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts(https://arxiv.org/abs/2404.05993)
• Keywords: attack, robust, generative, large language model
• Abstract: As Large Language Models (LLMs) and generative AI become more widespread, the content safety risks associated with their use also increase. We find a notable deficiency in high-quality content safety datasets and benchmarks that comprehensively cover a wide range of critical safety areas. To address this, we define a broad content safety risk taxonomy, comprising 13 critical risk and 9 sparse risk categories. Additionally, we curate AEGISSAFETYDATASET, a new dataset of approximately 26, 000 human-LLM interaction instances, complete with human annotations adhering to the taxonomy. We plan to release this dataset to the community to further research and to help benchmark LLM models for safety. To demonstrate the effectiveness of the dataset, we instruction-tune multiple LLM-based safety models. We show that our models (named AEGISSAFETYEXPERTS), not only surpass or perform competitively with the state-of-the-art LLM-based safety models and general purpose LLMs, but also exhibit robustness across multiple jail-break attack categories. We also show how using AEGISSAFETYDATASET during the LLM alignment phase does not negatively impact the performance of the aligned models on MT Bench scores. Furthermore, we propose AEGIS, a novel application of a no-regret online adaptation framework with strong theoretical guarantees, to perform content moderation with an ensemble of LLM content safety experts in deployment

Title: Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis

• Authors: Junlin Hou, Jilan Xu, Hao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05997
• Pdf URL: https://arxiv.org/pdf/2404.05997
• Copy Paste: [[2404.05997]] Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis(https://arxiv.org/abs/2404.05997)
• Keywords: interpretability
• Abstract: The black-box nature of deep learning models has raised concerns about their interpretability for successful deployment in real-world clinical applications. To address the concerns, eXplainable Artificial Intelligence (XAI) aims to provide clear and understandable explanations of the decision-making process. In the medical domain, concepts such as attributes of lesions or abnormalities serve as key evidence for deriving diagnostic results. However, existing concept-based models mainly depend on concepts that appear independently and require fine-grained concept annotations such as bounding boxes. A medical image usually contains multiple concepts and the fine-grained concept annotations are difficult to acquire. In this paper, we propose a novel Concept-Attention Whitening (CAW) framework for interpretable skin lesion diagnosis. CAW is comprised of a disease diagnosis branch and a concept alignment branch. In the former branch, we train the CNN with a CAW layer inserted to perform skin lesion diagnosis. The CAW layer decorrelates features and aligns image features to conceptual meanings via an orthogonal matrix. In the latter branch, we calculate the orthogonal matrix under the guidance of the concept attention mask. We particularly introduce a weakly-supervised concept mask generator that only leverages coarse concept labels for filtering local regions that are relevant to certain concepts, improving the optimization of the orthogonal matrix. Extensive experiments on two public skin lesion diagnosis datasets demonstrated that CAW not only enhanced interpretability but also maintained a state-of-the-art diagnostic performance.

Title: Privacy Preserving Prompt Engineering: A Survey

• Authors: Kennedy Edemacu, Xintao Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06001
• Pdf URL: https://arxiv.org/pdf/2404.06001
• Copy Paste: [[2404.06001]] Privacy Preserving Prompt Engineering: A Survey(https://arxiv.org/abs/2404.06001)
• Keywords: privacy, protect, large language model
• Abstract: Pre-trained language models (PLMs) have demonstrated significant proficiency in solving a wide range of general natural language processing (NLP) tasks. Researchers have observed a direct correlation between the performance of these models and their sizes. As a result, the sizes of these models have notably expanded in recent years, persuading researchers to adopt the term large language models (LLMs) to characterize the larger-sized PLMs. The increased size is accompanied by a distinct capability known as in-context learning (ICL), which represents a specialized form of prompting. This enables the utilization of LLMs for specific downstream tasks by presenting them with demonstration examples while keeping the model parameters frozen. Although interesting, privacy concerns have become a major obstacle in its widespread usage. Multiple studies have examined the privacy risks linked to ICL and prompting in general, and have devised techniques to alleviate these risks. Thus, there is a necessity to organize these mitigation techniques for the benefit of the community. This survey provides a systematic overview of the privacy protection methods employed during ICL and prompting in general. We review, analyze, and compare different methods under this paradigm. Furthermore, we provide a summary of the resources accessible for the development of these frameworks. Finally, we discuss the limitations of these frameworks and offer a detailed examination of the promising areas that necessitate further exploration.

Title: FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models

• Authors: Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, Shikun Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06003
• Pdf URL: https://arxiv.org/pdf/2404.06003
• Copy Paste: [[2404.06003]] FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models(https://arxiv.org/abs/2404.06003)
• Keywords: fair, large language model
• Abstract: The rapid development of large language model (LLM) evaluation methodologies and datasets has led to a profound challenge: integrating state-of-the-art evaluation techniques cost-effectively while ensuring reliability, reproducibility, and efficiency. Currently, there is a notable absence of a unified and adaptable framework that seamlessly integrates various evaluation approaches. Moreover, the reliability of evaluation findings is often questionable due to potential data contamination, with the evaluation efficiency commonly overlooked when facing the substantial costs associated with LLM inference. In response to these challenges, we introduce FreeEval, a modular and scalable framework crafted to enable trustworthy and efficient automatic evaluations of LLMs. Firstly, FreeEval's unified abstractions simplify the integration and improve the transparency of diverse evaluation methodologies, encompassing dynamic evaluation that demand sophisticated LLM interactions. Secondly, the framework integrates meta-evaluation techniques like human evaluation and data contamination detection, which, along with dynamic evaluation modules in the platform, enhance the fairness of the evaluation outcomes. Lastly, FreeEval is designed with a high-performance infrastructure, including distributed computation and caching strategies, enabling extensive evaluations across multi-node, multi-GPU clusters for open-source and proprietary LLMs.

Title: Diffusion-Based Point Cloud Super-Resolution for mmWave Radar Data

• Authors: Kai Luan, Chenghao Shi, Neng Wang, Yuwei Cheng, Huimin Lu, Xieyuanli Chen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06012
• Pdf URL: https://arxiv.org/pdf/2404.06012
• Copy Paste: [[2404.06012]] Diffusion-Based Point Cloud Super-Resolution for mmWave Radar Data(https://arxiv.org/abs/2404.06012)
• Keywords: diffusion
• Abstract: The millimeter-wave radar sensor maintains stable performance under adverse environmental conditions, making it a promising solution for all-weather perception tasks, such as outdoor mobile robotics. However, the radar point clouds are relatively sparse and contain massive ghost points, which greatly limits the development of mmWave radar technology. In this paper, we propose a novel point cloud super-resolution approach for 3D mmWave radar data, named Radar-diffusion. Our approach employs the diffusion model defined by mean-reverting stochastic differential equations(SDE). Using our proposed new objective function with supervision from corresponding LiDAR point clouds, our approach efficiently handles radar ghost points and enhances the sparse mmWave radar point clouds to dense LiDAR-like point clouds. We evaluate our approach on two different datasets, and the experimental results show that our method outperforms the state-of-the-art baseline methods in 3D radar super-resolution tasks. Furthermore, we demonstrate that our enhanced radar point cloud is capable of downstream radar point-based registration tasks.

Title: Band-Attention Modulated RetNet for Face Forgery Detection

• Authors: Zhida Zhang, Jie Cao, Wenkui Yang, Qihang Fan, Kai Zhou, Ran He
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.06022
• Pdf URL: https://arxiv.org/pdf/2404.06022
• Copy Paste: [[2404.06022]] Band-Attention Modulated RetNet for Face Forgery Detection(https://arxiv.org/abs/2404.06022)
• Keywords: transformer
• Abstract: The transformer networks are extensively utilized in face forgery detection due to their scalability across large datasets.Despite their success, transformers face challenges in balancing the capture of global context, which is crucial for unveiling forgery clues, with computational complexity.To mitigate this issue, we introduce Band-Attention modulated RetNet (BAR-Net), a lightweight network designed to efficiently process extensive visual contexts while avoiding catastrophic forgetting.Our approach empowers the target token to perceive global information by assigning differential attention levels to tokens at varying distances. We implement self-attention along both spatial axes, thereby maintaining spatial priors and easing the computational burden.Moreover, we present the adaptive frequency Band-Attention Modulation mechanism, which treats the entire Discrete Cosine Transform spectrogram as a series of frequency bands with learnable weights.Together, BAR-Net achieves favorable performance on several face forgery datasets, outperforming current state-of-the-art methods.

Title: Greedy-DiM: Greedy Algorithms for Unreasonably Effective Face Morphs

• Authors: Zander W. Blasingame, Chen Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06025
• Pdf URL: https://arxiv.org/pdf/2404.06025
• Copy Paste: [[2404.06025]] Greedy-DiM: Greedy Algorithms for Unreasonably Effective Face Morphs(https://arxiv.org/abs/2404.06025)
• Keywords: attack, biometric, diffusion, generative
• Abstract: Morphing attacks are an emerging threat to state-of-the-art Face Recognition (FR) systems, which aim to create a single image that contains the biometric information of multiple identities. Diffusion Morphs (DiM) are a recently proposed morphing attack that has achieved state-of-the-art performance for representation-based morphing attacks. However, none of the existing research on DiMs have leveraged the iterative nature of DiMs and left the DiM model as a black box, treating it no differently than one would a Generative Adversarial Network (GAN) or Varational AutoEncoder (VAE). We propose a greedy strategy on the iterative sampling process of DiM models which searches for an optimal step guided by an identity-based heuristic function. We compare our proposed algorithm against ten other state-of-the-art morphing algorithms using the open-source SYN-MAD 2022 competition dataset. We find that our proposed algorithm is unreasonably effective, fooling all of the tested FR systems with an MMPMR of 100%, outperforming all other morphing algorithms compared.

Title: Improving Facial Landmark Detection Accuracy and Efficiency with Knowledge Distillation

• Authors: Zong-Wei Hong, Yu-Chen Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06029
• Pdf URL: https://arxiv.org/pdf/2404.06029
• Copy Paste: [[2404.06029]] Improving Facial Landmark Detection Accuracy and Efficiency with Knowledge Distillation(https://arxiv.org/abs/2404.06029)
• Keywords: robust, segmentation
• Abstract: The domain of computer vision has experienced significant advancements in facial-landmark detection, becoming increasingly essential across various applications such as augmented reality, facial recognition, and emotion analysis. Unlike object detection or semantic segmentation, which focus on identifying objects and outlining boundaries, faciallandmark detection aims to precisely locate and track critical facial features. However, deploying deep learning-based facial-landmark detection models on embedded systems with limited computational resources poses challenges due to the complexity of facial features, especially in dynamic settings. Additionally, ensuring robustness across diverse ethnicities and expressions presents further obstacles. Existing datasets often lack comprehensive representation of facial nuances, particularly within populations like those in Taiwan. This paper introduces a novel approach to address these challenges through the development of a knowledge distillation method. By transferring knowledge from larger models to smaller ones, we aim to create lightweight yet powerful deep learning models tailored specifically for facial-landmark detection tasks. Our goal is to design models capable of accurately locating facial landmarks under varying conditions, including diverse expressions, orientations, and lighting environments. The ultimate objective is to achieve high accuracy and real-time performance suitable for deployment on embedded systems. This method was successfully implemented and achieved a top 6th place finish out of 165 participants in the IEEE ICME 2024 PAIR competition.

Title: Little Strokes Fell Great Oaks: Boosting the Hierarchical Features for Multi-exposure Image Fusion

• Authors: Pan Mu, Zhiying Du, Jinyuan Liu, Cong Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06033
• Pdf URL: https://arxiv.org/pdf/2404.06033
• Copy Paste: [[2404.06033]] Little Strokes Fell Great Oaks: Boosting the Hierarchical Features for Multi-exposure Image Fusion(https://arxiv.org/abs/2404.06033)
• Keywords: transformer
• Abstract: In recent years, deep learning networks have made remarkable strides in the domain of multi-exposure image fusion. Nonetheless, prevailing approaches often involve directly feeding over-exposed and under-exposed images into the network, which leads to the under-utilization of inherent information present in the source images. Additionally, unsupervised techniques predominantly employ rudimentary weighted summation for color channel processing, culminating in an overall desaturated final image tone. To partially mitigate these issues, this study proposes a gamma correction module specifically designed to fully leverage latent information embedded within source images. Furthermore, a modified transformer block, embracing with self-attention mechanisms, is introduced to optimize the fusion process. Ultimately, a novel color enhancement algorithm is presented to augment image saturation while preserving intricate details. The source code is available at this https://github.com/ZhiyingDu/BHFMEF url.

Title: Incremental Joint Learning of Depth, Pose and Implicit Scene Representation on Monocular Camera in Large-scale Scenes

• Authors: Tianchen Deng, Nailin Wang, Chongdi Wang, Shenghai Yuan, Jingchuan Wang, Danwei Wang, Weidong Chen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06050
• Pdf URL: https://arxiv.org/pdf/2404.06050
• Copy Paste: [[2404.06050]] Incremental Joint Learning of Depth, Pose and Implicit Scene Representation on Monocular Camera in Large-scale Scenes(https://arxiv.org/abs/2404.06050)
• Keywords: robust, transformer
• Abstract: Dense scene reconstruction for photo-realistic view synthesis has various applications, such as VR/AR, autonomous vehicles. However, most existing methods have difficulties in large-scale scenes due to three core challenges: \textit{(a) inaccurate depth input.} Accurate depth input is impossible to get in real-world large-scale scenes. \textit{(b) inaccurate pose estimation.} Most existing approaches rely on accurate pre-estimated camera poses. \textit{(c) insufficient scene representation capability.} A single global radiance field lacks the capacity to effectively scale to large-scale scenes. To this end, we propose an incremental joint learning framework, which can achieve accurate depth, pose estimation, and large-scale scene reconstruction. A vision transformer-based network is adopted as the backbone to enhance performance in scale information estimation. For pose estimation, a feature-metric bundle adjustment (FBA) method is designed for accurate and robust camera tracking in large-scale scenes. In terms of implicit scene representation, we propose an incremental scene representation method to construct the entire large-scale scene as multiple local radiance fields to enhance the scalability of 3D scene representation. Extended experiments have been conducted to demonstrate the effectiveness and accuracy of our method in depth estimation, pose estimation, and large-scale scene reconstruction.

Title: All in One: An Empirical Study of GPT for Few-Shot Aspect-Based Sentiment Anlaysis

• Authors: Baoxing Jiang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06063
• Pdf URL: https://arxiv.org/pdf/2404.06063
• Copy Paste: [[2404.06063]] All in One: An Empirical Study of GPT for Few-Shot Aspect-Based Sentiment Anlaysis(https://arxiv.org/abs/2404.06063)
• Keywords: transformer, generative
• Abstract: Aspect-Based Sentiment Analysis (ABSA) is an indispensable and highly challenging task in natural language processing. Current efforts have focused on specific sub-tasks, making it difficult to comprehensively cover all sub-tasks within the ABSA domain. With the development of Generative Pre-trained Transformers (GPTs), there came inspiration for a one-stop solution to sentiment analysis. In this study, we used GPTs for all sub-tasks of few-shot ABSA while defining a general learning paradigm for this application. We propose the All in One (AiO) model, a simple yet effective two-stage model for all ABSA sub-tasks. In the first stage, a specific backbone network learns the semantic information of the review and generates heuristically enhanced candidates. In the second stage, AiO leverages GPT contextual learning capabilities to generate predictions. The study conducted comprehensive comparative and ablation experiments on five benchmark datasets, and the results show that AiO can effectively handle all ABSA sub-tasks, even with few-shot data.

Title: LIPT: Latency-aware Image Processing Transformer

• Authors: Junbo Qiao, Wei Li, Haizhen Xie, Hanting Chen, Yunshuai Zhou, Zhijun Tu, Jie Hu, Shaohui Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06075
• Pdf URL: https://arxiv.org/pdf/2404.06075
• Copy Paste: [[2404.06075]] LIPT: Latency-aware Image Processing Transformer(https://arxiv.org/abs/2404.06075)
• Keywords: transformer
• Abstract: Transformer is leading a trend in the field of image processing. Despite the great success that existing lightweight image processing transformers have achieved, they are tailored to FLOPs or parameters reduction, rather than practical inference acceleration. In this paper, we present a latency-aware image processing transformer, termed LIPT. We devise the low-latency proportion LIPT block that substitutes memory-intensive operators with the combination of self-attention and convolutions to achieve practical speedup. Specifically, we propose a novel non-volatile sparse masking self-attention (NVSM-SA) that utilizes a pre-computing sparse mask to capture contextual information from a larger window with no extra computation overload. Besides, a high-frequency reparameterization module (HRM) is proposed to make LIPT block reparameterization friendly, which improves the model's detail reconstruction capability. Extensive experiments on multiple image processing tasks (e.g., image super-resolution (SR), JPEG artifact reduction, and image denoising) demonstrate the superiority of LIPT on both latency and PSNR. LIPT achieves real-time GPU inference with state-of-the-art performance on multiple image SR benchmarks.

Title: Fair Graph Neural Network with Supervised Contrastive Regularization

• Authors: Mahdi Tavassoli Kejani (UT3), Fadi Dornaika, Jean-Michel Loubes (IMT)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06090
• Pdf URL: https://arxiv.org/pdf/2404.06090
• Copy Paste: [[2404.06090]] Fair Graph Neural Network with Supervised Contrastive Regularization(https://arxiv.org/abs/2404.06090)
• Keywords: fair
• Abstract: In recent years, Graph Neural Networks (GNNs) have made significant advancements, particularly in tasks such as node classification, link prediction, and graph representation. However, challenges arise from biases that can be hidden not only in the node attributes but also in the connections between entities. Therefore, ensuring fairness in graph neural network learning has become a critical problem. To address this issue, we propose a novel model for training fairness-aware GNN, which enhances the Counterfactual Augmented Fair Graph Neural Network Framework (CAF). Our approach integrates Supervised Contrastive Loss and Environmental Loss to enhance both accuracy and fairness. Experimental validation on three real datasets demonstrates the superiority of our proposed model over CAF and several other existing graph-based learning methods.

Title: Hash3D: Training-free Acceleration for 3D Generation

• Authors: Xingyi Yang, Xinchao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06091
• Pdf URL: https://arxiv.org/pdf/2404.06091
• Copy Paste: [[2404.06091]] Hash3D: Training-free Acceleration for 3D Generation(https://arxiv.org/abs/2404.06091)
• Keywords: diffusion, generative
• Abstract: The evolution of 3D generative modeling has been notably propelled by the adoption of 2D diffusion models. Despite this progress, the cumbersome optimization process per se presents a critical hurdle to efficiency. In this paper, we introduce Hash3D, a universal acceleration for 3D generation without model training. Central to Hash3D is the insight that feature-map redundancy is prevalent in images rendered from camera positions and diffusion time-steps in close proximity. By effectively hashing and reusing these feature maps across neighboring timesteps and camera angles, Hash3D substantially prevents redundant calculations, thus accelerating the diffusion model's inference in 3D generation tasks. We achieve this through an adaptive grid-based hashing. Surprisingly, this feature-sharing mechanism not only speed up the generation but also enhances the smoothness and view consistency of the synthesized 3D objects. Our experiments covering 5 text-to-3D and 3 image-to-3D models, demonstrate Hash3D's versatility to speed up optimization, enhancing efficiency by 1.3 to 4 times. Additionally, Hash3D's integration with 3D Gaussian splatting largely speeds up 3D model creation, reducing text-to-3D processing to about 10 minutes and image-to-3D conversion to roughly 30 seconds. The project page is at https://adamdad.github.io/hash3D/.

Title: S-box Security Analysis of NIST Lightweight Cryptography Candidates: A Critical Empirical Study

• Authors: Mahnoor Naseer, Sundas Tariq, Naveed Riaz, Naveed Ahmed, Mureed Hussain
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.06094
• Pdf URL: https://arxiv.org/pdf/2404.06094
• Copy Paste: [[2404.06094]] S-box Security Analysis of NIST Lightweight Cryptography Candidates: A Critical Empirical Study(https://arxiv.org/abs/2404.06094)
• Keywords: security, attack
• Abstract: In the resource-constrained world of the digital landscape, lightweight cryptography plays a critical role in safeguarding information and ensuring the security of various systems, devices, and communication channels. Its efficient and resource-friendly nature makes it the ideal solution for applications where computational power is limited. In response to the growing need for platform-specific implementations, NIST issued a call for standardization of Lightweight cryptography algorithms in 2018. Ascon emerged as the winner of this competition. NIST initially established general evaluation criteria for a standard lightweight scheme including security strength, mitigation against side-channel and fault-injection attacks, and implementation efficiency. To verify the security claims, evaluating the individual components used in any cryptographic algorithm is a crucial step. The quality of a substitution box (S-box) significantly impacts the overall security of a cryptographic primitive. This paper analyzes the S-boxes of six finalists in the NIST Lightweight Cryptography (LWC) standardization process. We evaluate them based on well-established cryptographic properties. Our analysis explores how these properties influence the S-boxes' resistance against known cryptanalytic attacks and potential implementation-specific vulnerabilities, thus reflecting on their compliance with NIST's security requirements.

Title: Detection of fields of applications in biomedical abstracts with the support of argumentation elements

• Authors: Mariana Neves
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06121
• Pdf URL: https://arxiv.org/pdf/2404.06121
• Copy Paste: [[2404.06121]] Detection of fields of applications in biomedical abstracts with the support of argumentation elements(https://arxiv.org/abs/2404.06121)
• Keywords: extraction
• Abstract: Focusing on particular facts, instead of the complete text, can potentially improve searching for specific information in the scientific literature. In particular, argumentative elements allow focusing on specific parts of a publication, e.g., the background section or the claims from the authors. We evaluated some tools for the extraction of argumentation elements for a specific task in biomedicine, namely, for detecting the fields of the application in a biomedical publication, e.g, whether it addresses the problem of disease diagnosis or drug development. We performed experiments with the PubMedBERT pre-trained model, which was fine-tuned on a specific corpus for the task. We compared the use of title and abstract to restricting to only some argumentative elements. The top F1 scores ranged from 0.22 to 0.84, depending on the field of application. The best argumentative labels were the ones related the conclusion and background sections of an abstract.

Title: Hierarchical Insights: Exploiting Structural Similarities for Reliable 3D Semantic Segmentation

• Authors: Mariella Dreissig, Florian Piewak, Joschka Boedecker
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06124
• Pdf URL: https://arxiv.org/pdf/2404.06124
• Copy Paste: [[2404.06124]] Hierarchical Insights: Exploiting Structural Similarities for Reliable 3D Semantic Segmentation(https://arxiv.org/abs/2404.06124)
• Keywords: robust, segmentation
• Abstract: Safety-critical applications like autonomous driving call for robust 3D environment perception algorithms which can withstand highly diverse and ambiguous surroundings. The predictive performance of any classification model strongly depends on the underlying dataset and the prior knowledge conveyed by the annotated labels. While the labels provide a basis for the learning process, they usually fail to represent inherent relations between the classes - representations, which are a natural element of the human perception system. We propose a training strategy which enables a 3D LiDAR semantic segmentation model to learn structural relationships between the different classes through abstraction. We achieve this by implicitly modeling those relationships through a learning rule for hierarchical multi-label classification (HMC). With a detailed analysis we show, how this training strategy not only improves the model's confidence calibration, but also preserves additional information for downstream tasks like fusion, prediction and planning.

Title: FLEX: FLEXible Federated Learning Framework

• Authors: Francisco Herrera, Daniel Jiménez-López, Alberto Argente-Garrido, Nuria Rodríguez-Barroso, Cristina Zuheros, Ignacio Aguilera-Martos, Beatriz Bello, Mario García-Márquez, M. Victoria Luzón
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06127
• Pdf URL: https://arxiv.org/pdf/2404.06127
• Copy Paste: [[2404.06127]] FLEX: FLEXible Federated Learning Framework(https://arxiv.org/abs/2404.06127)
• Keywords: security, privacy, protect, attack, robust, federate
• Abstract: In the realm of Artificial Intelligence (AI), the need for privacy and security in data processing has become paramount. As AI applications continue to expand, the collection and handling of sensitive data raise concerns about individual privacy protection. Federated Learning (FL) emerges as a promising solution to address these challenges by enabling decentralized model training on local devices, thus preserving data privacy. This paper introduces FLEX: a FLEXible Federated Learning Framework designed to provide maximum flexibility in FL research experiments. By offering customizable features for data distribution, privacy parameters, and communication strategies, FLEX empowers researchers to innovate and develop novel FL techniques. The framework also includes libraries for specific FL implementations including: (1) anomalies, (2) blockchain, (3) adversarial attacks and defences, (4) natural language processing and (5) decision trees, enhancing its versatility and applicability in various domains. Overall, FLEX represents a significant advancement in FL research, facilitating the development of robust and efficient FL applications.

Title: Mansformer: Efficient Transformer of Mixed Attention for Image Deblurring and Beyond

• Authors: Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06135
• Pdf URL: https://arxiv.org/pdf/2404.06135
• Copy Paste: [[2404.06135]] Mansformer: Efficient Transformer of Mixed Attention for Image Deblurring and Beyond(https://arxiv.org/abs/2404.06135)
• Keywords: transformer
• Abstract: Transformer has made an enormous success in natural language processing and high-level vision over the past few years. However, the complexity of self-attention is quadratic to the image size, which makes it infeasible for high-resolution vision tasks. In this paper, we propose the Mansformer, a Transformer of mixed attention that combines multiple self-attentions, gate, and multi-layer perceptions (MLPs), to explore and employ more possibilities of self-attention. Taking efficiency into account, we design four kinds of self-attention, whose complexities are all linear. By elaborate adjustment of the tensor shapes and dimensions for the dot product, we split the typical self-attention of quadratic complexity into four operations of linear complexity. To adaptively merge these different kinds of self-attention, we take advantage of an architecture similar to Squeeze-and-Excitation Networks. Furthermore, we make it to merge the two-staged Transformer design into one stage by the proposed gated-dconv MLP. Image deblurring is our main target, while extensive quantitative and qualitative evaluations show that this method performs favorably against the state-of-the-art methods far more than simply deblurring. The source codes and trained models will be made available to the public.

Title: Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages

• Authors: Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Rifki Afina Putri, Emmanuel Dave, Jhonson Lee, Nuur Shadieq, Wawan Cenggoro, Salsabil Maulana Akbar, Muhammad Ihza Mahendra, Dea Annisayanti Putri, Bryan Wilie, Genta Indra Winata, Alham Fikri Aji, Ayu Purwarianti, Pascale Fung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06138
• Pdf URL: https://arxiv.org/pdf/2404.06138
• Copy Paste: [[2404.06138]] Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages(https://arxiv.org/abs/2404.06138)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder-decoder architectures across a range of model sizes. We highlight Cendol's effectiveness across a diverse array of tasks, attaining 20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning.

Title: DiffHarmony: Latent Diffusion Model Meets Image Harmonization

• Authors: Pengfei Zhou, Fangxiang Feng, Xiaojie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06139
• Pdf URL: https://arxiv.org/pdf/2404.06139
• Copy Paste: [[2404.06139]] DiffHarmony: Latent Diffusion Model Meets Image Harmonization(https://arxiv.org/abs/2404.06139)
• Keywords: diffusion
• Abstract: Image harmonization, which involves adjusting the foreground of a composite image to attain a unified visual consistency with the background, can be conceptualized as an image-to-image translation task. Diffusion models have recently promoted the rapid development of image-to-image translation tasks . However, training diffusion models from scratch is computationally intensive. Fine-tuning pre-trained latent diffusion models entails dealing with the reconstruction error induced by the image compression autoencoder, making it unsuitable for image generation tasks that involve pixel-level evaluation metrics. To deal with these issues, in this paper, we first adapt a pre-trained latent diffusion model to the image harmonization task to generate the harmonious but potentially blurry initial images. Then we implement two strategies: utilizing higher-resolution images during inference and incorporating an additional refinement stage, to further enhance the clarity of the initially harmonized images. Extensive experiments on iHarmony4 datasets demonstrate the superiority of our proposed method. The code and model will be made publicly available at https://github.com/nicecv/DiffHarmony .

Title: Differential Privacy for Anomaly Detection: Analyzing the Trade-off Between Privacy and Explainability

• Authors: Fatima Ezzeddine, Mirna Saad, Omran Ayoub, Davide Andreoletti, Martin Gjoreski, Ihab Sbeity, Marc Langheinrich, Silvia Giordano
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06144
• Pdf URL: https://arxiv.org/pdf/2404.06144
• Copy Paste: [[2404.06144]] Differential Privacy for Anomaly Detection: Analyzing the Trade-off Between Privacy and Explainability(https://arxiv.org/abs/2404.06144)
• Keywords: privacy, explainability
• Abstract: Anomaly detection (AD), also referred to as outlier detection, is a statistical process aimed at identifying observations within a dataset that significantly deviate from the expected pattern of the majority of the data. Such a process finds wide application in various fields, such as finance and healthcare. While the primary objective of AD is to yield high detection accuracy, the requirements of explainability and privacy are also paramount. The first ensures the transparency of the AD process, while the second guarantees that no sensitive information is leaked to untrusted parties. In this work, we exploit the trade-off of applying Explainable AI (XAI) through SHapley Additive exPlanations (SHAP) and differential privacy (DP). We perform AD with different models and on various datasets, and we thoroughly evaluate the cost of privacy in terms of decreased accuracy and explainability. Our results show that the enforcement of privacy through DP has a significant impact on detection accuracy and explainability, which depends on both the dataset and the considered AD model. We further show that the visual interpretation of explanations is also influenced by the choice of the AD algorithm.

Title: (Not) Understanding Latin Poetic Style with Deep Learning

• Authors: Ben Nagy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06150
• Pdf URL: https://arxiv.org/pdf/2404.06150
• Copy Paste: [[2404.06150]] (Not) Understanding Latin Poetic Style with Deep Learning(https://arxiv.org/abs/2404.06150)
• Keywords: interpretability
• Abstract: This article summarizes some mostly unsuccessful attempts to understand authorial style by examining the attention of various neural networks (LSTMs and CNNs) trained on a corpus of classical Latin verse that has been encoded to include sonic and metrical features. Carefully configured neural networks are shown to be extremely strong authorship classifiers, so it is hoped that they might therefore teach `traditional' readers something about how the authors differ in style. Sadly their reasoning is, so far, inscrutable. While the overall goal has not yet been reached, this work reports some useful findings in terms of effective ways to encode and embed verse, the relative strengths and weaknesses of the neural network families, and useful (and not so useful) techniques for designing and inspecting NN models in this domain. This article suggests that, for poetry, CNNs are better choices than LSTMs -- they train more quickly, have equivalent accuracy, and (potentially) offer better interpretability. Based on a great deal of experimentation, it also suggests that simple, trainable embeddings are more effective than domain-specific schemes, and stresses the importance of techniques to reduce overfitting, like dropout and batch normalization.

Title: scRDiT: Generating single-cell RNA-seq data by diffusion transformers and accelerating sampling

• Authors: Shengze Dong, Zhuorui Cui, Ding Liu, Jinzhi Lei
• Subjects: cs.LG, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2404.06153
• Pdf URL: https://arxiv.org/pdf/2404.06153
• Copy Paste: [[2404.06153]] scRDiT: Generating single-cell RNA-seq data by diffusion transformers and accelerating sampling(https://arxiv.org/abs/2404.06153)
• Keywords: diffusion, transformer, generative
• Abstract: Motivation: Single-cell RNA sequencing (scRNA-seq) is a groundbreaking technology extensively utilized in biological research, facilitating the examination of gene expression at the individual cell level within a given tissue sample. While numerous tools have been developed for scRNA-seq data analysis, the challenge persists in capturing the distinct features of such data and replicating virtual datasets that share analogous statistical properties. Results: Our study introduces a generative approach termed scRNA-seq Diffusion Transformer (scRDiT). This method generates virtual scRNA-seq data by leveraging a real dataset. The method is a neural network constructed based on Denoising Diffusion Probabilistic Models (DDPMs) and Diffusion Transformers (DiTs). This involves subjecting Gaussian noises to the real dataset through iterative noise-adding steps and ultimately restoring the noises to form scRNA-seq samples. This scheme allows us to learn data features from actual scRNA-seq samples during model training. Our experiments, conducted on two distinct scRNA-seq datasets, demonstrate superior performance. Additionally, the model sampling process is expedited by incorporating Denoising Diffusion Implicit Models (DDIM). scRDiT presents a unified methodology empowering users to train neural network models with their unique scRNA-seq datasets, enabling the generation of numerous high-quality scRNA-seq samples. Availability and implementation: https://github.com/DongShengze/scRDiT

Title: Efficient and Robust Point Cloud Registration via Heuristics-guided Parameter Search

• Authors: Tianyu Huang, Haoang Li, Liangzu Peng, Yinlong Liu, Yun-Hui Liu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06155
• Pdf URL: https://arxiv.org/pdf/2404.06155
• Copy Paste: [[2404.06155]] Efficient and Robust Point Cloud Registration via Heuristics-guided Parameter Search(https://arxiv.org/abs/2404.06155)
• Keywords: robust
• Abstract: Estimating the rigid transformation with 6 degrees of freedom based on a putative 3D correspondence set is a crucial procedure in point cloud registration. Existing correspondence identification methods usually lead to large outlier ratios ($>$ 95 $\%$ is common), underscoring the significance of robust registration methods. Many researchers turn to parameter search-based strategies (e.g., Branch-and-Bround) for robust registration. Although related methods show high robustness, their efficiency is limited to the high-dimensional search space. This paper proposes a heuristics-guided parameter search strategy to accelerate the search while maintaining high robustness. We first sample some correspondences (i.e., heuristics) and then just need to sequentially search the feasible regions that make each sample an inlier. Our strategy largely reduces the search space and can guarantee accuracy with only a few inlier samples, therefore enjoying an excellent trade-off between efficiency and robustness. Since directly parameterizing the 6-dimensional nonlinear feasible region for efficient search is intractable, we construct a three-stage decomposition pipeline to reparameterize the feasible region, resulting in three lower-dimensional sub-problems that are easily solvable via our strategy. Besides reducing the searching dimension, our decomposition enables the leverage of 1-dimensional interval stabbing at all three stages for searching acceleration. Moreover, we propose a valid sampling strategy to guarantee our sampling effectiveness, and a compatibility verification setup to further accelerate our search. Extensive experiments on both simulated and real-world datasets demonstrate that our approach exhibits comparable robustness with state-of-the-art methods while achieving a significant efficiency boost.

Title: Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports

• Authors: Tianyu Cao, Natraj Raman, Danial Dervovic, Chenhao Tan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06162
• Pdf URL: https://arxiv.org/pdf/2404.06162
• Copy Paste: [[2404.06162]] Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports(https://arxiv.org/abs/2404.06162)
• Keywords: large language model
• Abstract: As large language models (LLMs) expand the power of natural language processing to handle long inputs, rigorous and systematic analyses are necessary to understand their abilities and behavior. A salient application is summarization, due to its ubiquity and controversy (e.g., researchers have declared the death of summarization). In this paper, we use financial report summarization as a case study because financial reports not only are long but also use numbers and tables extensively. We propose a computational framework for characterizing multimodal long-form summarization and investigate the behavior of Claude 2.0/2.1, GPT-4/3.5, and Command. We find that GPT-3.5 and Command fail to perform this summarization task meaningfully. For Claude 2 and GPT-4, we analyze the extractiveness of the summary and identify a position bias in LLMs. This position bias disappears after shuffling the input for Claude, which suggests that Claude has the ability to recognize important information. We also conduct a comprehensive investigation on the use of numeric data in LLM-generated summaries and offer a taxonomy of numeric hallucination. We employ prompt engineering to improve GPT-4's use of numbers with limited success. Overall, our analyses highlight the strong capability of Claude 2 in handling long multimodal inputs compared to GPT-4.

Title: Enhanced Radar Perception via Multi-Task Learning: Towards Refined Data for Sensor Fusion Applications

• Authors: Huawei Sun, Hao Feng, Gianfranco Mauro, Julius Ott, Georg Stettinger, Lorenzo Servadei, Robert Wille
• Subjects: cs.CV, cs.MM, eess.IV, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.06165
• Pdf URL: https://arxiv.org/pdf/2404.06165
• Copy Paste: [[2404.06165]] Enhanced Radar Perception via Multi-Task Learning: Towards Refined Data for Sensor Fusion Applications(https://arxiv.org/abs/2404.06165)
• Keywords: robust
• Abstract: Radar and camera fusion yields robustness in perception tasks by leveraging the strength of both sensors. The typical extracted radar point cloud is 2D without height information due to insufficient antennas along the elevation axis, which challenges the network performance. This work introduces a learning-based approach to infer the height of radar points associated with 3D objects. A novel robust regression loss is introduced to address the sparse target challenge. In addition, a multi-task training strategy is employed, emphasizing important features. The average radar absolute height error decreases from 1.69 to 0.25 meters compared to the state-of-the-art height extension method. The estimated target height values are used to preprocess and enrich radar data for downstream perception tasks. Integrating this refined radar information further enhances the performance of existing radar camera fusion models for object detection and depth estimation tasks.

Title: scCDCG: Efficient Deep Structural Clustering for single-cell RNA-seq via Deep Cut-informed Graph Embedding

• Authors: Ping Xu, Zhiyuan Ning, Meng Xiao, Guihai Feng, Xin Li, Yuanchun Zhou, Pengfei Wang
• Subjects: cs.LG, cs.AI, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2404.06167
• Pdf URL: https://arxiv.org/pdf/2404.06167
• Copy Paste: [[2404.06167]] scCDCG: Efficient Deep Structural Clustering for single-cell RNA-seq via Deep Cut-informed Graph Embedding(https://arxiv.org/abs/2404.06167)
• Keywords: extraction
• Abstract: Single-cell RNA sequencing (scRNA-seq) is essential for unraveling cellular heterogeneity and diversity, offering invaluable insights for bioinformatics advancements. Despite its potential, traditional clustering methods in scRNA-seq data analysis often neglect the structural information embedded in gene expression profiles, crucial for understanding cellular correlations and dependencies. Existing strategies, including graph neural networks, face challenges in handling the inefficiency due to scRNA-seq data's intrinsic high-dimension and high-sparsity. Addressing these limitations, we introduce scCDCG (single-cell RNA-seq Clustering via Deep Cut-informed Graph), a novel framework designed for efficient and accurate clustering of scRNA-seq data that simultaneously utilizes intercellular high-order structural information. scCDCG comprises three main components: (i) A graph embedding module utilizing deep cut-informed techniques, which effectively captures intercellular high-order structural information, overcoming the over-smoothing and inefficiency issues prevalent in prior graph neural network methods. (ii) A self-supervised learning module guided by optimal transport, tailored to accommodate the unique complexities of scRNA-seq data, specifically its high-dimension and high-sparsity. (iii) An autoencoder-based feature learning module that simplifies model complexity through effective dimension reduction and feature extraction. Our extensive experiments on 6 datasets demonstrate scCDCG's superior performance and efficiency compared to 7 established models, underscoring scCDCG's potential as a transformative tool in scRNA-seq data analysis. Our code is available at: https://github.com/XPgogogo/scCDCG.

Title: Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank

• Authors: Jiaxin Wu, Chong-Wah Ngo, Wing-Kwong Chan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06173
• Pdf URL: https://arxiv.org/pdf/2404.06173
• Copy Paste: [[2404.06173]] Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank(https://arxiv.org/abs/2404.06173)
• Keywords: generative
• Abstract: Aligning a user query and video clips in cross-modal latent space and that with semantic concepts are two mainstream approaches for ad-hoc video search (AVS). However, the effectiveness of existing approaches is bottlenecked by the small sizes of available video-text datasets and the low quality of concept banks, which results in the failures of unseen queries and the out-of-vocabulary problem. This paper addresses these two problems by constructing a new dataset and developing a multi-word concept bank. Specifically, capitalizing on a generative model, we construct a new dataset consisting of 7 million generated text and video pairs for pre-training. To tackle the out-of-vocabulary problem, we develop a multi-word concept bank based on syntax analysis to enhance the capability of a state-of-the-art interpretable AVS method in modeling relationships between query words. We also study the impact of current advanced features on the method. Experimental results show that the integration of the above-proposed elements doubles the R@1 performance of the AVS method on the MSRVTT dataset and improves the xinfAP on the TRECVid AVS query sets for 2016-2023 (eight years) by a margin from 2% to 77%, with an average about 20%.

Title: Uncertainty-aware Evidential Fusion-based Learning for Semi-supervised Medical Image Segmentation

• Authors: Yuanpeng He, Lijian Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06177
• Pdf URL: https://arxiv.org/pdf/2404.06177
• Copy Paste: [[2404.06177]] Uncertainty-aware Evidential Fusion-based Learning for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2404.06177)
• Keywords: segmentation
• Abstract: Although the existing uncertainty-based semi-supervised medical segmentation methods have achieved excellent performance, they usually only consider a single uncertainty evaluation, which often fails to solve the problem related to credibility completely. Therefore, based on the framework of evidential deep learning, this paper integrates the evidential predictive results in the cross-region of mixed and original samples to reallocate the confidence degree and uncertainty measure of each voxel, which is realized by emphasizing uncertain information of probability assignments fusion rule of traditional evidence theory. Furthermore, we design a voxel-level asymptotic learning strategy by introducing information entropy to combine with the fused uncertainty measure to estimate voxel prediction more precisely. The model will gradually pay attention to the prediction results with high uncertainty in the learning process, to learn the features that are difficult to master. The experimental results on LA, Pancreas-CT, ACDC and TBAD datasets demonstrate the superior performance of our proposed method in comparison with the existing state of the arts.

Title: EPL: Evidential Prototype Learning for Semi-supervised Medical Image Segmentation

• Authors: Yuanpeng He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06181
• Pdf URL: https://arxiv.org/pdf/2404.06181
• Copy Paste: [[2404.06181]] EPL: Evidential Prototype Learning for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2404.06181)
• Keywords: segmentation
• Abstract: Although current semi-supervised medical segmentation methods can achieve decent performance, they are still affected by the uncertainty in unlabeled data and model predictions, and there is currently a lack of effective strategies that can explore the uncertain aspects of both simultaneously. To address the aforementioned issues, we propose Evidential Prototype Learning (EPL), which utilizes an extended probabilistic framework to effectively fuse voxel probability predictions from different sources and achieves prototype fusion utilization of labeled and unlabeled data under a generalized evidential framework, leveraging voxel-level dual uncertainty masking. The uncertainty not only enables the model to self-correct predictions but also improves the guided learning process with pseudo-labels and is able to feed back into the construction of hidden features. The method proposed in this paper has been experimented on LA, Pancreas-CT and TBAD datasets, achieving the state-of-the-art performance in three different labeled ratios, which strongly demonstrates the effectiveness of our strategy.

Title: Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles

• Authors: Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, Leonardo Rigutini
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06186
• Pdf URL: https://arxiv.org/pdf/2404.06186
• Copy Paste: [[2404.06186]] Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles(https://arxiv.org/abs/2404.06186)
• Keywords: large language model
• Abstract: Crossword puzzles are popular linguistic games often used as tools to engage students in learning. Educational crosswords are characterized by less cryptic and more factual clues that distinguish them from traditional crossword puzzles. Despite there exist several publicly available clue-answer pair databases for traditional crosswords, educational clue-answer pairs datasets are missing. In this article, we propose a methodology to build educational clue generation datasets that can be used to instruct Large Language Models (LLMs). By gathering from Wikipedia pages informative content associated with relevant keywords, we use Large Language Models to automatically generate pedagogical clues related to the given input keyword and its context. With such an approach, we created clue-instruct, a dataset containing 44,075 unique examples with text-keyword pairs associated with three distinct crossword clues. We used clue-instruct to instruct different LLMs to generate educational clues from a given input content and keyword. Both human and automatic evaluations confirmed the quality of the generated clues, thus validating the effectiveness of our approach.

Title: Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

• Authors: Xudong Yu, Chenjia Bai, Hongyi Guo, Changhong Wang, Zhen Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06188
• Pdf URL: https://arxiv.org/pdf/2404.06188
• Copy Paste: [[2404.06188]] Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning(https://arxiv.org/abs/2404.06188)
• Keywords: robust
• Abstract: Offline Reinforcement Learning (RL) faces distributional shift and unreliable value estimation, especially for out-of-distribution (OOD) actions. To address this, existing uncertainty-based methods penalize the value function with uncertainty quantification and demand numerous ensemble networks, posing computational challenges and suboptimal outcomes. In this paper, we introduce a novel strategy employing diverse randomized value functions to estimate the posterior distribution of $Q$-values. It provides robust uncertainty quantification and estimates lower confidence bounds (LCB) of $Q$-values. By applying moderate value penalties for OOD actions, our method fosters a provably pessimistic approach. We also emphasize on diversity within randomized value functions and enhance efficiency by introducing a diversity regularization method, reducing the requisite number of networks. These modules lead to reliable value estimation and efficient policy learning from offline data. Theoretical analysis shows that our method recovers the provably efficient LCB-penalty under linear MDP assumptions. Extensive empirical results also demonstrate that our proposed method significantly outperforms baseline methods in terms of performance and parametric efficiency.

Title: Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection

• Authors: Ting Lei, Shaofeng Yin, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06194
• Pdf URL: https://arxiv.org/pdf/2404.06194
• Copy Paste: [[2404.06194]] Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection(https://arxiv.org/abs/2404.06194)
• Keywords: large language model
• Abstract: Open-vocabulary human-object interaction (HOI) detection, which is concerned with the problem of detecting novel HOIs guided by natural language, is crucial for understanding human-centric scenes. However, prior zero-shot HOI detectors often employ the same levels of feature maps to model HOIs with varying distances, leading to suboptimal performance in scenes containing human-object pairs with a wide range of distances. In addition, these detectors primarily rely on category names and overlook the rich contextual information that language can provide, which is essential for capturing open vocabulary concepts that are typically rare and not well-represented by category names alone. In this paper, we introduce a novel end-to-end open vocabulary HOI detection framework with conditional multi-level decoding and fine-grained semantic enhancement (CMD-SE), harnessing the potential of Visual-Language Models (VLMs). Specifically, we propose to model human-object pairs with different distances with different levels of feature maps by incorporating a soft constraint during the bipartite matching process. Furthermore, by leveraging large language models (LLMs) such as GPT models, we exploit their extensive world knowledge to generate descriptions of human body part states for various interactions. Then we integrate the generalizable and fine-grained semantics of human body parts to improve interaction recognition. Experimental results on two datasets, SWIG-HOI and HICO-DET, demonstrate that our proposed method achieves state-of-the-art results in open vocabulary HOI detection. The code and models are available at https://github.com/ltttpku/CMD-SE-release.

Title: Automated National Urban Map Extraction

• Authors: Hasan Nasrallah, Abed Ellatif Samhat, Cristiano Nattero, Ali J. Ghandour
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06202
• Pdf URL: https://arxiv.org/pdf/2404.06202
• Copy Paste: [[2404.06202]] Automated National Urban Map Extraction(https://arxiv.org/abs/2404.06202)
• Keywords: extraction, segmentation
• Abstract: Developing countries usually lack the proper governance means to generate and regularly update a national rooftop map. Using traditional photogrammetry and surveying methods to produce a building map at the federal level is costly and time consuming. Using earth observation and deep learning methods, we can bridge this gap and propose an automated pipeline to fetch such national urban maps. This paper aims to exploit the power of fully convolutional neural networks for multi-class buildings' instance segmentation to leverage high object-wise accuracy results. Buildings' instance segmentation from sub-meter high-resolution satellite images can be achieved with relatively high pixel-wise metric scores. We detail all engineering steps to replicate this work and ensure highly accurate results in dense and slum areas witnessed in regions that lack proper urban planning in the Global South. We applied a case study of the proposed pipeline to Lebanon and successfully produced the first comprehensive national building footprint map with approximately 1 Million units with an 84% accuracy. The proposed architecture relies on advanced augmentation techniques to overcome dataset scarcity, which is often the case in developing countries.

Title: Leveraging edge detection and neural networks for better UAV localization

• Authors: Theo Di Piazza, Enric Meinhardt-Llopis, Gabriele Facciolo, Benedicte Bascle, Corentin Abgrall, Jean-Clement Devaux
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06207
• Pdf URL: https://arxiv.org/pdf/2404.06207
• Copy Paste: [[2404.06207]] Leveraging edge detection and neural networks for better UAV localization(https://arxiv.org/abs/2404.06207)
• Keywords: robust
• Abstract: We propose a novel method for geolocalizing Unmanned Aerial Vehicles (UAVs) in environments lacking Global Navigation Satellite Systems (GNSS). Current state-of-the-art techniques employ an offline-trained encoder to generate a vector representation (embedding) of the UAV's current view, which is then compared with pre-computed embeddings of geo-referenced images to determine the UAV's position. Here, we demonstrate that the performance of these methods can be significantly enhanced by preprocessing the images to extract their edges, which exhibit robustness to seasonal and illumination variations. Furthermore, we establish that utilizing edges enhances resilience to orientation and altitude inaccuracies. Additionally, we introduce a confidence criterion for localization. Our findings are substantiated through synthetic experiments.

Title: Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models

• Authors: Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, Rich Caruana
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06209
• Pdf URL: https://arxiv.org/pdf/2404.06209
• Copy Paste: [[2404.06209]] Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models(https://arxiv.org/abs/2404.06209)
• Keywords: robust, large language model
• Abstract: While many have shown how Large Language Models (LLMs) can be applied to a diverse set of tasks, the critical issues of data contamination and memorization are often glossed over. In this work, we address this concern for tabular data. Specifically, we introduce a variety of different techniques to assess whether a language model has seen a tabular dataset during training. This investigation reveals that LLMs have memorized many popular tabular datasets verbatim. We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training. We find that LLMs perform better on datasets seen during training, indicating that memorization leads to overfitting. At the same time, LLMs show non-trivial performance on novel datasets and are surprisingly robust to data transformations. We then investigate the in-context statistical learning abilities of LLMs. Without fine-tuning, we find them to be limited. This suggests that much of the few-shot performance on novel datasets is due to the LLM's world knowledge. Overall, our results highlight the importance of testing whether an LLM has seen an evaluation dataset during pre-training. We make the exposure tests we developed available as the tabmemcheck Python package at https://github.com/interpretml/LLM-Tabular-Memorization-Checker

Title: Unified Physical-Digital Attack Detection Challenge

• Authors: Haocheng Yuan, Ajian Liu, Junze Zheng, Jun Wan, Jiankang Deng, Sergio Escalera, Hugo Jair Escalante, Isabelle Guyon, Zhen Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06211
• Pdf URL: https://arxiv.org/pdf/2404.06211
• Copy Paste: [[2404.06211]] Unified Physical-Digital Attack Detection Challenge(https://arxiv.org/abs/2404.06211)
• Keywords: attack
• Abstract: Face Anti-Spoofing (FAS) is crucial to safeguard Face Recognition (FR) Systems. In real-world scenarios, FRs are confronted with both physical and digital attacks. However, existing algorithms often address only one type of attack at a time, which poses significant limitations in real-world scenarios where FR systems face hybrid physical-digital threats. To facilitate the research of Unified Attack Detection (UAD) algorithms, a large-scale UniAttackData dataset has been collected. UniAttackData is the largest public dataset for Unified Attack Detection, with a total of 28,706 videos, where each unique identity encompasses all advanced attack types. Based on this dataset, we organized a Unified Physical-Digital Face Attack Detection Challenge to boost the research in Unified Attack Detections. It attracted 136 teams for the development phase, with 13 qualifying for the final round. The results re-verified by the organizing team were used for the final ranking. This paper comprehensively reviews the challenge, detailing the dataset introduction, protocol definition, evaluation criteria, and a summary of published results. Finally, we focus on the detailed analysis of the highest-performing algorithms and offer potential directions for unified physical-digital attack detection inspired by this competition. Challenge Website: https://sites.google.com/view/face-anti-spoofing-challenge/welcome/challengecvpr2024.

Title: OmniFusion Technical Report

• Authors: Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk, Maxim Kurkin, Irina Abdullaeva, Matvey Skripkin, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06212
• Pdf URL: https://arxiv.org/pdf/2404.06212
• Copy Paste: [[2404.06212]] OmniFusion Technical Report(https://arxiv.org/abs/2404.06212)
• Keywords: transformer, large language model
• Abstract: Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekeeping, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion.

Title: Privacy-preserving Scanpath Comparison for Pervasive Eye Tracking

• Authors: Suleyman Ozdel, Efe Bozkir, Enkelejda Kasneci
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.06216
• Pdf URL: https://arxiv.org/pdf/2404.06216
• Copy Paste: [[2404.06216]] Privacy-preserving Scanpath Comparison for Pervasive Eye Tracking(https://arxiv.org/abs/2404.06216)
• Keywords: privacy
• Abstract: As eye tracking becomes pervasive with screen-based devices and head-mounted displays, privacy concerns regarding eye-tracking data have escalated. While state-of-the-art approaches for privacy-preserving eye tracking mostly involve differential privacy and empirical data manipulations, previous research has not focused on methods for scanpaths. We introduce a novel privacy-preserving scanpath comparison protocol designed for the widely used Needleman-Wunsch algorithm, a generalized version of the edit distance algorithm. Particularly, by incorporating the Paillier homomorphic encryption scheme, our protocol ensures that no private information is revealed. Furthermore, we introduce a random processing strategy and a multi-layered masking method to obfuscate the values while preserving the original order of encrypted editing operation costs. This minimizes communication overhead, requiring a single communication round for each iteration of the Needleman-Wunsch process. We demonstrate the efficiency and applicability of our protocol on three publicly available datasets with comprehensive computational performance analyses and make our source code publicly accessible.

Title: VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detection

• Authors: Li-Ming Zhan, Bo Liu, Xiao-Ming Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06217
• Pdf URL: https://arxiv.org/pdf/2404.06217
• Copy Paste: [[2404.06217]] VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detection(https://arxiv.org/abs/2404.06217)
• Keywords: transformer
• Abstract: Out-of-distribution (OOD) detection plays a crucial role in ensuring the safety and reliability of deep neural networks in various applications. While there has been a growing focus on OOD detection in visual data, the field of textual OOD detection has received less attention. Only a few attempts have been made to directly apply general OOD detection methods to natural language processing (NLP) tasks, without adequately considering the characteristics of textual data. In this paper, we delve into textual OOD detection with Transformers. We first identify a key problem prevalent in existing OOD detection methods: the biased representation learned through the maximization of the conditional likelihood $p(y\mid x)$ can potentially result in subpar performance. We then propose a novel variational inference framework for OOD detection (VI-OOD), which maximizes the likelihood of the joint distribution $p(x, y)$ instead of $p(y\mid x)$. VI-OOD is tailored for textual OOD detection by efficiently exploiting the representations of pre-trained Transformers. Through comprehensive experiments on various text classification tasks, VI-OOD demonstrates its effectiveness and wide applicability. Our code has been released at \url{https://github.com/liam0949/LLM-OOD}.

Title: Aggressive or Imperceptible, or Both: Network Pruning Assisted Hybrid Byzantines in Federated Learning

• Authors: Emre Ozfatura, Kerem Ozfatura, Alptekin Kupcu, Deniz Gunduz
• Subjects: cs.LG, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.06230
• Pdf URL: https://arxiv.org/pdf/2404.06230
• Copy Paste: [[2404.06230]] Aggressive or Imperceptible, or Both: Network Pruning Assisted Hybrid Byzantines in Federated Learning(https://arxiv.org/abs/2404.06230)
• Keywords: security, privacy, attack, federate
• Abstract: Federated learning (FL) has been introduced to enable a large number of clients, possibly mobile devices, to collaborate on generating a generalized machine learning model thanks to utilizing a larger number of local samples without sharing to offer certain privacy to collaborating clients. However, due to the participation of a large number of clients, it is often difficult to profile and verify each client, which leads to a security threat that malicious participants may hamper the accuracy of the trained model by conveying poisoned models during the training. Hence, the aggregation framework at the parameter server also needs to minimize the detrimental effects of these malicious clients. A plethora of attack and defence strategies have been analyzed in the literature. However, often the Byzantine problem is analyzed solely from the outlier detection perspective, being oblivious to the topology of neural networks (NNs). In the scope of this work, we argue that by extracting certain side information specific to the NN topology, one can design stronger attacks. Hence, inspired by the sparse neural networks, we introduce a hybrid sparse Byzantine attack that is composed of two parts: one exhibiting a sparse nature and attacking only certain NN locations with higher sensitivity, and the other being more silent but accumulating over time, where each ideally targets a different type of defence mechanism, and together they form a strong but imperceptible attack. Finally, we show through extensive simulations that the proposed hybrid Byzantine attack is effective against 8 different defence methods.

Title: Towards Robust Domain Generation Algorithm Classification

• Authors: Arthur Drichel, Marc Meyer, Ulrike Meyer
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06236
• Pdf URL: https://arxiv.org/pdf/2404.06236
• Copy Paste: [[2404.06236]] Towards Robust Domain Generation Algorithm Classification(https://arxiv.org/abs/2404.06236)
• Keywords: defense, attack, robust
• Abstract: In this work, we conduct a comprehensive study on the robustness of domain generation algorithm (DGA) classifiers. We implement 32 white-box attacks, 19 of which are very effective and induce a false-negative rate (FNR) of $\approx$ 100\% on unhardened classifiers. To defend the classifiers, we evaluate different hardening approaches and propose a novel training scheme that leverages adversarial latent space vectors and discretized adversarial domains to significantly improve robustness. In our study, we highlight a pitfall to avoid when hardening classifiers and uncover training biases that can be easily exploited by attackers to bypass detection, but which can be mitigated by adversarial training (AT). In our study, we do not observe any trade-off between robustness and performance, on the contrary, hardening improves a classifier's detection performance for known and unknown DGAs. We implement all attacks and defenses discussed in this paper as a standalone library, which we make publicly available to facilitate hardening of DGA classifiers: https://gitlab.com/rwth-itsec/robust-dga-detection

Title: Hyperparameter-Free Medical Image Synthesis for Sharing Data and Improving Site-Specific Segmentation

• Authors: Alexander Chebykin, Peter A. N. Bosman, Tanja Alderliesten
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06240
• Pdf URL: https://arxiv.org/pdf/2404.06240
• Copy Paste: [[2404.06240]] Hyperparameter-Free Medical Image Synthesis for Sharing Data and Improving Site-Specific Segmentation(https://arxiv.org/abs/2404.06240)
• Keywords: security, privacy, segmentation
• Abstract: Sharing synthetic medical images is a promising alternative to sharing real images that can improve patient privacy and data security. To get good results, existing methods for medical image synthesis must be manually adjusted when they are applied to unseen data. To remove this manual burden, we introduce a Hyperparameter-Free distributed learning method for automatic medical image Synthesis, Sharing, and Segmentation called HyFree-S3. For three diverse segmentation settings (pelvic MRIs, lung X-rays, polyp photos), the use of HyFree-S3 results in improved performance over training only with site-specific data (in the majority of cases). The hyperparameter-free nature of the method should make data synthesis and sharing easier, potentially leading to an increase in the quantity of available data and consequently the quality of the models trained that may ultimately be applied in the clinic. Our code is available at https://github.com/AwesomeLemon/HyFree-S3

Title: ActNetFormer: Transformer-ResNet Hybrid Method for Semi-Supervised Action Recognition in Videos

• Authors: Sharana Dharshikgan Suresh Dass, Hrishav Bakul Barua, Ganesh Krishnasamy, Raveendran Paramesran, Raphael C.-W. Phan
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.06243
• Pdf URL: https://arxiv.org/pdf/2404.06243
• Copy Paste: [[2404.06243]] ActNetFormer: Transformer-ResNet Hybrid Method for Semi-Supervised Action Recognition in Videos(https://arxiv.org/abs/2404.06243)
• Keywords: robust, transformer
• Abstract: Human action or activity recognition in videos is a fundamental task in computer vision with applications in surveillance and monitoring, self-driving cars, sports analytics, human-robot interaction and many more. Traditional supervised methods require large annotated datasets for training, which are expensive and time-consuming to acquire. This work proposes a novel approach using Cross-Architecture Pseudo-Labeling with contrastive learning for semi-supervised action recognition. Our framework leverages both labeled and unlabelled data to robustly learn action representations in videos, combining pseudo-labeling with contrastive learning for effective learning from both types of samples. We introduce a novel cross-architecture approach where 3D Convolutional Neural Networks (3D CNNs) and video transformers (VIT) are utilised to capture different aspects of action representations; hence we call it ActNetFormer. The 3D CNNs excel at capturing spatial features and local dependencies in the temporal domain, while VIT excels at capturing long-range dependencies across frames. By integrating these complementary architectures within the ActNetFormer framework, our approach can effectively capture both local and global contextual information of an action. This comprehensive representation learning enables the model to achieve better performance in semi-supervised action recognition tasks by leveraging the strengths of each of these architectures. Experimental results on standard action recognition datasets demonstrate that our approach performs better than the existing methods, achieving state-of-the-art performance with only a fraction of labeled data. The official website of this work is available at: https://github.com/rana2149/ActNetFormer.

Title: Anchor-based Robust Finetuning of Vision-Language Models

• Authors: Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao, Ke Yan, Shouhong Ding, Yuan Gao, Gui-Song Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06244
• Pdf URL: https://arxiv.org/pdf/2404.06244
• Copy Paste: [[2404.06244]] Anchor-based Robust Finetuning of Vision-Language Models(https://arxiv.org/abs/2404.06244)
• Keywords: robust
• Abstract: We aim at finetuning a vision-language model without hurting its out-of-distribution (OOD) generalization. We address two types of OOD generalization, i.e., i) domain shift such as natural to sketch images, and ii) zero-shot capability to recognize the category that was not contained in the finetune data. Arguably, the diminished OOD generalization after finetuning stems from the excessively simplified finetuning target, which only provides the class information, such as ``a photo of a [CLASS]''. This is distinct from the process in that CLIP was pretrained, where there is abundant text supervision with rich semantic information. Therefore, we propose to compensate for the finetune process using auxiliary supervision with rich semantic information, which acts as anchors to preserve the OOD generalization. Specifically, two types of anchors are elaborated in our method, including i) text-compensated anchor which uses the images from the finetune set but enriches the text supervision from a pretrained captioner, ii) image-text-pair anchor which is retrieved from the dataset similar to pretraining data of CLIP according to the downstream task, associating with the original CLIP text with rich semantics. Those anchors are utilized as auxiliary semantic information to maintain the original feature space of CLIP, thereby preserving the OOD generalization capabilities. Comprehensive experiments demonstrate that our method achieves in-distribution performance akin to conventional finetuning while attaining new state-of-the-art results on domain shift and zero-shot learning benchmarks.

Title: LRR: Language-Driven Resamplable Continuous Representation against Adversarial Tracking Attacks

• Authors: Jianlang Chen, Xuhong Ren, Qing Guo, Felix Juefei-Xu, Di Lin, Wei Feng, Lei Ma, Jianjun Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06247
• Pdf URL: https://arxiv.org/pdf/2404.06247
• Copy Paste: [[2404.06247]] LRR: Language-Driven Resamplable Continuous Representation against Adversarial Tracking Attacks(https://arxiv.org/abs/2404.06247)
• Keywords: security, attack, robust
• Abstract: Visual object tracking plays a critical role in visual-based autonomous systems, as it aims to estimate the position and size of the object of interest within a live video. Despite significant progress made in this field, state-of-the-art (SOTA) trackers often fail when faced with adversarial perturbations in the incoming frames. This can lead to significant robustness and security issues when these trackers are deployed in the real world. To achieve high accuracy on both clean and adversarial data, we propose building a spatial-temporal continuous representation using the semantic text guidance of the object of interest. This novel continuous representation enables us to reconstruct incoming frames to maintain semantic and appearance consistency with the object of interest and its clean counterparts. As a result, our proposed method successfully defends against different SOTA adversarial tracking attacks while maintaining high accuracy on clean data. In particular, our method significantly increases tracking accuracy under adversarial attacks with around 90% relative improvement on UAV123, which is even higher than the accuracy on clean data.

Title: From Barlow Twins to Triplet Training: Differentiating Dementia with Limited Data

• Authors: Yitong Li, Tom Nuno Wolf, Sebastian Pölsterl, Igor Yakushev, Dennis M. Hedderich, Christian Wachinger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06253
• Pdf URL: https://arxiv.org/pdf/2404.06253
• Copy Paste: [[2404.06253]] From Barlow Twins to Triplet Training: Differentiating Dementia with Limited Data(https://arxiv.org/abs/2404.06253)
• Keywords: robust
• Abstract: Differential diagnosis of dementia is challenging due to overlapping symptoms, with structural magnetic resonance imaging (MRI) being the primary method for diagnosis. Despite the clinical value of computer-aided differential diagnosis, research has been limited, mainly due to the absence of public datasets that contain diverse types of dementia. This leaves researchers with small in-house datasets that are insufficient for training deep neural networks (DNNs). Self-supervised learning shows promise for utilizing unlabeled MRI scans in training, but small batch sizes for volumetric brain scans make its application challenging. To address these issues, we propose Triplet Training for differential diagnosis with limited target data. It consists of three key stages: (i) self-supervised pre-training on unlabeled data with Barlow Twins, (ii) self-distillation on task-related data, and (iii) fine-tuning on the target dataset. Our approach significantly outperforms traditional training strategies, achieving a balanced accuracy of 75.6%. We further provide insights into the training process by visualizing changes in the latent space after each step. Finally, we validate the robustness of Triplet Training in terms of its individual components in a comprehensive ablation study. Our code is available at https://github.com/ai-med/TripletTraining.

Title: Robust feature knowledge distillation for enhanced performance of lightweight crack segmentation models

• Authors: Zhaohui Chen, Elyas Asadi Shamsabadi, Sheng Jiang, Luming Shen, Daniel Dias-da-Costa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06258
• Pdf URL: https://arxiv.org/pdf/2404.06258
• Copy Paste: [[2404.06258]] Robust feature knowledge distillation for enhanced performance of lightweight crack segmentation models(https://arxiv.org/abs/2404.06258)
• Keywords: robust, segmentation
• Abstract: Vision-based crack detection faces deployment challenges due to the size of robust models and edge device limitations. These can be addressed with lightweight models trained with knowledge distillation (KD). However, state-of-the-art (SOTA) KD methods compromise anti-noise robustness. This paper develops Robust Feature Knowledge Distillation (RFKD), a framework to improve robustness while retaining the precision of light models for crack segmentation. RFKD distils knowledge from a teacher model's logit layers and intermediate feature maps while leveraging mixed clean and noisy images to transfer robust patterns to the student model, improving its precision, generalisation, and anti-noise performance. To validate the proposed RFKD, a lightweight crack segmentation model, PoolingCrack Tiny (PCT), with only 0.5 M parameters, is also designed and used as the student to run the framework. The results show a significant enhancement in noisy images, with RFKD reaching a 62% enhanced mean Dice score (mDS) compared to SOTA KD methods.

Title: Playing to Vision Foundation Model's Strengths in Stereo Matching

• Authors: Chuang-Wei Liu, Qijun Chen, Rui Fan
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06261
• Pdf URL: https://arxiv.org/pdf/2404.06261
• Copy Paste: [[2404.06261]] Playing to Vision Foundation Model's Strengths in Stereo Matching(https://arxiv.org/abs/2404.06261)
• Keywords: extraction, transformer
• Abstract: Stereo matching has become a key technique for 3D environment perception in intelligent vehicles. For a considerable time, convolutional neural networks (CNNs) have remained the mainstream choice for feature extraction in this domain. Nonetheless, there is a growing consensus that the existing paradigm should evolve towards vision foundation models (VFM), particularly those developed based on vision Transformers (ViTs) and pre-trained through self-supervision on extensive, unlabeled datasets. While VFMs are adept at extracting informative, general-purpose visual features, specifically for dense prediction tasks, their performance often lacks in geometric vision tasks. This study serves as the first exploration of a viable approach for adapting VFMs to stereo matching. Our ViT adapter, referred to as ViTAS, is constructed upon three types of modules: spatial differentiation, patch attention fusion, and cross-attention. The first module initializes feature pyramids, while the latter two aggregate stereo and multi-scale contextual information into fine-grained features, respectively. ViTAStereo, which combines ViTAS with cost volume-based stereo matching back-end processes, achieves the top rank on the KITTI Stereo 2012 dataset and outperforms the second-best network StereoBase by approximately 7.9% in terms of the percentage of error pixels, with a tolerance of 3 pixels. Additional experiments across diverse scenarios further demonstrate its superior generalizability compared to all other state-of-the-art approaches. We believe this new paradigm will pave the way for the next generation of stereo matching networks.

Title: Spatial-Temporal Multi-level Association for Video Object Segmentation

• Authors: Deshui Miao, Xin Li, Zhenyu He, Huchuan Lu, Ming-Hsuan Yang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.06265
• Pdf URL: https://arxiv.org/pdf/2404.06265
• Copy Paste: [[2404.06265]] Spatial-Temporal Multi-level Association for Video Object Segmentation(https://arxiv.org/abs/2404.06265)
• Keywords: extraction, segmentation
• Abstract: Existing semi-supervised video object segmentation methods either focus on temporal feature matching or spatial-temporal feature modeling. However, they do not address the issues of sufficient target interaction and efficient parallel processing simultaneously, thereby constraining the learning of dynamic, target-aware features. To tackle these limitations, this paper proposes a spatial-temporal multi-level association framework, which jointly associates reference frame, test frame, and object features to achieve sufficient interaction and parallel target ID association with a spatial-temporal memory bank for efficient video object segmentation. Specifically, we construct a spatial-temporal multi-level feature association module to learn better target-aware features, which formulates feature extraction and interaction as the efficient operations of object self-attention, reference object enhancement, and test reference correlation. In addition, we propose a spatial-temporal memory to assist feature association and temporal ID assignment and correlation. We evaluate the proposed method by conducting extensive experiments on numerous video object segmentation datasets, including DAVIS 2016/2017 val, DAVIS 2017 test-dev, and YouTube-VOS 2018/2019 val. The favorable performance against the state-of-the-art methods demonstrates the effectiveness of our approach. All source code and trained models will be made publicly available.

Title: PGTNet: A Process Graph Transformer Network for Remaining Time Prediction of Business Process Instances

• Authors: Keyvan Amiri Elyasi, Han van der Aa, Heiner Stuckenschmidt
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06267
• Pdf URL: https://arxiv.org/pdf/2404.06267
• Copy Paste: [[2404.06267]] PGTNet: A Process Graph Transformer Network for Remaining Time Prediction of Business Process Instances(https://arxiv.org/abs/2404.06267)
• Keywords: transformer
• Abstract: We present PGTNet, an approach that transforms event logs into graph datasets and leverages graph-oriented data for training Process Graph Transformer Networks to predict the remaining time of business process instances. PGTNet consistently outperforms state-of-the-art deep learning approaches across a diverse range of 20 publicly available real-world event logs. Notably, our approach is most promising for highly complex processes, where existing deep learning approaches encounter difficulties stemming from their limited ability to learn control-flow relationships among process activities and capture long-range dependencies. PGTNet addresses these challenges, while also being able to consider multiple process perspectives during the learning process.

Title: Robust Confidence Intervals in Stereo Matching using Possibility Theory

• Authors: Roman Malinowski, Emmanuelle Sarrazin, Loïc Dumas, Emmanuel Dubois, Sébastien Destercke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06273
• Pdf URL: https://arxiv.org/pdf/2404.06273
• Copy Paste: [[2404.06273]] Robust Confidence Intervals in Stereo Matching using Possibility Theory(https://arxiv.org/abs/2404.06273)
• Keywords: robust
• Abstract: We propose a method for estimating disparity confidence intervals in stereo matching problems. Confidence intervals provide complementary information to usual confidence measures. To the best of our knowledge, this is the first method creating disparity confidence intervals based on the cost volume. This method relies on possibility distributions to interpret the epistemic uncertainty of the cost volume. Our method has the benefit of having a white-box nature, differing in this respect from current state-of-the-art deep neural networks approaches. The accuracy and size of confidence intervals are validated using the Middlebury stereo datasets as well as a dataset of satellite images. This contribution is freely available on GitHub.

Title: Learning Embeddings with Centroid Triplet Loss for Object Identification in Robotic Grasping

• Authors: Anas Gouda, Max Schwarz, Christopher Reining, Sven Behnke, Alice Kirchheim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06277
• Pdf URL: https://arxiv.org/pdf/2404.06277
• Copy Paste: [[2404.06277]] Learning Embeddings with Centroid Triplet Loss for Object Identification in Robotic Grasping(https://arxiv.org/abs/2404.06277)
• Keywords: segmentation
• Abstract: Foundation models are a strong trend in deep learning and computer vision. These models serve as a base for applications as they require minor or no further fine-tuning by developers to integrate into their applications. Foundation models for zero-shot object segmentation such as Segment Anything (SAM) output segmentation masks from images without any further object information. When they are followed in a pipeline by an object identification model, they can perform object detection without training. Here, we focus on training such an object identification model. A crucial practical aspect for an object identification model is to be flexible in input size. As object identification is an image retrieval problem, a suitable method should handle multi-query multi-gallery situations without constraining the number of input images (e.g. by having fixed-size aggregation layers). The key solution to train such a model is the centroid triplet loss (CTL), which aggregates image features to their centroids. CTL yields high accuracy, avoids misleading training signals and keeps the model input size flexible. In our experiments, we establish a new state of the art on the ArmBench object identification task, which shows general applicability of our model. We furthermore demonstrate an integrated unseen object detection pipeline on the challenging HOPE dataset, which requires fine-grained detection. There, our pipeline matches and surpasses related methods which have been trained on dataset-specific data.

Title: NoiseNCA: Noisy Seed Improves Spatio-Temporal Continuity of Neural Cellular Automata

• Authors: Ehsan Pajouheshgar, Yitao Xu, Sabine Süsstrunk
• Subjects: cs.CV, cs.AI, cs.GR, cs.MA
• Abstract URL: https://arxiv.org/abs/2404.06279
• Pdf URL: https://arxiv.org/pdf/2404.06279
• Copy Paste: [[2404.06279]] NoiseNCA: Noisy Seed Improves Spatio-Temporal Continuity of Neural Cellular Automata(https://arxiv.org/abs/2404.06279)
• Keywords: diffusion
• Abstract: Neural Cellular Automata (NCA) is a class of Cellular Automata where the update rule is parameterized by a neural network that can be trained using gradient descent. In this paper, we focus on NCA models used for texture synthesis, where the update rule is inspired by partial differential equations (PDEs) describing reaction-diffusion systems. To train the NCA model, the spatio-termporal domain is discretized, and Euler integration is used to numerically simulate the PDE. However, whether a trained NCA truly learns the continuous dynamic described by the corresponding PDE or merely overfits the discretization used in training remains an open question. We study NCA models at the limit where space-time discretization approaches continuity. We find that existing NCA models tend to overfit the training discretization, especially in the proximity of the initial condition, also called "seed". To address this, we propose a solution that utilizes uniform noise as the initial condition. We demonstrate the effectiveness of our approach in preserving the consistency of NCA dynamics across a wide range of spatio-temporal granularities. Our improved NCA model enables two new test-time interactions by allowing continuous control over the speed of pattern formation and the scale of the synthesized patterns. We demonstrate this new NCA feature in our interactive online demo. Our work reveals that NCA models can learn continuous dynamics and opens new venues for NCA research from a dynamical systems' perspective.

Title: Algorithms for Caching and MTS with reduced number of predictions

• Authors: Karim Abdel Sadek, Marek Elias
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2404.06280
• Pdf URL: https://arxiv.org/pdf/2404.06280
• Copy Paste: [[2404.06280]] Algorithms for Caching and MTS with reduced number of predictions(https://arxiv.org/abs/2404.06280)
• Keywords: robust
• Abstract: ML-augmented algorithms utilize predictions to achieve performance beyond their worst-case bounds. Producing these predictions might be a costly operation -- this motivated Im et al. '22 to introduce the study of algorithms which use predictions parsimoniously. We design parsimonious algorithms for caching and MTS with action predictions, proposed by Antoniadis et al. '20, focusing on the parameters of consistency (performance with perfect predictions) and smoothness (dependence of their performance on the prediction error). Our algorithm for caching is 1-consistent, robust, and its smoothness deteriorates with the decreasing number of available predictions. We propose an algorithm for general MTS whose consistency and smoothness both scale linearly with the decreasing number of predictions. Without the restriction on the number of available predictions, both algorithms match the earlier guarantees achieved by Antoniadis et al. '20.

Title: LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements

• Authors: Victoria Basmov, Yoav Goldberg, Reut Tsarfaty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06283
• Pdf URL: https://arxiv.org/pdf/2404.06283
• Copy Paste: [[2404.06283]] LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements(https://arxiv.org/abs/2404.06283)
• Keywords: large language model
• Abstract: The task of reading comprehension (RC), often implemented as context-based question answering (QA), provides a primary means to assess language models' natural language understanding (NLU) capabilities. Yet, when applied to large language models (LLMs) with extensive built-in world knowledge, this method can be deceptive. If the context aligns with the LLMs' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from LLMs' internal information. Conversely, using data that conflicts with the models' knowledge creates erroneous trends which distort the results. To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities. This task is entirely independent of the models' world knowledge, enabling us to evaluate LLMs' linguistic abilities without the interference of parametric knowledge. Testing ChatGPT, GPT-4, LLaMA 2 and Mixtral on such imaginary data, we uncover a class of linguistic phenomena posing a challenge to current LLMs, involving thinking in terms of alternative, hypothetical scenarios. While all the models handle simple affirmative and negative contexts with high accuracy, they are much more prone to error when dealing with modal and conditional contexts. Crucially, these phenomena also trigger the LLMs' vulnerability to knowledge-conflicts again. In particular, while some models prove virtually unaffected by knowledge conflicts in affirmative and negative contexts, when faced with more semantically involved modal and conditional environments, they often fail to separate the text from their internal knowledge.

Title: On adversarial training and the 1 Nearest Neighbor classifier

• Authors: Amir Hagai, Yair Weiss
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06313
• Pdf URL: https://arxiv.org/pdf/2404.06313
• Copy Paste: [[2404.06313]] On adversarial training and the 1 Nearest Neighbor classifier(https://arxiv.org/abs/2404.06313)
• Keywords: robust
• Abstract: The ability to fool deep learning classifiers with tiny perturbations of the input has lead to the development of adversarial training in which the loss with respect to adversarial examples is minimized in addition to the training examples. While adversarial training improves the robustness of the learned classifiers, the procedure is computationally expensive, sensitive to hyperparameters and may still leave the classifier vulnerable to other types of small perturbations. In this paper we analyze the adversarial robustness of the 1 Nearest Neighbor (1NN) classifier and compare its performance to adversarial training. We prove that under reasonable assumptions, the 1 NN classifier will be robust to {\em any} small image perturbation of the training images and will give high adversarial accuracy on test images as the number of training examples goes to infinity. In experiments with 45 different binary image classification problems taken from CIFAR10, we find that 1NN outperform TRADES (a powerful adversarial training algorithm) in terms of average adversarial accuracy. In additional experiments with 69 pretrained robust models for CIFAR10, we find that 1NN outperforms almost all of them in terms of robustness to perturbations that are only slightly different from those seen during training. Taken together, our results suggest that modern adversarial training methods still fall short of the robustness of the simple 1NN classifier. our code can be found at https://github.com/amirhagai/On-Adversarial-Training-And-The-1-Nearest-Neighbor-Classifier

Title: Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning

• Authors: Yanjie Li, Weijun Li, Lina Yu, Min Wu, Jingyi Liu, Wenqiang Li, Meilan Hao, Shu Wei, Yusong Deng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06330
• Pdf URL: https://arxiv.org/pdf/2404.06330
• Copy Paste: [[2404.06330]] Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning(https://arxiv.org/abs/2404.06330)
• Keywords: robust, transformer, generative
• Abstract: The mathematical formula is the human language to describe nature and is the essence of scientific research. Finding mathematical formulas from observational data is a major demand of scientific research and a major challenge of artificial intelligence. This area is called symbolic regression. Originally symbolic regression was often formulated as a combinatorial optimization problem and solved using GP or reinforcement learning algorithms. These two kinds of algorithms have strong noise robustness ability and good Versatility. However, inference time usually takes a long time, so the search efficiency is relatively low. Later, based on large-scale pre-training data proposed, such methods use a large number of synthetic data points and expression pairs to train a Generative Pre-Trained Transformer(GPT). Then this GPT can only need to perform one forward propagation to obtain the results, the advantage is that the inference speed is very fast. However, its performance is very dependent on the training data and performs poorly on data outside the training set, which leads to poor noise robustness and Versatility of such methods. So, can we combine the advantages of the above two categories of SR algorithms? In this paper, we propose \textbf{FormulaGPT}, which trains a GPT using massive sparse reward learning histories of reinforcement learning-based SR algorithms as training data. After training, the SR algorithm based on reinforcement learning is distilled into a Transformer. When new test data comes, FormulaGPT can directly generate a "reinforcement learning process" and automatically update the learning policy in context. Tested on more than ten datasets including SRBench, formulaGPT achieves the state-of-the-art performance in fitting ability compared with four baselines. In addition, it achieves satisfactory results in noise robustness, versatility, and inference efficiency.

Title: X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model

• Authors: Jan Held, Hani Itani, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, Marc Van Droogenbroeck
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06332
• Pdf URL: https://arxiv.org/pdf/2404.06332
• Copy Paste: [[2404.06332]] X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model(https://arxiv.org/abs/2404.06332)
• Keywords: explainability, large language model
• Abstract: The rapid advancement of artificial intelligence has led to significant improvements in automated decision-making. However, the increased performance of models often comes at the cost of explainability and transparency of their decision-making processes. In this paper, we investigate the capabilities of large language models to explain decisions, using football refereeing as a testing ground, given its decision complexity and subjectivity. We introduce the Explainable Video Assistant Referee System, X-VARS, a multi-modal large language model designed for understanding football videos from the point of view of a referee. X-VARS can perform a multitude of tasks, including video description, question answering, action recognition, and conducting meaningful conversations based on video content and in accordance with the Laws of the Game for football referees. We validate X-VARS on our novel dataset, SoccerNet-XFoul, which consists of more than 22k video-question-answer triplets annotated by over 70 experienced football referees. Our experiments and human study illustrate the impressive capabilities of X-VARS in interpreting complex football clips. Furthermore, we highlight the potential of X-VARS to reach human performance and support football referees in the future.

Title: Finding fake reviews in e-commerce platforms by using hybrid algorithms

• Authors: Mathivanan Periasamy, Rohith Mahadevan, Bagiya Lakshmi S, Raja CSP Raman, Hasan Kumar S, Jasper Jessiman
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06339
• Pdf URL: https://arxiv.org/pdf/2404.06339
• Copy Paste: [[2404.06339]] Finding fake reviews in e-commerce platforms by using hybrid algorithms(https://arxiv.org/abs/2404.06339)
• Keywords: robust
• Abstract: Sentiment analysis, a vital component in natural language processing, plays a crucial role in understanding the underlying emotions and opinions expressed in textual data. In this paper, we propose an innovative ensemble approach for sentiment analysis for finding fake reviews that amalgamate the predictive capabilities of Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Decision Tree classifiers. Our ensemble architecture strategically combines these diverse models to capitalize on their strengths while mitigating inherent weaknesses, thereby achieving superior accuracy and robustness in fake review prediction. By combining all the models of our classifiers, the predictive performance is boosted and it also fosters adaptability to varied linguistic patterns and nuances present in real-world datasets. The metrics accounted for on fake reviews demonstrate the efficacy and competitiveness of the proposed ensemble method against traditional single-model approaches. Our findings underscore the potential of ensemble techniques in advancing the state-of-the-art in finding fake reviews using hybrid algorithms, with implications for various applications in different social media and e-platforms to find the best reviews and neglect the fake ones, eliminating puffery and bluffs.

Title: CausalBench: A Comprehensive Benchmark for Causal Learning Capability of Large Language Models

• Authors: Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, Kay Chen Tan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06349
• Pdf URL: https://arxiv.org/pdf/2404.06349
• Copy Paste: [[2404.06349]] CausalBench: A Comprehensive Benchmark for Causal Learning Capability of Large Language Models(https://arxiv.org/abs/2404.06349)
• Keywords: large language model
• Abstract: Causality reveals fundamental principles behind data distributions in real-world scenarios, and the capability of large language models (LLMs) to understand causality directly impacts their efficacy across explaining outputs, adapting to new evidence, and generating counterfactuals. With the proliferation of LLMs, the evaluation of this capacity is increasingly garnering attention. However, the absence of a comprehensive benchmark has rendered existing evaluation studies being straightforward, undiversified, and homogeneous. To address these challenges, this paper proposes a comprehensive benchmark, namely CausalBench, to evaluate the causality understanding capabilities of LLMs. Originating from the causal research community, CausalBench encompasses three causal learning-related tasks, which facilitate a convenient comparison of LLMs' performance with classic causal learning algorithms. Meanwhile, causal networks of varying scales and densities are integrated in CausalBench, to explore the upper limits of LLMs' capabilities across task scenarios of varying difficulty. Notably, background knowledge and structured data are also incorporated into CausalBench to thoroughly unlock the underlying potential of LLMs for long-text comprehension and prior information utilization. Based on CausalBench, this paper evaluates nineteen leading LLMs and unveils insightful conclusions in diverse aspects. Firstly, we present the strengths and weaknesses of LLMs and quantitatively explore the upper limits of their capabilities across various scenarios. Meanwhile, we further discern the adaptability and abilities of LLMs to specific structural networks and complex chain of thought structures. Moreover, this paper quantitatively presents the differences across diverse information sources and uncovers the gap between LLMs' capabilities in causal understanding within textual contexts and numerical domains.

Title: DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning

• Authors: Senthil Yogamani, David Unger, Venkatraman Narayanan, Varun Ravi Kumar
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06352
• Pdf URL: https://arxiv.org/pdf/2404.06352
• Copy Paste: [[2404.06352]] DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning(https://arxiv.org/abs/2404.06352)
• Keywords: segmentation
• Abstract: Semantic segmentation is an effective way to perform scene understanding. Recently, segmentation in 3D Bird's Eye View (BEV) space has become popular as its directly used by drive policy. However, there is limited work on BEV segmentation for surround-view fisheye cameras, commonly used in commercial vehicles. As this task has no real-world public dataset and existing synthetic datasets do not handle amodal regions due to occlusion, we create a synthetic dataset using the Cognata simulator comprising diverse road types, weather, and lighting conditions. We generalize the BEV segmentation to work with any camera model; this is useful for mixing diverse cameras. We implement a baseline by applying cylindrical rectification on the fisheye images and using a standard LSS-based BEV segmentation model. We demonstrate that we can achieve better performance without undistortion, which has the adverse effects of increased runtime due to pre-processing, reduced field-of-view, and resampling artifacts. Further, we introduce a distortion-aware learnable BEV pooling strategy that is more effective for the fisheye cameras. We extend the model with an occlusion reasoning module, which is critical for estimating in BEV space. Qualitative performance of DaF-BEVSeg is showcased in the video at https://streamable.com/ge4v51.

Title: High Noise Scheduling is a Must

• Authors: Mahmut S. Gokmen, Cody Bumgardner, Jie Zhang, Ge Wang, Jin Chen
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06353
• Pdf URL: https://arxiv.org/pdf/2404.06353
• Copy Paste: [[2404.06353]] High Noise Scheduling is a Must(https://arxiv.org/abs/2404.06353)
• Keywords: robust, fair
• Abstract: Consistency models possess high capabilities for image generation, advancing sampling steps to a single step through their advanced techniques. Current advancements move one step forward consistency training techniques and eliminates the limitation of distillation training. Even though the proposed curriculum and noise scheduling in improved training techniques yield better results than basic consistency models, it lacks well balanced noise distribution and its consistency between curriculum. In this study, it is investigated the balance between high and low noise levels in noise distribution and offered polynomial noise distribution to maintain the stability. This proposed polynomial noise distribution is also supported with a predefined Karras noises to prevent unique noise levels arises with Karras noise generation algorithm. Furthermore, by elimination of learned noisy steps with a curriculum based on sinusoidal function increase the performance of the model in denoising. To make a fair comparison with the latest released consistency model training techniques, experiments are conducted with same hyper-parameters except curriculum and noise distribution. The models utilized during experiments are determined with low depth to prove the robustness of our proposed technique. The results show that the polynomial noise distribution outperforms the model trained with log-normal noise distribution, yielding a 33.54 FID score after 100,000 training steps with constant discretization steps. Additionally, the implementation of a sinusoidal-based curriculum enhances denoising performance, resulting in a FID score of 30.48.

Title: Policy-Guided Diffusion

• Authors: Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, Jakob Foerster
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06356
• Pdf URL: https://arxiv.org/pdf/2404.06356
• Copy Paste: [[2404.06356]] Policy-Guided Diffusion(https://arxiv.org/abs/2404.06356)
• Keywords: diffusion
• Abstract: In many real-world settings, agents must learn from an offline dataset gathered by some prior behavior policy. Such a setting naturally leads to distribution shift between the behavior policy and the target policy being trained - requiring policy conservatism to avoid instability and overestimation bias. Autoregressive world models offer a different solution to this by generating synthetic, on-policy experience. However, in practice, model rollouts must be severely truncated to avoid compounding error. As an alternative, we propose policy-guided diffusion. Our method uses diffusion models to generate entire trajectories under the behavior distribution, applying guidance from the target policy to move synthetic experience further on-policy. We show that policy-guided diffusion models a regularized form of the target distribution that balances action likelihood under both the target and behavior policies, leading to plausible trajectories with high target policy probability, while retaining a lower dynamics error than an offline world model baseline. Using synthetic experience from policy-guided diffusion as a drop-in substitute for real data, we demonstrate significant improvements in performance across a range of standard offline reinforcement learning algorithms and environments. Our approach provides an effective alternative to autoregressive offline world models, opening the door to the controllable generation of synthetic training data.

Title: Generalizable Sarcasm Detection Is Just Around The Corner, Of Course!

• Authors: Hyewon Jang, Diego Frassinelli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06357
• Pdf URL: https://arxiv.org/pdf/2404.06357
• Copy Paste: [[2404.06357]] Generalizable Sarcasm Detection Is Just Around The Corner, Of Course!(https://arxiv.org/abs/2404.06357)
• Keywords: robust
• Abstract: We tested the robustness of sarcasm detection models by examining their behavior when fine-tuned on four sarcasm datasets containing varying characteristics of sarcasm: label source (authors vs. third-party), domain (social media/online vs. offline conversations/dialogues), style (aggressive vs. humorous mocking). We tested their prediction performance on the same dataset (intra-dataset) and across different datasets (cross-dataset). For intra-dataset predictions, models consistently performed better when fine-tuned with third-party labels rather than with author labels. For cross-dataset predictions, most models failed to generalize well to the other datasets, implying that one type of dataset cannot represent all sorts of sarcasm with different styles and domains. Compared to the existing datasets, models fine-tuned on the new dataset we release in this work showed the highest generalizability to other datasets. With a manual inspection of the datasets and post-hoc analysis, we attributed the difficulty in generalization to the fact that sarcasm actually comes in different domains and styles. We argue that future sarcasm research should take the broad scope of sarcasm into account.

Title: Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation

• Authors: Sidra Aleem, Fangyijie Wang, Mayug Maniparambil, Eric Arazo, Julia Dietlmeier, Kathleen Curran, Noel E. O'Connor, Suzanne Little
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06362
• Pdf URL: https://arxiv.org/pdf/2404.06362
• Copy Paste: [[2404.06362]] Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation(https://arxiv.org/abs/2404.06362)
• Keywords: segmentation
• Abstract: The Segment Anything Model (SAM) and CLIP are remarkable vision foundation models (VFMs). SAM, a prompt driven segmentation model, excels in segmentation tasks across diverse domains, while CLIP is renowned for its zero shot recognition capabilities. However, their unified potential has not yet been explored in medical image segmentation. To adapt SAM to medical imaging, existing methods primarily rely on tuning strategies that require extensive data or prior prompts tailored to the specific task, making it particularly challenging when only a limited number of data samples are available. This work presents an in depth exploration of integrating SAM and CLIP into a unified framework for medical image segmentation. Specifically, we propose a simple unified framework, SaLIP, for organ segmentation. Initially, SAM is used for part based segmentation within the image, followed by CLIP to retrieve the mask corresponding to the region of interest (ROI) from the pool of SAM generated masks. Finally, SAM is prompted by the retrieved ROI to segment a specific organ. Thus, SaLIP is training and fine tuning free and does not rely on domain expertise or labeled data for prompt engineering. Our method shows substantial enhancements in zero shot segmentation, showcasing notable improvements in DICE scores across diverse segmentation tasks like brain (63.46%), lung (50.11%), and fetal head (30.82%), when compared to un prompted SAM. Code and text prompts will be available online.

Title: Dynamic Resolution Guidance for Facial Expression Recognition

• Authors: Jie Ou, Xu Li, Tianxiang Jiang, Yuanlun Xie
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.06365
• Pdf URL: https://arxiv.org/pdf/2404.06365
• Copy Paste: [[2404.06365]] Dynamic Resolution Guidance for Facial Expression Recognition(https://arxiv.org/abs/2404.06365)
• Keywords: robust
• Abstract: Facial expression recognition (FER) is vital for human-computer interaction and emotion analysis, yet recognizing expressions in low-resolution images remains challenging. This paper introduces a practical method called Dynamic Resolution Guidance for Facial Expression Recognition (DRGFER) to effectively recognize facial expressions in images with varying resolutions without compromising FER model accuracy. Our framework comprises two main components: the Resolution Recognition Network (RRN) and the Multi-Resolution Adaptation Facial Expression Recognition Network (MRAFER). The RRN determines image resolution, outputs a binary vector, and the MRAFER assigns images to suitable facial expression recognition networks based on resolution. We evaluated DRGFER on widely-used datasets RAFDB and FERPlus, demonstrating that our method retains optimal model performance at each resolution and outperforms alternative resolution approaches. The proposed framework exhibits robustness against resolution variations and facial expressions, offering a promising solution for real-world applications.

Title: ClinLinker: Medical Entity Linking of Clinical Concept Mentions in Spanish

• Authors: Fernando Gallego, Guillermo López-García, Luis Gasco-Sánchez, Martin Krallinger, Francisco J. Veredas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06367
• Pdf URL: https://arxiv.org/pdf/2404.06367
• Copy Paste: [[2404.06367]] ClinLinker: Medical Entity Linking of Clinical Concept Mentions in Spanish(https://arxiv.org/abs/2404.06367)
• Keywords: extraction
• Abstract: Advances in natural language processing techniques, such as named entity recognition and normalization to widely used standardized terminologies like UMLS or SNOMED-CT, along with the digitalization of electronic health records, have significantly advanced clinical text analysis. This study presents ClinLinker, a novel approach employing a two-phase pipeline for medical entity linking that leverages the potential of in-domain adapted language models for biomedical text mining: initial candidate retrieval using a SapBERT-based bi-encoder and subsequent re-ranking with a cross-encoder, trained by following a contrastive-learning strategy to be tailored to medical concepts in Spanish. This methodology, focused initially on content in Spanish, substantially outperforming multilingual language models designed for the same purpose. This is true even for complex scenarios involving heterogeneous medical terminologies and being trained on a subset of the original data. Our results, evaluated using top-k accuracy at 25 and other top-k metrics, demonstrate our approach's performance on two distinct clinical entity linking Gold Standard corpora, DisTEMIST (diseases) and MedProcNER (clinical procedures), outperforming previous benchmarks by 40 points in DisTEMIST and 43 points in MedProcNER, both normalized to SNOMED-CT codes. These findings highlight our approach's ability to address language-specific nuances and set a new benchmark in entity linking, offering a potent tool for enhancing the utility of digital medical records. The resulting system is of practical value, both for large scale automatic generation of structured data derived from clinical records, as well as for exhaustive extraction and harmonization of predefined clinical variables of interest.

Title: VISION2UI: A Real-World Dataset with Layout for Code Generation from UI Designs

• Authors: Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Shaoling Dong, Xing Zhou, Wenbin Jiang
• Subjects: cs.CV, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.06369
• Pdf URL: https://arxiv.org/pdf/2404.06369
• Copy Paste: [[2404.06369]] VISION2UI: A Real-World Dataset with Layout for Code Generation from UI Designs(https://arxiv.org/abs/2404.06369)
• Keywords: large language model
• Abstract: Automatically generating UI code from webpage design visions can significantly alleviate the burden of developers, enabling beginner developers or designers to directly generate Web pages from design diagrams. Currently, prior research has accomplished the objective of generating UI code from rudimentary design visions or sketches through designing deep neural networks. Inspired by the groundbreaking advancements achieved by Multimodal Large Language Models (MLLMs), the automatic generation of UI code from high-fidelity design images is now emerging as a viable possibility. Nevertheless, our investigation reveals that existing MLLMs are hampered by the scarcity of authentic, high-quality, and large-scale datasets, leading to unsatisfactory performance in automated UI code generation. To mitigate this gap, we present a novel dataset, termed VISION2UI, extracted from real-world scenarios, augmented with comprehensive layout information, tailored specifically for finetuning MLLMs in UI code generation. Specifically, this dataset is derived through a series of operations, encompassing collecting, cleaning, and filtering of the open-source Common Crawl dataset. In order to uphold its quality, a neural scorer trained on labeled samples is utilized to refine the data, retaining higher-quality instances. Ultimately, this process yields a dataset comprising 2,000 (Much more is coming soon) parallel samples encompassing design visions and UI code. The dataset is available at https://huggingface.co/datasets/xcodemind/vision2ui.

Title: Latent Distance Guided Alignment Training for Large Language Models

• Authors: Haotian Luo, Wenhao Zheng, Huaxiu Yao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06390
• Pdf URL: https://arxiv.org/pdf/2404.06390
• Copy Paste: [[2404.06390]] Latent Distance Guided Alignment Training for Large Language Models(https://arxiv.org/abs/2404.06390)
• Keywords: large language model
• Abstract: Ensuring alignment with human preferences is a crucial characteristic of large language models (LLMs). Presently, the primary alignment methods, RLHF and DPO, require extensive human annotation, which is expensive despite their efficacy. The significant expenses associated with current alignment techniques motivate researchers to investigate the development of annotation-free alignment training methods. In pursuit of improved alignment without relying on external annotation, we introduce Latent Distance Guided Alignment Training (LD-Align). This approach seeks to align the model with a high-quality supervised fine-tune dataset using guidance from a latent space. The latent space is generated through sample reconstruction, akin to auto-encoding. Consequently, we utilize the distance between sample pairs in the latent space to guide DPO-based alignment training. Extensive experimentation and evaluation show the efficacy of our proposed method in achieving notable alignment.

Title: Event Extraction in Basque: Typologically motivated Cross-Lingual Transfer-Learning Analysis

• Authors: Mikel Zubillaga, Oscar Sainz, Ainara Estarrona, Oier Lopez de Lacalle, Eneko Agirre
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06392
• Pdf URL: https://arxiv.org/pdf/2404.06392
• Copy Paste: [[2404.06392]] Event Extraction in Basque: Typologically motivated Cross-Lingual Transfer-Learning Analysis(https://arxiv.org/abs/2404.06392)
• Keywords: extraction
• Abstract: Cross-lingual transfer-learning is widely used in Event Extraction for low-resource languages and involves a Multilingual Language Model that is trained in a source language and applied to the target language. This paper studies whether the typological similarity between source and target languages impacts the performance of cross-lingual transfer, an under-explored topic. We first focus on Basque as the target language, which is an ideal target language because it is typologically different from surrounding languages. Our experiments on three Event Extraction tasks show that the shared linguistic characteristic between source and target languages does have an impact on transfer quality. Further analysis of 72 language pairs reveals that for tasks that involve token classification such as entity and event trigger identification, common writing script and morphological features produce higher quality cross-lingual transfer. In contrast, for tasks involving structural prediction like argument extraction, common word order is the most relevant feature. In addition, we show that when increasing the training size, not all the languages scale in the same way in the cross-lingual setting. To perform the experiments we introduce EusIE, an event extraction dataset for Basque, which follows the Multilingual Event Extraction dataset (MEE). The dataset and code are publicly available.

Title: MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

• Authors: Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06395
• Pdf URL: https://arxiv.org/pdf/2404.06395
• Copy Paste: [[2404.06395]] MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies(https://arxiv.org/abs/2404.06395)
• Keywords: large language model
• Abstract: The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly at https://github.com/OpenBMB/MiniCPM .

Title: Take a Look at it! Rethinking How to Evaluate Language Model Jailbreak

• Authors: Hongyu Cai, Arjun Arunasalam, Leo Y. Lin, Antonio Bianchi, Z. Berkay Celik
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06407
• Pdf URL: https://arxiv.org/pdf/2404.06407
• Copy Paste: [[2404.06407]] Take a Look at it! Rethinking How to Evaluate Language Model Jailbreak(https://arxiv.org/abs/2404.06407)
• Keywords: large language model
• Abstract: Large language models (LLMs) have become increasingly integrated with various applications. To ensure that LLMs do not generate unsafe responses, they are aligned with safeguards that specify what content is restricted. However, such alignment can be bypassed to produce prohibited content using a technique commonly referred to as jailbreak. Different systems have been proposed to perform the jailbreak automatically. These systems rely on evaluation methods to determine whether a jailbreak attempt is successful. However, our analysis reveals that current jailbreak evaluation methods have two limitations. (1) Their objectives lack clarity and do not align with the goal of identifying unsafe responses. (2) They oversimplify the jailbreak result as a binary outcome, successful or not. In this paper, we propose three metrics, safeguard violation, informativeness, and relative truthfulness, to evaluate language model jailbreak. Additionally, we demonstrate how these metrics correlate with the goal of different malicious actors. To compute these metrics, we introduce a multifaceted approach that extends the natural language generation evaluation method after preprocessing the response. We evaluate our metrics on a benchmark dataset produced from three malicious intent datasets and three jailbreak systems. The benchmark dataset is labeled by three annotators. We compare our multifaceted approach with three existing jailbreak evaluation methods. Experiments demonstrate that our multifaceted evaluation outperforms existing methods, with F1 scores improving on average by 17% compared to existing baselines. Our findings motivate the need to move away from the binary view of the jailbreak problem and incorporate a more comprehensive evaluation to ensure the safety of the language model.

Title: Studying the Impact of Latent Representations in Implicit Neural Networks for Scientific Continuous Field Reconstruction

• Authors: Wei Xu, Derek Freeman DeSantis, Xihaier Luo, Avish Parmar, Klaus Tan, Balu Nadiga, Yihui Ren, Shinjae Yoo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06418
• Pdf URL: https://arxiv.org/pdf/2404.06418
• Copy Paste: [[2404.06418]] Studying the Impact of Latent Representations in Implicit Neural Networks for Scientific Continuous Field Reconstruction(https://arxiv.org/abs/2404.06418)
• Keywords: explainability
• Abstract: Learning a continuous and reliable representation of physical fields from sparse sampling is challenging and it affects diverse scientific disciplines. In a recent work, we present a novel model called MMGN (Multiplicative and Modulated Gabor Network) with implicit neural networks. In this work, we design additional studies leveraging explainability methods to complement the previous experiments and further enhance the understanding of latent representations generated by the model. The adopted methods are general enough to be leveraged for any latent space inspection. Preliminary results demonstrate the contextual information incorporated in the latent representations and their impact on the model performance. As a work in progress, we will continue to verify our findings and develop novel explainability approaches.

Title: ZeST: Zero-Shot Material Transfer from a Single Image

• Authors: Ta-Ying Cheng, Prafull Sharma, Andrew Markham, Niki Trigoni, Varun Jampani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06425
• Pdf URL: https://arxiv.org/pdf/2404.06425
• Copy Paste: [[2404.06425]] ZeST: Zero-Shot Material Transfer from a Single Image(https://arxiv.org/abs/2404.06425)
• Keywords: robust, diffusion
• Abstract: We propose ZeST, a method for zero-shot material transfer to an object in the input image given a material exemplar image. ZeST leverages existing diffusion adapters to extract implicit material representation from the exemplar image. This representation is used to transfer the material using pre-trained inpainting diffusion model on the object in the input image using depth estimates as geometry cue and grayscale object shading as illumination cues. The method works on real images without any training resulting a zero-shot approach. Both qualitative and quantitative results on real and synthetic datasets demonstrate that ZeST outputs photorealistic images with transferred materials. We also show the application of ZeST to perform multiple edits and robust material assignment under different illuminations. Project Page: https://ttchengab.github.io/zest

Title: Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion

• Authors: Fan Yang, Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang, Jiashi Feng, Guosheng Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06429
• Pdf URL: https://arxiv.org/pdf/2404.06429
• Copy Paste: [[2404.06429]] Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion(https://arxiv.org/abs/2404.06429)
• Keywords: robust, diffusion, generative
• Abstract: Benefiting from the rapid development of 2D diffusion models, 3D content creation has made significant progress recently. One promising solution involves the fine-tuning of pre-trained 2D diffusion models to harness their capacity for producing multi-view images, which are then lifted into accurate 3D models via methods like fast-NeRFs or large reconstruction models. However, as inconsistency still exists and limited generated resolution, the generation results of such methods still lack intricate textures and complex geometries. To solve this problem, we propose Magic-Boost, a multi-view conditioned diffusion model that significantly refines coarse generative results through a brief period of SDS optimization ($\sim15$min). Compared to the previous text or single image based diffusion models, Magic-Boost exhibits a robust capability to generate images with high consistency from pseudo synthesized multi-view images. It provides precise SDS guidance that well aligns with the identity of the input images, enriching the local detail in both geometry and texture of the initial generative results. Extensive experiments show Magic-Boost greatly enhances the coarse inputs and generates high-quality 3D assets with rich geometric and textural details. (Project Page: https://magic-research.github.io/magic-boost/)

Title: pfl-research: simulation framework for accelerating research in Private Federated Learning

• Authors: Filip Granqvist, Congzheng Song, Áine Cahill, Rogier van Dalen, Martin Pelikan, Yi Sheng Chan, Xiaojun Feng, Natarajan Krishnaswami, Vojta Jina, Mona Chitnis
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06430
• Pdf URL: https://arxiv.org/pdf/2404.06430
• Copy Paste: [[2404.06430]] pfl-research: simulation framework for accelerating research in Private Federated Learning(https://arxiv.org/abs/2404.06430)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is an emerging machine learning (ML) training paradigm where clients own their data and collaborate to train a global model, without revealing any data to the server and other participants. Researchers commonly perform experiments in a simulation environment to quickly iterate on ideas. However, existing open-source tools do not offer the efficiency required to simulate FL on larger and more realistic FL datasets. We introduce pfl-research, a fast, modular, and easy-to-use Python framework for simulating FL. It supports TensorFlow, PyTorch, and non-neural network models, and is tightly integrated with state-of-the-art privacy algorithms. We study the speed of open-source FL frameworks and show that pfl-research is 7-72$\times$ faster than alternative open-source frameworks on common cross-device setups. Such speedup will significantly boost the productivity of the FL research community and enable testing hypotheses on realistic FL datasets that were previously too resource intensive. We release a suite of benchmarks that evaluates an algorithm's overall performance on a diverse set of realistic scenarios. The code is available on GitHub at https://github.com/apple/pfl-research.

Title: Software-based Security Framework for Edge and Mobile IoT

• Authors: José Cecílio, Alan Oliveira de Sá, André Souto
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2404.06435
• Pdf URL: https://arxiv.org/pdf/2404.06435
• Copy Paste: [[2404.06435]] Software-based Security Framework for Edge and Mobile IoT(https://arxiv.org/abs/2404.06435)
• Keywords: secure, security, robust
• Abstract: With the proliferation of Internet of Things (IoT) devices, ensuring secure communications has become imperative. Due to their low cost and embedded nature, many of these devices operate with computational and energy constraints, neglecting the potential security vulnerabilities that they may bring. This work-in-progress is focused on designing secure communication among remote servers and embedded IoT devices to balance security robustness and energy efficiency. The proposed approach uses lightweight cryptography, optimizing device performance and security without overburdening their limited resources. Our architecture stands out for integrating Edge servers and a central Name Server, allowing secure and decentralized authentication and efficient connection transitions between different Edge servers. This architecture enhances the scalability of the IoT network and reduces the load on each server, distributing the responsibility for authentication and key management.

Title: Seasonal Fire Prediction using Spatio-Temporal Deep Neural Networks

• Authors: Dimitrios Michail, Lefki-Ioanna Panagiotou, Charalampos Davalas, Ioannis Prapas, Spyros Kondylatos, Nikolaos Ioannis Bountos, Ioannis Papoutsis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06437
• Pdf URL: https://arxiv.org/pdf/2404.06437
• Copy Paste: [[2404.06437]] Seasonal Fire Prediction using Spatio-Temporal Deep Neural Networks(https://arxiv.org/abs/2404.06437)
• Keywords: robust
• Abstract: With climate change expected to exacerbate fire weather conditions, the accurate anticipation of wildfires on a global scale becomes increasingly crucial for disaster mitigation. In this study, we utilize SeasFire, a comprehensive global wildfire dataset with climate, vegetation, oceanic indices, and human-related variables, to enable seasonal wildfire forecasting with machine learning. For the predictive analysis, we train deep learning models with different architectures that capture the spatio-temporal context leading to wildfires. Our investigation focuses on assessing the effectiveness of these models in predicting the presence of burned areas at varying forecasting time horizons globally, extending up to six months into the future, and on how different spatial or/and temporal context affects the performance of the models. Our findings demonstrate the great potential of deep learning models in seasonal fire forecasting; longer input time-series leads to more robust predictions across varying forecasting horizons, while integrating spatial information to capture wildfire spatio-temporal dynamics boosts performance. Finally, our results hint that in order to enhance performance at longer forecasting horizons, a larger receptive field spatially needs to be considered.

Title: QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding

• Authors: Yash Mehan, Kumaraditya Gupta, Rohit Jayanti, Anirudh Govil, Sourav Garg, Madhava Krishna
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.06442
• Pdf URL: https://arxiv.org/pdf/2404.06442
• Copy Paste: [[2404.06442]] QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding(https://arxiv.org/abs/2404.06442)
• Keywords: extraction, transformer, segmentation
• Abstract: Understanding the structural organisation of 3D indoor scenes in terms of rooms is often accomplished via floorplan extraction. Robotic tasks such as planning and navigation require a semantic understanding of the scene as well. This is typically achieved via object-level semantic segmentation. However, such methods struggle to segment out topological regions like "kitchen" in the scene. In this work, we introduce a two-step pipeline. First, we extract a topological map, i.e., floorplan of the indoor scene using a novel multi-channel occupancy representation. Then, we generate CLIP-aligned features and semantic labels for every room instance based on the objects it contains using a self-attention transformer. Our language-topology alignment supports natural language querying, e.g., a "place to cook" locates the "kitchen". We outperform the current state-of-the-art on room segmentation by ~20% and room classification by ~12%. Our detailed qualitative analysis and ablation studies provide insights into the problem of joint structural and semantic 3D scene understanding.

Title: Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models

• Authors: Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, Yuguang Fang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06448
• Pdf URL: https://arxiv.org/pdf/2404.06448
• Copy Paste: [[2404.06448]] Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models(https://arxiv.org/abs/2404.06448)
• Keywords: privacy, federate, generative, large language model
• Abstract: Recently, there has been a surge in the development of advanced intelligent generative content (AIGC), especially large language models (LLMs). However, for many downstream tasks, it is necessary to fine-tune LLMs using private data. While federated learning offers a promising privacy-preserving solution to LLM fine-tuning, the substantial size of an LLM, combined with high computational and communication demands, makes it hard to apply to downstream tasks. More importantly, private edge servers often possess varying computing and network resources in real-world scenarios, introducing additional complexities to LLM fine-tuning. To tackle these problems, we design and implement an automated federated pipeline, named FedPipe, to fine-tune LLMs with minimal training cost but without adding any inference latency. FedPipe firstly identifies the weights to be fine-tuned based on their contributions to the LLM training. It then configures a low-rank adapter for each selected weight to train local low-rank adapters on an edge server, and aggregate local adapters of all edge servers to fine-tune the whole LLM. Finally, it appropriately quantizes the parameters of LLM to reduce memory space according to the requirements of edge servers. Extensive experiments demonstrate that FedPipe expedites the model training and achieves higher accuracy than state-of-the-art benchmarks.

Title: PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits

• Authors: Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojciech Samek, Sebastian Lapuschkin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06453
• Pdf URL: https://arxiv.org/pdf/2404.06453
• Copy Paste: [[2404.06453]] PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits(https://arxiv.org/abs/2404.06453)
• Keywords: interpretability
• Abstract: The field of mechanistic interpretability aims to study the role of individual neurons in Deep Neural Networks. Single neurons, however, have the capability to act polysemantically and encode for multiple (unrelated) features, which renders their interpretation difficult. We present a method for disentangling polysemanticity of any Deep Neural Network by decomposing a polysemantic neuron into multiple monosemantic "virtual" neurons. This is achieved by identifying the relevant sub-graph ("circuit") for each "pure" feature. We demonstrate how our approach allows us to find and disentangle various polysemantic units of ResNet models trained on ImageNet. While evaluating feature visualizations using CLIP, our method effectively disentangles representations, improving upon methods based on neuron activations. Our code is available at https://github.com/maxdreyer/PURE.

Title: Learning State-Invariant Representations of Objects from Image Collections with State, Pose, and Viewpoint Changes

• Authors: Rohan Sarkar, Avinash Kak
• Subjects: cs.CV, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.06470
• Pdf URL: https://arxiv.org/pdf/2404.06470
• Copy Paste: [[2404.06470]] Learning State-Invariant Representations of Objects from Image Collections with State, Pose, and Viewpoint Changes(https://arxiv.org/abs/2404.06470)
• Keywords: robust
• Abstract: We add one more invariance - state invariance - to the more commonly used other invariances for learning object representations for recognition and retrieval. By state invariance, we mean robust with respect to changes in the structural form of the object, such as when an umbrella is folded, or when an item of clothing is tossed on the floor. Since humans generally have no difficulty in recognizing objects despite such state changes, we are naturally faced with the question of whether it is possible to devise a neural architecture with similar abilities. To that end, we present a novel dataset, ObjectsWithStateChange, that captures state and pose variations in the object images recorded from arbitrary viewpoints. We believe that this dataset will facilitate research in fine-grained object recognition and retrieval of objects that are capable of state changes. The goal of such research would be to train models capable of generating object embeddings that remain invariant to state changes while also staying invariant to transformations induced by changes in viewpoint, pose, illumination, etc. To demonstrate the usefulness of the ObjectsWithStateChange dataset, we also propose a curriculum learning strategy that uses the similarity relationships in the learned embedding space after each epoch to guide the training process. The model learns discriminative features by comparing visually similar objects within and across different categories, encouraging it to differentiate between objects that may be challenging to distinguish due to changes in their state. We believe that this strategy enhances the model's ability to capture discriminative features for fine-grained tasks that may involve objects with state changes, leading to performance improvements on object-level tasks not only on our new dataset, but also on two other challenging multi-view datasets such as ModelNet40 and ObjectPI.

Title: Text-Based Reasoning About Vector Graphics

• Authors: Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06479
• Pdf URL: https://arxiv.org/pdf/2404.06479
• Copy Paste: [[2404.06479]] Text-Based Reasoning About Vector Graphics(https://arxiv.org/abs/2404.06479)
• Keywords: interpretability
• Abstract: While large multimodal models excel in broad vision-language benchmarks, they often struggle with tasks requiring precise perception of low-level visual details, such as comparing line lengths or solving simple mazes. In particular, this failure mode persists in question-answering tasks about vector graphics -- images composed purely of 2D objects and shapes. To address this challenge, we propose the Visually Descriptive Language Model (VDLM), which performs text-based reasoning about vector graphics. VDLM leverages Scalable Vector Graphics (SVG) for a more precise visual description and first uses an off-the-shelf raster-to-SVG algorithm for encoding. Since existing language models cannot understand raw SVGs in a zero-shot setting, VDLM then bridges SVG with pretrained language models through a newly introduced intermediate symbolic representation, Primal Visual Description (PVD), comprising primitive attributes (e.g., shape, position, measurement) with their corresponding predicted values. PVD is task-agnostic and represents visual primitives that are universal across all vector graphics. It can be learned with procedurally generated (SVG, PVD) pairs and also enables the direct use of LLMs for generalization to complex reasoning tasks. By casting an image to a text-based representation, we can leverage the power of language models to learn alignment from SVG to visual primitives and generalize to unseen question-answering tasks. Empirical results show that VDLM achieves stronger zero-shot performance compared to state-of-the-art LMMs, such as GPT-4V, in various low-level multimodal perception and reasoning tasks on vector graphics. We additionally present extensive analyses on VDLM's performance, demonstrating that our framework offers better interpretability due to its disentangled perception and reasoning processes. Project page: https://mikewangwzhl.github.io/VDLM/

Title: Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks

• Authors: Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, Kai Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06480
• Pdf URL: https://arxiv.org/pdf/2404.06480
• Copy Paste: [[2404.06480]] Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks(https://arxiv.org/abs/2404.06480)
• Keywords: large language model
• Abstract: Recently, the large language model (LLM) community has shown increasing interest in enhancing LLMs' capability to handle extremely long documents. As various long-text techniques and model architectures emerge, the precise and detailed evaluation of models' long-text capabilities has become increasingly important. Existing long-text evaluation benchmarks, such as L-Eval and LongBench, construct long-text test sets based on open-source datasets, focusing mainly on QA and summarization tasks. These datasets include test samples of varying lengths (from 2k to 32k+) entangled together, making it challenging to assess model capabilities across different length ranges. Moreover, they do not cover the ultralong settings (100k+ tokens) that the latest LLMs claim to achieve. In this paper, we introduce Ada-LEval, a length-adaptable benchmark for evaluating the long-context understanding of LLMs. Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable a more reliable evaluation of LLMs' long context capabilities. These benchmarks support intricate manipulation of the length of test cases, and can easily produce text samples up to 128k tokens. We evaluate 4 state-of-the-art closed-source API models and 6 open-source models with Ada-LEval. The evaluation results demonstrate the limitations of current LLMs, especially in ultra-long-context settings. Our code is available at https://github.com/open-compass/Ada-LEval.

Title: RhythmMamba: Fast Remote Physiological Measurement with Arbitrary Length Videos

• Authors: Bochao Zou, Zizheng Guo, Xiaocheng Hu, Huimin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.06483
• Pdf URL: https://arxiv.org/pdf/2404.06483
• Copy Paste: [[2404.06483]] RhythmMamba: Fast Remote Physiological Measurement with Arbitrary Length Videos(https://arxiv.org/abs/2404.06483)
• Keywords: robust
• Abstract: Remote photoplethysmography (rPPG) is a non-contact method for detecting physiological signals from facial videos, holding great potential in various applications such as healthcare, affective computing, and anti-spoofing. Existing deep learning methods struggle to address two core issues of rPPG simultaneously: extracting weak rPPG signals from video segments with large spatiotemporal redundancy and understanding the periodic patterns of rPPG among long contexts. This represents a trade-off between computational complexity and the ability to capture long-range dependencies, posing a challenge for rPPG that is suitable for deployment on mobile devices. Based on the in-depth exploration of Mamba's comprehension of spatial and temporal information, this paper introduces RhythmMamba, an end-to-end Mamba-based method that employs multi-temporal Mamba to constrain both periodic patterns and short-term trends, coupled with frequency domain feed-forward to enable Mamba to robustly understand the quasi-periodic patterns of rPPG. Extensive experiments show that RhythmMamba achieves state-of-the-art performance with reduced parameters and lower computational complexity. The proposed RhythmMamba can be applied to video segments of any length without performance degradation. The codes are available at https://github.com/zizheng-guo/RhythmMamba.

Title: Pitfalls of Conversational LLMs on News Debiasing

• Authors: Ipek Baris Schlicht, Defne Altiok, Maryanne Taouk, Lucie Flek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.06488
• Pdf URL: https://arxiv.org/pdf/2404.06488
• Copy Paste: [[2404.06488]] Pitfalls of Conversational LLMs on News Debiasing(https://arxiv.org/abs/2404.06488)
• Keywords: large language model
• Abstract: This paper addresses debiasing in news editing and evaluates the effectiveness of conversational Large Language Models in this task. We designed an evaluation checklist tailored to news editors' perspectives, obtained generated texts from three popular conversational models using a subset of a publicly available dataset in media bias, and evaluated the texts according to the designed checklist. Furthermore, we examined the models as evaluator for checking the quality of debiased model outputs. Our findings indicate that none of the LLMs are perfect in debiasing. Notably, some models, including ChatGPT, introduced unnecessary changes that may impact the author's style and create misinformation. Lastly, we show that the models do not perform as proficiently as domain experts in evaluating the quality of debiased outputs.

Title: Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?

• Authors: Nathan Brake, Thomas Schaaf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06503
• Pdf URL: https://arxiv.org/pdf/2404.06503
• Copy Paste: [[2404.06503]] Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?(https://arxiv.org/abs/2404.06503)
• Keywords: transformer
• Abstract: Following an interaction with a patient, physicians are responsible for the submission of clinical documentation, often organized as a SOAP note. A clinical note is not simply a summary of the conversation but requires the use of appropriate medical terminology. The relevant information can then be extracted and organized according to the structure of the SOAP note. In this paper we analyze two different approaches to generate the different sections of a SOAP note based on the audio recording of the conversation, and specifically examine them in terms of note consistency. The first approach generates the sections independently, while the second method generates them all together. In this work we make use of PEGASUS-X Transformer models and observe that both methods lead to similar ROUGE values (less than 1% difference) and have no difference in terms of the Factuality metric. We perform a human evaluation to measure aspects of consistency and demonstrate that LLMs like Llama2 can be used to perform the same tasks with roughly the same agreement as the human annotators. Between the Llama2 analysis and the human reviewers we observe a Cohen Kappa inter-rater reliability of 0.79, 1.00, and 0.32 for consistency of age, gender, and body part injury, respectively. With this we demonstrate the usefulness of leveraging an LLM to measure quality indicators that can be identified by humans but are not currently captured by automatic metrics. This allows scaling evaluation to larger data sets, and we find that clinical note consistency improves by generating each new section conditioned on the output of all previously generated sections.

Title: InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

• Authors: Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.06512
• Pdf URL: https://arxiv.org/pdf/2404.06512
• Copy Paste: [[2404.06512]] InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD(https://arxiv.org/abs/2404.06512)
• Keywords: transformer
• Abstract: The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.