2024-08-22

Title: StructuredRAG: JSON Response Formatting with Large Language Models

• Authors: Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, Bob van Luijt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11061
• Pdf URL: https://arxiv.org/pdf/2408.11061
• Copy Paste: [[2408.11061]] StructuredRAG: JSON Response Formatting with Large Language Models(https://arxiv.org/abs/2408.11061)
• Keywords: large language model
• Abstract: The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON, is crucial for their use in Compound AI Systems. However, evaluating and improving this capability remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions. We evaluate two state-of-the-art LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting. Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro. We observe that task complexity significantly influences performance, with tasks involving lists or composite object outputs proving more challenging. Our findings highlight the need for further research into improving the reliability and consistency of structured output generation in LLMs. We have open-sourced our experimental code and results at this http URL.

Title: Interactive-T2S: Multi-Turn Interactions for Text-to-SQL with Large Language Models

• Authors: Guanming Xiong, Junwei Bao, Hongfei Jiang, Yang Song, Wen Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11062
• Pdf URL: https://arxiv.org/pdf/2408.11062
• Copy Paste: [[2408.11062]] Interactive-T2S: Multi-Turn Interactions for Text-to-SQL with Large Language Models(https://arxiv.org/abs/2408.11062)
• Keywords: robust, large language model
• Abstract: This study explores text-to-SQL parsing by leveraging the powerful reasoning capabilities of large language models (LLMs). Despite recent advancements, existing LLM-based methods have not adequately addressed scalability, leading to inefficiencies when processing wide tables. Furthermore, current interaction-based approaches either lack a step-by-step, interpretable SQL generation process or fail to provide an efficient and universally applicable interaction design. To address these challenges, we introduce Interactive-T2S, a framework that generates SQL queries through direct interactions with databases. This framework includes four general tools that facilitate proactive and efficient information retrieval by the LLM. Additionally, we have developed detailed exemplars to demonstrate the step-wise reasoning processes within our framework. Our experiments on the BIRD-Dev dataset, employing a setting without oracle knowledge, reveal that our method achieves state-of-the-art results with only two exemplars, underscoring the effectiveness and robustness of our framework.

Title: Tabular Transfer Learning via Prompting LLMs

• Authors: Jaehyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, Kyu Hwan Oh, Jinwoo Shin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11063
• Pdf URL: https://arxiv.org/pdf/2408.11063
• Copy Paste: [[2408.11063]] Tabular Transfer Learning via Prompting LLMs(https://arxiv.org/abs/2408.11063)
• Keywords: large language model
• Abstract: Learning with a limited number of labeled data is a central problem in real-world applications of machine learning, as it is often expensive to obtain annotations. To deal with the scarcity of labeled data, transfer learning is a conventional approach; it suggests to learn a transferable knowledge by training a neural network from multiple other sources. In this paper, we investigate transfer learning of tabular tasks, which has been less studied and successful in the literature, compared to other domains, e.g., vision and language. This is because tables are inherently heterogeneous, i.e., they contain different columns and feature spaces, making transfer learning difficult. On the other hand, recent advances in natural language processing suggest that the label scarcity issue can be mitigated by utilizing in-context learning capability of large language models (LLMs). Inspired by this and the fact that LLMs can also process tables within a unified language space, we ask whether LLMs can be effective for tabular transfer learning, in particular, under the scenarios where the source and target datasets are of different format. As a positive answer, we propose a novel tabular transfer learning framework, coined Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with LLMs. Specifically, P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts. Experimental results demonstrate that P2T outperforms previous methods on various tabular learning benchmarks, showing good promise for the important, yet underexplored tabular transfer learning problem. Code is available at this https URL.

Title: DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization

• Authors: Pucheng Dang, Xing Hu, Dong Li, Rui Zhang, Qi Guo, Kaidi Xu
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11071
• Pdf URL: https://arxiv.org/pdf/2408.11071
• Copy Paste: [[2408.11071]] DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization(https://arxiv.org/abs/2408.11071)
• Keywords: attack, diffusion, generative
• Abstract: Current text-to-image (T2I) synthesis diffusion models raise misuse concerns, particularly in creating prohibited or not-safe-for-work (NSFW) images. To address this, various safety mechanisms and red teaming attack methods are proposed to enhance or expose the T2I model's capability to generate unsuitable content. However, many red teaming attack methods assume knowledge of the text encoders, limiting their practical usage. In this work, we rethink the case of \textit{purely black-box} attacks without prior knowledge of the T2l model. To overcome the unavailability of gradients and the inability to optimize attacks within a discrete prompt space, we propose DiffZOO which applies Zeroth Order Optimization to procure gradient approximations and harnesses both C-PRV and D-PRV to enhance attack prompts within the discrete prompt domain. We evaluated our method across multiple safety mechanisms of the T2I diffusion model and online servers. Experiments on multiple state-of-the-art safety mechanisms show that DiffZOO attains an 8.5% higher average attack success rate than previous works, hence its promise as a practical red teaming tool for T2l models.

Title: Solving Oscillator ODEs via Soft-constrained Physics-informed Neural Network with Small Data

• Authors: Kai-liang Lu, Yu-meng Su, Cheng Qiu, Zhuo Bi, Wen-jun Zhang
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.11077
• Pdf URL: https://arxiv.org/pdf/2408.11077
• Copy Paste: [[2408.11077]] Solving Oscillator ODEs via Soft-constrained Physics-informed Neural Network with Small Data(https://arxiv.org/abs/2408.11077)
• Keywords: robust
• Abstract: This paper compared physics-informed neural network (PINN), conventional neural network (NN) and numerical discretization methods on solving differential equations through literature research. We formalized the mathematical framework and computational flow of the soft-constrained PINN method for solving differential equations (e.g., ODEs/PDEs). Its working mechanism and its accuracy and efficiency were experimentally verified by solving typical linear and non-linear oscillator ODEs. The implementation of the PINN method based on DeepXDE is not only light code and efficient in training, but also flexible across platforms. PINN greatly reduces the need for labeled data: when the nonlinearity of the ODE is weak, a very small amount of supervised training data plus a small amount of collocation points are sufficient to predict the solution; in the minimalist case, only one or two training points (with initial values) are needed for first- or second-order ODEs, respectively. Strongly nonlinear ODE also require only an appropriate increase in the number of training and collocation points, which still has significant advantages over conventional NN. With the aid of collocation points and the use of physical information, PINN has the ability to extrapolate data outside the time domain covered by the training set, and is robust to noisy data, thus with enhanced generalization capabilities. Training is accelerated when the gains obtained along with the reduction in the amount of data outweigh the delay caused by the increase in the loss function terms. The soft-constrained PINN method can easily impose a physical law (e.g., energy conservation) constraint by adding a regularization term to the total loss function, thus improving the solution performance of ODEs that obey this physical law.

Title: ARAP: Demystifying Anti Runtime Analysis Code in Android Apps

• Authors: Dewen Suo, Lei Xue, Runze Tan, Weihao Huang, Guozi Sun
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.11080
• Pdf URL: https://arxiv.org/pdf/2408.11080
• Copy Paste: [[2408.11080]] ARAP: Demystifying Anti Runtime Analysis Code in Android Apps(https://arxiv.org/abs/2408.11080)
• Keywords: security
• Abstract: With the continuous growth in the usage of Android apps, ensuring their security has become critically important. An increasing number of malicious apps adopt anti-analysis techniques to evade security measures. Although some research has started to consider anti-runtime analysis (ARA), it is unfortunate that they have not systematically examined ARA techniques. Furthermore, the rapid evolution of ARA technology exacerbates the issue, leading to increasingly inaccurate analysis results. To effectively analyze Android apps, understanding their adopted ARA techniques is necessary. However, no systematic investigation has been conducted thus far. In this paper, we conduct the first systematic study of the ARA implementations in a wide range of 117,171 Android apps (including both malicious and benign ones) collected between 2016 and 2023. Additionally, we propose a specific investigation tool named ARAP to assist this study by leveraging both static and dynamic analysis. According to the evaluation results, ARAP not only effectively identifies the ARA implementations in Android apps but also reveals many important findings. For instance, almost all apps have implemented at least one category of ARA technology (99.6% for benign apps and 97.0% for malicious apps).

Title: GSLoc: Efficient Camera Pose Refinement via 3D Gaussian Splatting

• Authors: Changkun Liu, Shuai Chen, Yash Bhalgat, Siyan Hu, Zirui Wang, Ming Cheng, Victor Adrian Prisacariu, Tristan Braud
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11085
• Pdf URL: https://arxiv.org/pdf/2408.11085
• Copy Paste: [[2408.11085]] GSLoc: Efficient Camera Pose Refinement via 3D Gaussian Splatting(https://arxiv.org/abs/2408.11085)
• Keywords: robust
• Abstract: We leverage 3D Gaussian Splatting (3DGS) as a scene representation and propose a novel test-time camera pose refinement framework, GSLoc. This framework enhances the localization accuracy of state-of-the-art absolute pose regression and scene coordinate regression methods. The 3DGS model renders high-quality synthetic images and depth maps to facilitate the establishment of 2D-3D correspondences. GSLoc obviates the need for training feature extractors or descriptors by operating directly on RGB images, utilizing the 3D vision foundation model, MASt3R, for precise 2D matching. To improve the robustness of our model in challenging outdoor environments, we incorporate an exposure-adaptive module within the 3DGS framework. Consequently, GSLoc enables efficient pose refinement given a single RGB query and a coarse initial pose estimation. Our proposed approach surpasses leading NeRF-based optimization methods in both accuracy and runtime across indoor and outdoor visual localization benchmarks, achieving state-of-the-art accuracy on two indoor datasets.

Title: Post-Quantum Secure UE-to-UE Communications

• Authors: Sanzida Hoque, Abdullah Aydeger, Engin Zeydan
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.11117
• Pdf URL: https://arxiv.org/pdf/2408.11117
• Copy Paste: [[2408.11117]] Post-Quantum Secure UE-to-UE Communications(https://arxiv.org/abs/2408.11117)
• Keywords: secure, security, attack
• Abstract: The rapid development of quantum computing poses a significant threat to the security of current cryptographic systems, including those used in User Equipment (UE) for mobile communications. Conventional cryptographic algorithms such as Rivest-Shamir-Adleman (RSA) and Elliptic curve cryptography (ECC) are vulnerable to quantum computing attacks, which could jeopardize the confidentiality, integrity, and availability of sensitive data transmitted by UEs. This demo paper proposes the integration of Post-Quantum Cryptography (PQC) in TLS for UE Communication to mitigate the risks of quantum attacks. We present our setup and explain each of the components used. We also provide the entire workflow of the demo for other researchers to replicate the same setup. By addressing the implementation of PQC within a 5G network to secure UE-to-UE communication, this research aims to pave the way for developing quantum-resistant mobile devices and securing the future of wireless communications.

Title: DOMBA: Double Model Balancing for Access-Controlled Language Models via Minimum-Bounded Aggregation

• Authors: Tom Segal, Asaf Shabtai, Yuval Elovici
• Subjects: cs.LG, cs.AI, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11121
• Pdf URL: https://arxiv.org/pdf/2408.11121
• Copy Paste: [[2408.11121]] DOMBA: Double Model Balancing for Access-Controlled Language Models via Minimum-Bounded Aggregation(https://arxiv.org/abs/2408.11121)
• Keywords: secure, security, large language model
• Abstract: The utility of large language models (LLMs) depends heavily on the quality and quantity of their training data. Many organizations possess large data corpora that could be leveraged to train or fine-tune LLMs tailored to their specific needs. However, these datasets often come with access restrictions that are based on user privileges and enforced by access control mechanisms. Training LLMs on such datasets could result in exposure of sensitive information to unauthorized users. A straightforward approach for preventing such exposure is to train a separate model for each access level. This, however, may result in low utility models due to the limited amount of training data per model compared to the amount in the entire organizational corpus. Another approach is to train a single LLM on all the data while limiting the exposure of unauthorized information. However, current exposure-limiting methods for LLMs are ineffective for access-controlled data, where sensitive information appears frequently across many training examples. We propose DOMBA - double model balancing - a simple approach for training and deploying LLMs that provides high utility and access-control functionality with security guarantees. DOMBA aggregates the probability distributions of two models, each trained on documents with (potentially many) different access levels, using a "min-bounded" average function (a function that is bounded by the smaller value, e.g., harmonic mean). A detailed mathematical analysis and extensive evaluation show that DOMBA safeguards restricted information while offering utility comparable to non-secure models.

Title: Towards the Unmanned Aerial Vehicle Traffic Management Systems (UTMs): Security Risks and Challenges

• Authors: Konstantinos Spalas
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.11125
• Pdf URL: https://arxiv.org/pdf/2408.11125
• Copy Paste: [[2408.11125]] Towards the Unmanned Aerial Vehicle Traffic Management Systems (UTMs): Security Risks and Challenges(https://arxiv.org/abs/2408.11125)
• Keywords: security
• Abstract: Every aspect of our life depends on the ability to communicate effectively. Organizations that manage to establish communication routines, protocols and means thrive. An Aerial Traffic Management System operates similarly as an organization but certainly in a more strict manner. Third party agencies ensure several aspects of their functionality, the utmost to be consider safety. Many people take safety as granted but it is a pretty difficult part our daily functions. Thus, apart from digesting new things and habits of the new era, simultaneously we have to ensure safety in every part of it. It is true that the more data we produce, the more information we create and the more specialization we must introduce in order to be effective in a reasonable time basis. A Unmanned Aircraft System Traffic Management (UTM) is a system that consists of miscellaneous modules where each of them needs its consideration regarding safety. In other words, a UTM is the state-of-the-art system that demand a high quality of services and specialization, if we need to consider them reliable.

Title: MS$^3$D: A RG Flow-Based Regularization for GAN Training with Limited Data

• Authors: Jian Wang, Xin Lan, Yuxin Tian, Jiancheng Lv
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11135
• Pdf URL: https://arxiv.org/pdf/2408.11135
• Copy Paste: [[2408.11135]] MS$^3$D: A RG Flow-Based Regularization for GAN Training with Limited Data(https://arxiv.org/abs/2408.11135)
• Keywords: robust, generative
• Abstract: Generative adversarial networks (GANs) have made impressive advances in image generation, but they often require large-scale training data to avoid degradation caused by discriminator overfitting. To tackle this issue, we investigate the challenge of training GANs with limited data, and propose a novel regularization method based on the idea of renormalization group (RG) in physics.We observe that in the limited data setting, the gradient pattern that the generator obtains from the discriminator becomes more aggregated over time. In RG context, this aggregated pattern exhibits a high discrepancy from its coarse-grained versions, which implies a high-capacity and sensitive system, prone to overfitting and collapse. To address this problem, we introduce a \textbf{m}ulti-\textbf{s}cale \textbf{s}tructural \textbf{s}elf-\textbf{d}issimilarity (MS$^3$D) regularization, which constrains the gradient field to have a consistent pattern across different scales, thereby fostering a more redundant and robust system. We show that our method can effectively enhance the performance and stability of GANs under limited data scenarios, and even allow them to generate high-quality images with very few data.

Title: ISLES 2024: The first longitudinal multimodal multi-center real-world dataset in (sub-)acute stroke

• Authors: Evamaria O. Riedel, Ezequiel de la Rosa, The Anh Baran, Moritz Hernandez Petzsche, Hakim Baazaoui, Kaiyuan Yang, David Robben, Joaquin Oscar Seia, Roland Wiest, Mauricio Reyes, Ruisheng Su, Claus Zimmer, Tobias Boeckh-Behrens, Maria Berndt, Bjoern Menze, Benedikt Wiestler, Susanne Wegener, Jan S. Kirschke
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11142
• Pdf URL: https://arxiv.org/pdf/2408.11142
• Copy Paste: [[2408.11142]] ISLES 2024: The first longitudinal multimodal multi-center real-world dataset in (sub-)acute stroke(https://arxiv.org/abs/2408.11142)
• Keywords: segmentation
• Abstract: Stroke remains a leading cause of global morbidity and mortality, placing a heavy socioeconomic burden. Over the past decade, advances in endovascular reperfusion therapy and the use of CT and MRI imaging for treatment guidance have significantly improved patient outcomes and are now standard in clinical practice. To develop machine learning algorithms that can extract meaningful and reproducible models of brain function for both clinical and research purposes from stroke images - particularly for lesion identification, brain health quantification, and prognosis - large, diverse, and well-annotated public datasets are essential. While only a few datasets with (sub-)acute stroke data were previously available, several large, high-quality datasets have recently been made publicly accessible. However, these existing datasets include only MRI data. In contrast, our dataset is the first to offer comprehensive longitudinal stroke data, including acute CT imaging with angiography and perfusion, follow-up MRI at 2-9 days, as well as acute and longitudinal clinical data up to a three-month outcome. The dataset includes a training dataset of n = 150 and a test dataset of n = 100 scans. Training data is publicly available, while test data will be used exclusively for model validation. We are making this dataset available as part of the 2024 edition of the Ischemic Stroke Lesion Segmentation (ISLES) challenge (this https URL), which continuously aims to establish benchmark methods for acute and sub-acute ischemic stroke lesion segmentation, aiding in creating open stroke imaging datasets and evaluating cutting-edge image processing algorithms.

Title: Total Uncertainty Quantification in Inverse PDE Solutions Obtained with Reduced-Order Deep Learning Surrogate Models

• Authors: Yuanzhe Wang, Alexandre M. Tartakovsky
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11145
• Pdf URL: https://arxiv.org/pdf/2408.11145
• Copy Paste: [[2408.11145]] Total Uncertainty Quantification in Inverse PDE Solutions Obtained with Reduced-Order Deep Learning Surrogate Models(https://arxiv.org/abs/2408.11145)
• Keywords: diffusion
• Abstract: We propose an approximate Bayesian method for quantifying the total uncertainty in inverse PDE solutions obtained with machine learning surrogate models, including operator learning models. The proposed method accounts for uncertainty in the observations and PDE and surrogate models. First, we use the surrogate model to formulate a minimization problem in the reduced space for the maximum a posteriori (MAP) inverse solution. Then, we randomize the MAP objective function and obtain samples of the posterior distribution by minimizing different realizations of the objective function. We test the proposed framework by comparing it with the iterative ensemble smoother and deep ensembling methods for a non-linear diffusion equation with an unknown space-dependent diffusion coefficient. Among other problems, this equation describes groundwater flow in an unconfined aquifer. Depending on the training dataset and ensemble sizes, the proposed method provides similar or more descriptive posteriors of the parameters and states than the iterative ensemble smoother method. Deep ensembling underestimates uncertainty and provides less informative posteriors than the other two methods.

Title: An Interpretable Deep Learning Approach for Morphological Script Type Analysis

• Authors: Malamatenia Vlachou-Efstathiou, Ioannis Siglidis, Dominique Stutzmann, Mathieu Aubry
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11150
• Pdf URL: https://arxiv.org/pdf/2408.11150
• Copy Paste: [[2408.11150]] An Interpretable Deep Learning Approach for Morphological Script Type Analysis(https://arxiv.org/abs/2408.11150)
• Keywords: segmentation
• Abstract: Defining script types and establishing classification criteria for medieval handwriting is a central aspect of palaeographical analysis. However, existing typologies often encounter methodological challenges, such as descriptive limitations and subjective criteria. We propose an interpretable deep learning-based approach to morphological script type analysis, which enables systematic and objective analysis and contributes to bridging the gap between qualitative observations and quantitative measurements. More precisely, we adapt a deep instance segmentation method to learn comparable character prototypes, representative of letter morphology, and provide qualitative and quantitative tools for their comparison and analysis. We demonstrate our approach by applying it to the Textualis Formata script type and its two subtypes formalized by A. Derolez: Northern and Southern Textualis

Title: SubgoalXL: Subgoal-based Expert Learning for Theorem Proving

• Authors: Xueliang Zhao, Lin Zheng, Haige Bo, Changran Hu, Urmish Thakker, Lingpeng Kong
• Subjects: cs.LG, cs.AI, cs.CL, cs.LO
• Abstract URL: https://arxiv.org/abs/2408.11172
• Pdf URL: https://arxiv.org/pdf/2408.11172
• Copy Paste: [[2408.11172]] SubgoalXL: Subgoal-based Expert Learning for Theorem Proving(https://arxiv.org/abs/2408.11172)
• Keywords: large language model
• Abstract: Formal theorem proving, a field at the intersection of mathematics and computer science, has seen renewed interest with advancements in large language models (LLMs). This paper introduces SubgoalXL, a novel approach that synergizes subgoal-based proofs with expert learning to enhance LLMs' capabilities in formal theorem proving within the Isabelle environment. SubgoalXL addresses two critical challenges: the scarcity of specialized mathematics and theorem-proving data, and the need for improved multi-step reasoning abilities in LLMs. By optimizing data efficiency and employing subgoal-level supervision, SubgoalXL extracts richer information from limited human-generated proofs. The framework integrates subgoal-oriented proof strategies with an expert learning system, iteratively refining formal statement, proof, and subgoal generators. Leveraging the Isabelle environment's advantages in subgoal-based proofs, SubgoalXL achieves a new state-of-the-art performance of 56.1\% in Isabelle on the standard miniF2F dataset, marking an absolute improvement of 4.9\%. Notably, SubgoalXL successfully solves 41 AMC12, 9 AIME, and 3 IMO problems from miniF2F. These results underscore the effectiveness of maximizing limited data utility and employing targeted guidance for complex reasoning in formal theorem proving, contributing to the ongoing advancement of AI reasoning capabilities. The implementation is available at \url{this https URL}.

Title: Combining Objective and Subjective Perspectives for Political News Understanding

• Authors: Evan Dufraisse, Adrian Popescu, Julien Tourille, Armelle Brun, Olivier Hamon
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.11174
• Pdf URL: https://arxiv.org/pdf/2408.11174
• Copy Paste: [[2408.11174]] Combining Objective and Subjective Perspectives for Political News Understanding(https://arxiv.org/abs/2408.11174)
• Keywords: explainability
• Abstract: Researchers and practitioners interested in computational politics rely on automatic content analysis tools to make sense of the large amount of political texts available on the Web. Such tools should provide objective and subjective aspects at different granularity levels to make the analyses useful in practice. Existing methods produce interesting insights for objective aspects, but are limited for subjective ones, are often limited to national contexts, and have limited explainability. We introduce a text analysis framework which integrates both perspectives and provides a fine-grained processing of subjective aspects. Information retrieval techniques and knowledge bases complement powerful natural language processing components to allow a flexible aggregation of results at different granularity levels. Importantly, the proposed bottom-up approach facilitates the explainability of the obtained results. We illustrate its functioning with insights on news outlets, political orientations, topics, individual entities, and demographic segments. The approach is instantiated on a large corpus of French news, but is designed to work seamlessly for other languages and countries.

Title: A Full DAG Score-Based Algorithm for Learning Causal Bayesian Networks with Latent Confounders

• Authors: Christophe Gonzales, Amir-Hosein Valizadeh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11181
• Pdf URL: https://arxiv.org/pdf/2408.11181
• Copy Paste: [[2408.11181]] A Full DAG Score-Based Algorithm for Learning Causal Bayesian Networks with Latent Confounders(https://arxiv.org/abs/2408.11181)
• Keywords: robust
• Abstract: Causal Bayesian networks (CBN) are popular graphical probabilistic models that encode causal relations among variables. Learning their graphical structure from observational data has received a lot of attention in the literature. When there exists no latent (unobserved) confounder, i.e., no unobserved direct common cause of some observed variables, learning algorithms can be divided essentially into two classes: constraint-based and score-based approaches. The latter are often thought to be more robust than the former and to produce better results. However, to the best of our knowledge, when variables are discrete, no score-based algorithm is capable of dealing with latent confounders. This paper introduces the first fully score-based structure learning algorithm searching the space of DAGs (directed acyclic graphs) that is capable of identifying the presence of some latent confounders. It is justified mathematically and experiments highlight its effectiveness.

Title: Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Neural Carrier Articles

• Authors: Zhilong Wang, Haizhou Wang, Nanqing Luo, Lan Zhang, Xiaoyan Sun, Yebo Cao, Peng Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11182
• Pdf URL: https://arxiv.org/pdf/2408.11182
• Copy Paste: [[2408.11182]] Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Neural Carrier Articles(https://arxiv.org/abs/2408.11182)
• Keywords: attack, large language model
• Abstract: Jailbreak attacks on Language Model Models (LLMs) entail crafting prompts aimed at exploiting the models to generate malicious content. This paper proposes a new type of jailbreak attacks which shift the attention of the LLM by inserting a prohibited query into a carrier article. The proposed attack leverage the knowledge graph and a composer LLM to automatically generating a carrier article that is similar to the topic of the prohibited query but does not violate LLM's safeguards. By inserting the malicious query to the carrier article, the assembled attack payload can successfully jailbreak LLM. To evaluate the effectiveness of our method, we leverage 4 popular categories of ``harmful behaviors'' adopted by related researches to attack 6 popular LLMs. Our experiment results show that the proposed attacking method can successfully jailbreak all the target LLMs which high success rate, except for Claude-3.

Title: CRACKS: Crowdsourcing Resources for Analysis and Categorization of Key Subsurface faults

• Authors: Mohit Prabhushankar, Kiran Kokilepersaud, Jorge Quesada, Yavuz Yarici, Chen Zhou, Mohammad Alotaibi, Ghassan AlRegib, Ahmad Mustafa, Yusufjon Kumakov
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11185
• Pdf URL: https://arxiv.org/pdf/2408.11185
• Copy Paste: [[2408.11185]] CRACKS: Crowdsourcing Resources for Analysis and Categorization of Key Subsurface faults(https://arxiv.org/abs/2408.11185)
• Keywords: segmentation
• Abstract: Crowdsourcing annotations has created a paradigm shift in the availability of labeled data for machine learning. Availability of large datasets has accelerated progress in common knowledge applications involving visual and language data. However, specialized applications that require expert labels lag in data availability. One such application is fault segmentation in subsurface imaging. Detecting, tracking, and analyzing faults has broad societal implications in predicting fluid flows, earthquakes, and storing excess atmospheric CO$_2$. However, delineating faults with current practices is a labor-intensive activity that requires precise analysis of subsurface imaging data by geophysicists. In this paper, we propose the $\texttt{CRACKS}$ dataset to detect and segment faults in subsurface images by utilizing crowdsourced resources. We leverage Amazon Mechanical Turk to obtain fault delineations from sections of the Netherlands North Sea subsurface images from (i) $26$ novices who have no exposure to subsurface data and were shown a video describing and labeling faults, (ii) $8$ practitioners who have previously interacted and worked on subsurface data, (iii) one geophysicist to label $7636$ faults in the region. Note that all novices, practitioners, and the expert segment faults on the same subsurface volume with disagreements between and among the novices and practitioners. Additionally, each fault annotation is equipped with the confidence level of the annotator. The paper provides benchmarks on detecting and segmenting the expert labels, given the novice and practitioner labels. Additional details along with the dataset links and codes are available at $\href{this https URL}{link}$.

Title: Reading with Intent

• Authors: Benjamin Reichman, Kartik Talamadupula, Toshish Jawale, Larry Heck
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11189
• Pdf URL: https://arxiv.org/pdf/2408.11189
• Copy Paste: [[2408.11189]] Reading with Intent(https://arxiv.org/abs/2408.11189)
• Keywords: large language model
• Abstract: Retrieval augmented generation (RAG) systems augment how knowledge language models are by integrating external information sources such as Wikipedia, internal documents, scientific papers, or the open internet. RAG systems that rely on the open internet as their knowledge source have to contend with the complexities of human-generated content. Human communication extends much deeper than just the words rendered as text. Intent, tonality, and connotation can all change the meaning of what is being conveyed. Recent real-world deployments of RAG systems have shown some difficulty in understanding these nuances of human communication. One significant challenge for these systems lies in processing sarcasm. Though the Large Language Models (LLMs) that make up the backbone of these RAG systems are able to detect sarcasm, they currently do not always use these detections for the subsequent processing of text. To address these issues, in this paper, we synthetically generate sarcastic passages from Natural Question's Wikipedia retrieval corpus. We then test the impact of these passages on the performance of both the retriever and reader portion of the RAG pipeline. We introduce a prompting system designed to enhance the model's ability to interpret and generate responses in the presence of sarcasm, thus improving overall system performance. Finally, we conduct ablation studies to validate the effectiveness of our approach, demonstrating improvements in handling sarcastic content within RAG systems.

Title: Compress Guidance in Conditional Diffusion Sampling

• Authors: Anh-Dung Dinh, Daochang Liu, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11194
• Pdf URL: https://arxiv.org/pdf/2408.11194
• Copy Paste: [[2408.11194]] Compress Guidance in Conditional Diffusion Sampling(https://arxiv.org/abs/2408.11194)
• Keywords: diffusion, generative
• Abstract: Enforcing guidance throughout the entire sampling process often proves counterproductive due to the model-fitting issue., where samples are generated to match the classifier's parameters rather than generalizing the expected condition. This work identifies and quantifies the problem, demonstrating that reducing or excluding guidance at numerous timesteps can mitigate this issue. By distributing the guidance densely in the early stages of the process, we observe a significant improvement in image quality and diversity while also reducing the required guidance timesteps by nearly 40%. This approach addresses a major challenge in applying guidance effectively to generative tasks. Consequently, our proposed method, termed Compress Guidance, allows for the exclusion of a substantial number of guidance timesteps while still surpassing baseline models in image quality. We validate our approach through benchmarks on label conditional and text-to-image generative tasks across various datasets and models.

Title: Proposal of an Electronic Auditing System Applied to the Brazilian Electronic Voting Machine

• Authors: Marcelo Ferreira Guimarães, Carlos Antônio Sell, Renato Parenti Turcato, Carlos Henrique Assuiti, Ricardo Custódio, Ricardo Antônio Pralon Santos
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11195
• Pdf URL: https://arxiv.org/pdf/2408.11195
• Copy Paste: [[2408.11195]] Proposal of an Electronic Auditing System Applied to the Brazilian Electronic Voting Machine(https://arxiv.org/abs/2408.11195)
• Keywords: security, robust
• Abstract: A new system, called SELA -- Auditing Electronic System, has been developed to be applied to the Brazilian Electronic Voting Machine. The SELA was designed to use open hardware and software, making it widely known by society. The security of the auditing process is guaranteed by the application of a Fingerprint Algorithm, a Hash Function. This system is robust and requires minimal modifications to the Electronic Voting Machine. In this paper, SELA is described, and its use during the election process is analyzed. A comparison between SELA and the use of thermal printers as a secondary voting record system is also presented. The authors recommend a pilot implementation of SELA for the 2002 Brazilian Elections.

Title: Robust Long-Range Perception Against Sensor Misalignment in Autonomous Vehicles

• Authors: Zi-Xiang Xia, Sudeep Fadadu, Yi Shi, Louis Foucard
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11196
• Pdf URL: https://arxiv.org/pdf/2408.11196
• Copy Paste: [[2408.11196]] Robust Long-Range Perception Against Sensor Misalignment in Autonomous Vehicles(https://arxiv.org/abs/2408.11196)
• Keywords: robust
• Abstract: Advances in machine learning algorithms for sensor fusion have significantly improved the detection and prediction of other road users, thereby enhancing safety. However, even a small angular displacement in the sensor's placement can cause significant degradation in output, especially at long range. In this paper, we demonstrate a simple yet generic and efficient multi-task learning approach that not only detects misalignment between different sensor modalities but is also robust against them for long-range perception. Along with the amount of misalignment, our method also predicts calibrated uncertainty, which can be useful for filtering and fusing predicted misalignment values over time. In addition, we show that the predicted misalignment parameters can be used for self-correcting input sensor data, further improving the perception performance under sensor misalignment.

Title: UKAN: Unbound Kolmogorov-Arnold Network Accompanied with Accelerated Library

• Authors: Alireza Moradzadeh, Lukasz Wawrzyniak, Miles Macklin, Saee G. Paliwal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11200
• Pdf URL: https://arxiv.org/pdf/2408.11200
• Copy Paste: [[2408.11200]] UKAN: Unbound Kolmogorov-Arnold Network Accompanied with Accelerated Library(https://arxiv.org/abs/2408.11200)
• Keywords: generative
• Abstract: In this work, we present a GPU-accelerated library for the underlying components of Kolmogorov-Arnold Networks (KANs), along with an algorithm to eliminate bounded grids in KANs. The GPU-accelerated library reduces the computational complexity of Basis Spline (B-spline) evaluation by a factor of $\mathcal{O}$(grid size) compared to existing codes, enabling batch computation for large-scale learning. To overcome the limitations of traditional KANs, we introduce Unbounded KANs (UKANs), which eliminate the need for a bounded grid and a fixed number of B-spline coefficients. To do so, we replace the KAN parameters (B-spline coefficients) with a coefficient generator (CG) model. The inputs to the CG model are designed based on the idea of an infinite symmetric grid extending from negative infinity to positive infinity. The positional encoding of grid group, a sequential collection of B-spline grid indexes, is fed into the CG model, and coefficients are consumed by the efficient implementation (matrix representations) of B-spline functions to generate outputs. We perform several experiments on regression, classification, and generative tasks, which are promising. In particular, UKAN does not require data normalization or a bounded domain for evaluation. Additionally, our benchmarking results indicate the superior memory and computational efficiency of our library compared to existing codes.

Title: Detecting Fraudulent Services on Quantum Cloud Platforms via Dynamic Fingerprinting

• Authors: Jindi Wu, Tianjie Hu, Qun Li
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2408.11203
• Pdf URL: https://arxiv.org/pdf/2408.11203
• Copy Paste: [[2408.11203]] Detecting Fraudulent Services on Quantum Cloud Platforms via Dynamic Fingerprinting(https://arxiv.org/abs/2408.11203)
• Keywords: protect, attack
• Abstract: Noisy Intermediate-Scale Quantum (NISQ) devices, while accessible via cloud platforms, face challenges due to limited availability and suboptimal quality. These challenges raise the risk of cloud providers offering fraudulent services. This emphasizes the need for users to detect such fraud to protect their investments and ensure computational integrity. This study introduces a novel dynamic fingerprinting method for detecting fraudulent service provision on quantum cloud platforms, specifically targeting machine substitution and profile fabrication attacks. The dynamic fingerprint is constructed using a \textit{single} probing circuit to capture the unique error characteristics of quantum devices, making this approach practical because of its trivial computational costs. When the user examines the service, the execution results of the probing circuit act as the device-side fingerprint of the quantum device providing the service. The user then generates the user-side fingerprint by estimating the expected execution result, assuming the correct device is in use. We propose an algorithm for users to construct the user-side fingerprint with linear complexity. By comparing the device-side and user-side fingerprints, users can effectively detect fraudulent services. Our experiments on the IBM Quantum platform, involving seven devices with varying capabilities, confirm the method's effectiveness.

Title: Quantum Inverse Contextual Vision Transformers (Q-ICVT): A New Frontier in 3D Object Detection for AVs

• Authors: Sanjay Bhargav Dharavath, Tanmoy Dam, Supriyo Chakraborty, Prithwiraj Roy, Aniruddha Maiti
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11207
• Pdf URL: https://arxiv.org/pdf/2408.11207
• Copy Paste: [[2408.11207]] Quantum Inverse Contextual Vision Transformers (Q-ICVT): A New Frontier in 3D Object Detection for AVs(https://arxiv.org/abs/2408.11207)
• Keywords: transformer
• Abstract: The field of autonomous vehicles (AVs) predominantly leverages multi-modal integration of LiDAR and camera data to achieve better performance compared to using a single modality. However, the fusion process encounters challenges in detecting distant objects due to the disparity between the high resolution of cameras and the sparse data from LiDAR. Insufficient integration of global perspectives with local-level details results in sub-optimal fusion this http URL address this issue, we have developed an innovative two-stage fusion process called Quantum Inverse Contextual Vision Transformers (Q-ICVT). This approach leverages adiabatic computing in quantum concepts to create a novel reversible vision transformer known as the Global Adiabatic Transformer (GAT). GAT aggregates sparse LiDAR features with semantic features in dense images for cross-modal integration in a global form. Additionally, the Sparse Expert of Local Fusion (SELF) module maps the sparse LiDAR 3D proposals and encodes position information of the raw point cloud onto the dense camera feature space using a gating point fusion approach. Our experiments show that Q-ICVT achieves an mAPH of 82.54 for L2 difficulties on the Waymo dataset, improving by 1.88% over current state-of-the-art fusion methods. We also analyze GAT and SELF in ablation studies to highlight the impact of Q-ICVT. Our code is available at this https URL Q-ICVT

Title: A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation

• Authors: Yufan He, Pengfei Guo, Yucheng Tang, Andriy Myronenko, Vishwesh Nath, Ziyue Xu, Dong Yang, Can Zhao, Daguang Xu, Wenqi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11210
• Pdf URL: https://arxiv.org/pdf/2408.11210
• Copy Paste: [[2408.11210]] A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation(https://arxiv.org/abs/2408.11210)
• Keywords: segmentation
• Abstract: Since the release of Segment Anything 2 (SAM2), the medical imaging community has been actively evaluating its performance for 3D medical image segmentation. However, different studies have employed varying evaluation pipelines, resulting in conflicting outcomes that obscure a clear understanding of SAM2's capabilities and potential applications. We shortly review existing benchmarks and point out that the SAM2 paper clearly outlines a zero-shot evaluation pipeline, which simulates user clicks iteratively for up to eight iterations. We reproduced this interactive annotation simulation on 3D CT datasets and provided the results and code~\url{this https URL}. Our findings reveal that directly applying SAM2 on 3D medical imaging in a zero-shot manner is far from satisfactory. It is prone to generating false positives when foreground objects disappear, and annotating more slices cannot fully offset this tendency. For smaller single-connected objects like kidney and aorta, SAM2 performs reasonably well but for most organs it is still far behind state-of-the-art 3D annotation methods. More research and innovation are needed for 3D medical imaging community to use SAM2 correctly.

Title: Revisiting Min-Max Optimization Problem in Adversarial Training

• Authors: Sina Hajer Ahmadi, Hassan Bahrami
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11218
• Pdf URL: https://arxiv.org/pdf/2408.11218
• Copy Paste: [[2408.11218]] Revisiting Min-Max Optimization Problem in Adversarial Training(https://arxiv.org/abs/2408.11218)
• Keywords: security, attack, robust
• Abstract: The rise of computer vision applications in the real world puts the security of the deep neural networks at risk. Recent works demonstrate that convolutional neural networks are susceptible to adversarial examples - where the input images look similar to the natural images but are classified incorrectly by the model. To provide a rebuttal to this problem, we propose a new method to build robust deep neural networks against adversarial attacks by reformulating the saddle point optimization problem in \cite{madry2017towards}. Our proposed method offers significant resistance and a concrete security guarantee against multiple adversaries. The goal of this paper is to act as a stepping stone for a new variation of deep learning models which would lead towards fully robust deep learning models.

Title: A Little Confidence Goes a Long Way

• Authors: John Scoville, Shang Gao, Devanshu Agrawal, Javed Qadrud-Din
• Subjects: cs.LG, cs.AI, cs.CL, cs.IT, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.11239
• Pdf URL: https://arxiv.org/pdf/2408.11239
• Copy Paste: [[2408.11239]] A Little Confidence Goes a Long Way(https://arxiv.org/abs/2408.11239)
• Keywords: large language model
• Abstract: We introduce a group of related methods for binary classification tasks using probes of the hidden state activations in large language models (LLMs). Performance is on par with the largest and most advanced LLMs currently available, but requiring orders of magnitude fewer computational resources and not requiring labeled data. This approach involves translating class labels into a semantically rich description, spontaneous symmetry breaking of multilayer perceptron probes for unsupervised learning and inference, training probes to generate confidence scores (prior probabilities) from hidden state activations subject to known constraints via entropy maximization, and selecting the most confident probe model from an ensemble for prediction. These techniques are evaluated on four datasets using five base LLMs.

Title: CooPre: Cooperative Pretraining for V2X Cooperative Perception

• Authors: Seth Z. Zhao, Hao Xiang, Chenfeng Xu, Xin Xia, Bolei Zhou, Jiaqi Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11241
• Pdf URL: https://arxiv.org/pdf/2408.11241
• Copy Paste: [[2408.11241]] CooPre: Cooperative Pretraining for V2X Cooperative Perception(https://arxiv.org/abs/2408.11241)
• Keywords: robust
• Abstract: Existing Vehicle-to-Everything (V2X) cooperative perception methods rely on accurate multi-agent 3D annotations. Nevertheless, it is time-consuming and expensive to collect and annotate real-world data, especially for V2X systems. In this paper, we present a self-supervised learning method for V2X cooperative perception, which utilizes the vast amount of unlabeled 3D V2X data to enhance the perception performance. Beyond simply extending the previous pre-training methods for point-cloud representation learning, we introduce a novel self-supervised Cooperative Pretraining framework (termed as CooPre) customized for a collaborative scenario. We point out that cooperative point-cloud sensing compensates for information loss among agents. This motivates us to design a novel proxy task for the 3D encoder to reconstruct LiDAR point clouds across different agents. Besides, we develop a V2X bird-eye-view (BEV) guided masking strategy which effectively allows the model to pay attention to 3D features across heterogeneous V2X agents (i.e., vehicles and infrastructure) in the BEV space. Noticeably, such a masking strategy effectively pretrains the 3D encoder and is compatible with mainstream cooperative perception backbones. Our approach, validated through extensive experiments on representative datasets (i.e., V2X-Real, V2V4Real, and OPV2V), leads to a performance boost across all V2X settings. Additionally, we demonstrate the framework's improvements in cross-domain transferability, data efficiency, and robustness under challenging scenarios. The code will be made publicly available.

Title: Unboxing Occupational Bias: Grounded Debiasing LLMs with U.S. Labor Data

• Authors: Atmika Gorti, Manas Gaur, Aman Chadha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11247
• Pdf URL: https://arxiv.org/pdf/2408.11247
• Copy Paste: [[2408.11247]] Unboxing Occupational Bias: Grounded Debiasing LLMs with U.S. Labor Data(https://arxiv.org/abs/2408.11247)
• Keywords: fair, large language model
• Abstract: Large Language Models (LLMs) are prone to inheriting and amplifying societal biases embedded within their training data, potentially reinforcing harmful stereotypes related to gender, occupation, and other sensitive categories. This issue becomes particularly problematic as biased LLMs can have far-reaching consequences, leading to unfair practices and exacerbating social inequalities across various domains, such as recruitment, online content moderation, or even the criminal justice system. Although prior research has focused on detecting bias in LLMs using specialized datasets designed to highlight intrinsic biases, there has been a notable lack of investigation into how these findings correlate with authoritative datasets, such as those from the U.S. National Bureau of Labor Statistics (NBLS). To address this gap, we conduct empirical research that evaluates LLMs in a ``bias-out-of-the-box" setting, analyzing how the generated outputs compare with the distributions found in NBLS data. Furthermore, we propose a straightforward yet effective debiasing mechanism that directly incorporates NBLS instances to mitigate bias within LLMs. Our study spans seven different LLMs, including instructable, base, and mixture-of-expert models, and reveals significant levels of bias that are often overlooked by existing bias detection techniques. Importantly, our debiasing method, which does not rely on external datasets, demonstrates a substantial reduction in bias scores, highlighting the efficacy of our approach in creating fairer and more reliable LLMs.

Title: CNN-based Labelled Crack Detection for Image Annotation

• Authors: Mohsen Asghari Ilani, Leila Amini, Hossein Karimi, Maryam Shavali Kuhshuri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11250
• Pdf URL: https://arxiv.org/pdf/2408.11250
• Copy Paste: [[2408.11250]] CNN-based Labelled Crack Detection for Image Annotation(https://arxiv.org/abs/2408.11250)
• Keywords: extraction
• Abstract: Numerous image processing techniques (IPTs) have been employed to detect crack defects, offering an alternative to human-conducted onsite inspections. These IPTs manipulate images to extract defect features, particularly cracks in surfaces produced through Additive Manufacturing (AM). This article presents a vision-based approach that utilizes deep convolutional neural networks (CNNs) for crack detection in AM surfaces. Traditional image processing techniques face challenges with diverse real-world scenarios and varying crack types. To overcome these challenges, our proposed method leverages CNNs, eliminating the need for extensive feature extraction. Annotation for CNN training is facilitated by LabelImg without the requirement for additional IPTs. The trained CNN, enhanced by OpenCV preprocessing techniques, achieves an outstanding 99.54% accuracy on a dataset of 14,982 annotated images with resolutions of 1536 x 1103 pixels. Evaluation metrics exceeding 96% precision, 98% recall, and a 97% F1-score highlight the precision and effectiveness of the entire process.

Title: Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Models

• Authors: Sepehr Kamahi, Yadollah Yaghoobzadeh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11252
• Pdf URL: https://arxiv.org/pdf/2408.11252
• Copy Paste: [[2408.11252]] Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Models(https://arxiv.org/abs/2408.11252)
• Keywords: explainability
• Abstract: Despite the widespread adoption of autoregressive language models, explainability evaluation research has predominantly focused on span infilling and masked language models (MLMs). Evaluating the faithfulness of an explanation method -- how accurately the method explains the inner workings and decision-making of the model -- is very challenging because it is very hard to separate the model from its explanation. Most faithfulness evaluation techniques corrupt or remove some input tokens considered important according to a particular attribution (feature importance) method and observe the change in the model's output. This approach creates out-of-distribution inputs for causal language models (CLMs) due to their training objective of next token prediction. In this study, we propose a technique that leverages counterfactual generation to evaluate the faithfulness of attribution methods for autoregressive language modeling scenarios. Our technique creates fluent and in-distribution counterfactuals that makes evaluation protocol more reliable. Code is available at this https URL

Title: Automatic Image Annotation (AIA) of AlmondNet-20 Method for Almond Detection by Improved CNN-based Model

• Authors: Mohsen Asghari Ilani, Saba Moftakhar Tehran, Ashkan Kavei, Arian Radmehr
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11253
• Pdf URL: https://arxiv.org/pdf/2408.11253
• Copy Paste: [[2408.11253]] Automatic Image Annotation (AIA) of AlmondNet-20 Method for Almond Detection by Improved CNN-based Model(https://arxiv.org/abs/2408.11253)
• Keywords: robust
• Abstract: In response to the burgeoning global demand for premium agricultural products, particularly within the competitive nut market, this paper introduces an innovative methodology aimed at enhancing the grading process for almonds and their shells. Leveraging state-of-the-art Deep Convolutional Neural Networks (CNNs), specifically the AlmondNet-20 architecture, our study achieves exceptional accuracy exceeding 99%, facilitated by the utilization of a 20-layer CNN model. To bolster robustness in differentiating between almonds and shells, data augmentation techniques are employed, ensuring the reliability and accuracy of our classification system. Our model, meticulously trained over 1000 epochs, demonstrates remarkable performance, boasting an accuracy rate of 99% alongside a minimal loss function of 0.0567. Rigorous evaluation through test datasets further validates the efficacy of our approach, revealing impeccable precision, recall, and F1-score metrics for almond detection. Beyond its technical prowess, this advanced classification system offers tangible benefits to both industry experts and non-specialists alike, ensuring globally reliable almond classification. The application of deep learning algorithms, as showcased in our study, not only enhances grading accuracy but also presents opportunities for product patents, thereby contributing to the economic value of our nation. Through the adoption of cutting-edge technologies such as the AlmondNet-20 model, we pave the way for future advancements in agricultural product classification, ultimately enriching global trade and economic prosperity.

Title: Privacy-Preserving Data Management using Blockchains

• Authors: Michael Mireku Kwakye
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2408.11263
• Pdf URL: https://arxiv.org/pdf/2408.11263
• Copy Paste: [[2408.11263]] Privacy-Preserving Data Management using Blockchains(https://arxiv.org/abs/2408.11263)
• Keywords: secure, privacy, protect
• Abstract: Privacy-preservation policies are guidelines formulated to protect data providers private data. Previous privacy-preservation methodologies have addressed privacy in which data are permanently stored in repositories and disconnected from changing data provider privacy preferences. This occurrence becomes evident as data moves to another data repository. Hence, the need for data providers to control and flexibly update their existing privacy preferences due to changing data usage continues to remain a problem. This paper proposes a blockchain-based methodology for preserving data providers private and sensitive data. The research proposes to tightly couple data providers private attribute data element to privacy preferences and data accessor data element into a privacy tuple. The implementation presents a framework of tightly-coupled relational database and blockchains. This delivers secure, tamper-resistant, and query-efficient platform for data management and query processing. The evaluation analysis from the implementation validates efficient query processing of privacy-aware queries on the privacy infrastructure.

Title: Correlation Analysis of Adversarial Attack in Time Series Classification

• Authors: Zhengyang Li, Wenhao Liang, Chang Dong, Weitong Chen, Dong Huang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11264
• Pdf URL: https://arxiv.org/pdf/2408.11264
• Copy Paste: [[2408.11264]] Correlation Analysis of Adversarial Attack in Time Series Classification(https://arxiv.org/abs/2408.11264)
• Keywords: defense, attack
• Abstract: This study investigates the vulnerability of time series classification models to adversarial attacks, with a focus on how these models process local versus global information under such conditions. By leveraging the Normalized Auto Correlation Function (NACF), an exploration into the inclination of neural networks is conducted. It is demonstrated that regularization techniques, particularly those employing Fast Fourier Transform (FFT) methods and targeting frequency components of perturbations, markedly enhance the effectiveness of attacks. Meanwhile, the defense strategies, like noise introduction and Gaussian filtering, are shown to significantly lower the Attack Success Rate (ASR), with approaches based on noise introducing notably effective in countering high-frequency distortions. Furthermore, models designed to prioritize global information are revealed to possess greater resistance to adversarial manipulations. These results underline the importance of designing attack and defense mechanisms, informed by frequency domain analysis, as a means to considerably reinforce the resilience of neural network models against adversarial threats.

Title: Inverting the Leverage Score Gradient: An Efficient Approximate Newton Method

• Authors: Chenyang Li, Zhao Song, Zhaoxing Xu, Junze Yin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11267
• Pdf URL: https://arxiv.org/pdf/2408.11267
• Copy Paste: [[2408.11267]] Inverting the Leverage Score Gradient: An Efficient Approximate Newton Method(https://arxiv.org/abs/2408.11267)
• Keywords: security, privacy
• Abstract: Leverage scores have become essential in statistics and machine learning, aiding regression analysis, randomized matrix computations, and various other tasks. This paper delves into the inverse problem, aiming to recover the intrinsic model parameters given the leverage scores gradient. This endeavor not only enriches the theoretical understanding of models trained with leverage score techniques but also has substantial implications for data privacy and adversarial security. We specifically scrutinize the inversion of the leverage score gradient, denoted as $g(x)$. An innovative iterative algorithm is introduced for the approximate resolution of the regularized least squares problem stated as $\min_{x \in \mathbb{R}^d} 0.5 \|g(x) - c\|_2^2 + 0.5\|\mathrm{diag}(w)Ax\|_2^2$. Our algorithm employs subsampled leverage score distributions to compute an approximate Hessian in each iteration, under standard assumptions, considerably mitigating the time complexity. Given that a total of $T = \log(\| x_0 - x^* \|_2/ \epsilon)$ iterations are required, the cost per iteration is optimized to the order of $O( (\mathrm{nnz}(A) + d^{\omega} ) \cdot \mathrm{poly}(\log(n/\delta))$, where $\mathrm{nnz}(A)$ denotes the number of non-zero entries of $A$.

Title: On Missing Scores in Evolving Multibiometric Systems

• Authors: Melissa R Dale, Anil Jain, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11271
• Pdf URL: https://arxiv.org/pdf/2408.11271
• Copy Paste: [[2408.11271]] On Missing Scores in Evolving Multibiometric Systems(https://arxiv.org/abs/2408.11271)
• Keywords: biometric
• Abstract: The use of multiple modalities (e.g., face and fingerprint) or multiple algorithms (e.g., three face comparators) has shown to improve the recognition accuracy of an operational biometric system. Over time a biometric system may evolve to add new modalities, retire old modalities, or be merged with other biometric systems. This can lead to scenarios where there are missing scores corresponding to the input probe set. Previous work on this topic has focused on either the verification or identification tasks, but not both. Further, the proportion of missing data considered has been less than 50%. In this work, we study the impact of missing score data for both the verification and identification tasks. We show that the application of various score imputation methods along with simple sum fusion can improve recognition accuracy, even when the proportion of missing scores increases to 90%. Experiments show that fusion after score imputation outperforms fusion with no imputation. Specifically, iterative imputation with K nearest neighbors consistently surpasses other imputation methods in both the verification and identification tasks, regardless of the amount of scores missing, and provides imputed values that are consistent with the ground truth complete dataset.

Title: The Key of Parameter Skew in Federated Learning

• Authors: Sifan Wang, Junfeng Liao, Ye Yuan, Riquan Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11278
• Pdf URL: https://arxiv.org/pdf/2408.11278
• Copy Paste: [[2408.11278]] The Key of Parameter Skew in Federated Learning(https://arxiv.org/abs/2408.11278)
• Keywords: federate
• Abstract: Federated Learning (FL) has emerged as an excellent solution for performing deep learning on different data owners without exchanging raw data. However, statistical heterogeneity in FL presents a key challenge, leading to a phenomenon of skewness in local model parameter distributions that researchers have largely overlooked. In this work, we propose the concept of parameter skew to describe the phenomenon that can substantially affect the accuracy of global model parameter estimation. Additionally, we introduce FedSA, an aggregation strategy to obtain a high-quality global model, to address the implication from parameter skew. Specifically, we categorize parameters into high-dispersion and low-dispersion groups based on the coefficient of variation. For high-dispersion parameters, Micro-Classes (MIC) and Macro-Classes (MAC) represent the dispersion at the micro and macro levels, respectively, forming the foundation of FedSA. To evaluate the effectiveness of FedSA, we conduct extensive experiments with different FL algorithms on three computer vision datasets. FedSA outperforms eight state-of-the-art baselines by about 4.7% in test accuracy.

Title: Exploring Scene Coherence for Semi-Supervised 3D Semantic Segmentation

• Authors: Chuandong Liu, Shuguo Jiang, Xingxing Weng, Lei Yu, Pengcheng Li, Gui-Song Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11280
• Pdf URL: https://arxiv.org/pdf/2408.11280
• Copy Paste: [[2408.11280]] Exploring Scene Coherence for Semi-Supervised 3D Semantic Segmentation(https://arxiv.org/abs/2408.11280)
• Keywords: segmentation
• Abstract: Semi-supervised semantic segmentation, which efficiently addresses the limitation of acquiring dense annotations, is essential for 3D scene understanding. Most methods leverage the teacher model to generate pseudo labels, and then guide the learning of the student model on unlabeled scenes. However, they focus only on points with pseudo labels while directly overlooking points without pseudo labels, namely intra-scene inconsistency, leading to semantic ambiguity. Moreover, inter-scene correlation between labeled and unlabeled scenes contribute to transferring rich annotation information, yet this has not been explored for the semi-supervised tasks. To address these two problems, we propose to explore scene coherence for semi-supervised 3D semantic segmentation, dubbed CoScene. Inspired by the unstructured and unordered nature of the point clouds, our CoScene adopts the straightforward point erasure strategy to ensure the intra-scene consistency. Moreover, patch-based data augmentation is proposed to enhance the inter-scene information transfer between labeled and unlabeled scenes at both scene and instance levels. Extensive experimental results on SemanticKITTI and nuScenes show that our approach outperforms existing methods.

Title: Video Emotion Open-vocabulary Recognition Based on Multimodal Large Language Model

• Authors: Mengying Ge, Dongkai Tang, Mingyang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11286
• Pdf URL: https://arxiv.org/pdf/2408.11286
• Copy Paste: [[2408.11286]] Video Emotion Open-vocabulary Recognition Based on Multimodal Large Language Model(https://arxiv.org/abs/2408.11286)
• Keywords: large language model
• Abstract: Multimodal emotion recognition is a task of great concern. However, traditional data sets are based on fixed labels, resulting in models that often focus on main emotions and ignore detailed emotional changes in complex scenes. This report introduces the solution of using MLLMs technology to generate open-vocabulary emotion labels from a video. The solution includes the use of framework, data generation and processing, training methods, results generation and multi-model co-judgment. In the MER-OV (Open-Word Emotion Recognition) of the MER2024 challenge, our method achieved significant advantages, leading to its superior capabilities in complex emotion computation.

Title: Taming Generative Diffusion for Universal Blind Image Restoration

• Authors: Siwei Tu, Weidong Yang, Ben Fei
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11287
• Pdf URL: https://arxiv.org/pdf/2408.11287
• Copy Paste: [[2408.11287]] Taming Generative Diffusion for Universal Blind Image Restoration(https://arxiv.org/abs/2408.11287)
• Keywords: diffusion, generative
• Abstract: Diffusion models have been widely utilized for image restoration. However, previous blind image restoration methods still need to assume the type of degradation model while leaving the parameters to be optimized, limiting their real-world applications. Therefore, we aim to tame generative diffusion prior for universal blind image restoration dubbed BIR-D, which utilizes an optimizable convolutional kernel to simulate the degradation model and dynamically update the parameters of the kernel in the diffusion steps, enabling it to achieve blind image restoration results even in various complex situations. Besides, based on mathematical reasoning, we have provided an empirical formula for the chosen of adaptive guidance scale, eliminating the need for a grid search for the optimal parameter. Experimentally, Our BIR-D has demonstrated superior practicality and versatility than off-the-shelf unsupervised methods across various tasks both on real-world and synthetic datasets, qualitatively and quantitatively. BIR-D is able to fulfill multi-guidance blind image restoration. Moreover, BIR-D can also restore images that undergo multiple and complicated degradations, demonstrating the practical applications.

Title: RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining

• Authors: Anh-Dung Vo, Minseong Jung, Wonbeen Lee, Daewoo Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11294
• Pdf URL: https://arxiv.org/pdf/2408.11294
• Copy Paste: [[2408.11294]] RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining(https://arxiv.org/abs/2408.11294)
• Keywords: large language model
• Abstract: The field of Natural Language Processing (NLP) has seen significant advancements with the development of Large Language Models (LLMs). However, much of this research remains focused on English, often overlooking low-resource languages like Korean. This oversight presents challenges due to the unique non-alphabetic token structure of Korean and the substantial memory and computational demands required for LLM training, which frequently lead to memory constraints and out-of-memory errors. To address these issues, we present RedWhale, a model specifically tailored for Korean language processing. RedWhale is developed using an efficient continual pretraining approach that includes a comprehensive Korean corpus preprocessing pipeline, a specialized tokenizer, an optimized model initialization technique, and a multistage pretraining strategy. These innovations collectively reduce training time and computational costs while maintaining high levels of accuracy and comprehension. By leveraging cross-lingual transfer learning, which exploits shared linguistic similarities across languages, RedWhale builds on English models to enhance Korean language processing. Experimental results demonstrate that RedWhale outperforms other leading models on Korean NLP benchmarks, including the Korean Balanced Evaluation of Significant Tasks (KoBEST), showing superior understanding and generation of Korean text. Furthermore, RedWhale showed no signs of convergence even after pretraining on 9.7 billion tokens, indicating the potential for further improvements with additional training. This work represents a significant advancement in bridging the linguistic divide, particularly in enhancing NLP capabilities for the Korean language.

Title: FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts

• Authors: Hanzi Mei, Dongqi Cai, Ao Zhou, Shangguang Wang, Mengwei Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11304
• Pdf URL: https://arxiv.org/pdf/2408.11304
• Copy Paste: [[2408.11304]] FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts(https://arxiv.org/abs/2408.11304)
• Keywords: federate, large language model
• Abstract: As Large Language Models (LLMs) push the boundaries of AI capabilities, their demand for data is growing. Much of this data is private and distributed across edge devices, making Federated Learning (FL) a de-facto alternative for fine-tuning (i.e., FedLLM). However, it faces significant challenges due to the inherent heterogeneity among clients, including varying data distributions and diverse task types. Towards a versatile FedLLM, we replace traditional dense model with a sparsely-activated Mixture-of-Experts (MoE) architecture, whose parallel feed-forward networks enable greater flexibility. To make it more practical in resource-constrained environments, we present FedMoE, the efficient personalized FL framework to address data heterogeneity, constructing an optimal sub-MoE for each client and bringing the knowledge back to global MoE. FedMoE is composed of two fine-tuning stages. In the first stage, FedMoE simplifies the problem by conducting a heuristic search based on observed activation patterns, which identifies a suboptimal submodel for each client. In the second stage, these submodels are distributed to clients for further training and returned for server aggregating through a novel modular aggregation strategy. Meanwhile, FedMoE progressively adjusts the submodels to optimal through global expert recommendation. Experimental results demonstrate the superiority of our method over previous personalized FL methods.

Title: UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation

• Authors: Xiangyu Zhao, Yuehan Zhang, Wenlong Zhang, Xiao-Ming Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11305
• Pdf URL: https://arxiv.org/pdf/2408.11305
• Copy Paste: [[2408.11305]] UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation(https://arxiv.org/abs/2408.11305)
• Keywords: diffusion, generative, large language model
• Abstract: The fashion domain encompasses a variety of real-world multimodal tasks, including multimodal retrieval and multimodal generation. The rapid advancements in artificial intelligence generated content, particularly in technologies like large language models for text generation and diffusion models for visual generation, have sparked widespread research interest in applying these multimodal models in the fashion domain. However, tasks involving embeddings, such as image-to-text or text-to-image retrieval, have been largely overlooked from this perspective due to the diverse nature of the multimodal fashion domain. And current research on multi-task single models lack focus on image generation. In this work, we present UniFashion, a unified framework that simultaneously tackles the challenges of multimodal generation and retrieval tasks within the fashion domain, integrating image generation with retrieval tasks and text generation tasks. UniFashion unifies embedding and generative tasks by integrating a diffusion model and LLM, enabling controllable and high-fidelity generation. Our model significantly outperforms previous single-task state-of-the-art models across diverse fashion tasks, and can be readily adapted to manage complex vision-language tasks. This work demonstrates the potential learning synergy between multimodal generation and retrieval, offering a promising direction for future research in the fashion domain. The source code is available at this https URL.

Title: KAN4TSF: Are KAN and KAN-based models Effective for Time Series Forecasting?

• Authors: Xiao Han, Xinfeng Zhang, Yiling Wu, Zhenduo Zhang, Zhe Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11306
• Pdf URL: https://arxiv.org/pdf/2408.11306
• Copy Paste: [[2408.11306]] KAN4TSF: Are KAN and KAN-based models Effective for Time Series Forecasting?(https://arxiv.org/abs/2408.11306)
• Keywords: interpretability
• Abstract: Time series forecasting is a crucial task that predicts the future values of variables based on historical data. Time series forecasting techniques have been developing in parallel with the machine learning community, from early statistical learning methods to current deep learning methods. Although existing methods have made significant progress, they still suffer from two challenges. The mathematical theory of mainstream deep learning-based methods does not establish a clear relation between network sizes and fitting capabilities, and these methods often lack interpretability. To this end, we introduce the Kolmogorov-Arnold Network (KAN) into time series forecasting research, which has better mathematical properties and interpretability. First, we propose the Reversible Mixture of KAN experts (RMoK) model, which is a KAN-based model for time series forecasting. RMoK uses a mixture-of-experts structure to assign variables to KAN experts. Then, we compare performance, integration, and speed between RMoK and various baselines on real-world datasets, and the experimental results show that RMoK achieves the best performance in most cases. And we find the relationship between temporal feature weights and data periodicity through visualization, which roughly explains RMoK's mechanism. Thus, we conclude that KAN and KAN-based models (RMoK) are effective in time series forecasting. Code is available at KAN4TSF: this https URL.

Title: Improving Out-of-Distribution Data Handling and Corruption Resistance via Modern Hopfield Networks

• Authors: Saleh Sargolzaei, Luis Rueda
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11309
• Pdf URL: https://arxiv.org/pdf/2408.11309
• Copy Paste: [[2408.11309]] Improving Out-of-Distribution Data Handling and Corruption Resistance via Modern Hopfield Networks(https://arxiv.org/abs/2408.11309)
• Keywords: defense, robust
• Abstract: This study explores the potential of Modern Hopfield Networks (MHN) in improving the ability of computer vision models to handle out-of-distribution data. While current computer vision models can generalize to unseen samples from the same distribution, they are susceptible to minor perturbations such as blurring, which limits their effectiveness in real-world applications. We suggest integrating MHN into the baseline models to enhance their robustness. This integration can be implemented during the test time for any model and combined with any adversarial defense method. Our research shows that the proposed integration consistently improves model performance on the MNIST-C dataset, achieving a state-of-the-art increase of 13.84% in average corruption accuracy, a 57.49% decrease in mean Corruption Error (mCE), and a 60.61% decrease in relative mCE compared to the baseline model. Additionally, we investigate the capability of MHN to converge to the original non-corrupted data. Notably, our method does not require test-time adaptation or augmentation with corruptions, underscoring its practical viability for real-world deployment. (Source code publicly available at: this https URL)

Title: TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models

• Authors: Hyeongmin Lee, Jin-Young Kim, Kyungjune Baek, Jihwan Kim, Hyojun Go, Seongsu Ha, Seokjin Han, Jiho Jang, Raehyuk Jung, Daewoo Kim, GeunOh Kim, JongMok Kim, Jongseok Kim, Junwan Kim, Soonwoo Kwon, Jangwon Lee, Seungjoon Park, Minjoon Seo, Jay Suh, Jaehyuk Yi, Aiden Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11318
• Pdf URL: https://arxiv.org/pdf/2408.11318
• Copy Paste: [[2408.11318]] TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models(https://arxiv.org/abs/2408.11318)
• Keywords: robust, fair
• Abstract: In this work, we discuss evaluating video foundation models in a fair and robust manner. Unlike language or image foundation models, many video foundation models are evaluated with differing parameters (such as sampling rate, number of frames, pretraining steps, etc.), making fair and robust comparisons challenging. Therefore, we present a carefully designed evaluation framework for measuring two core capabilities of video comprehension: appearance and motion understanding. Our findings reveal that existing video foundation models, whether text-supervised like UMT or InternVideo2, or self-supervised like V-JEPA, exhibit limitations in at least one of these capabilities. As an alternative, we introduce TWLV-I, a new video foundation model that constructs robust visual representations for both motion- and appearance-based videos. Based on the average top-1 accuracy of linear probing on five action recognition benchmarks, pretrained only on publicly accessible datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a 7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a 2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from videos of several commonly used video benchmarks, along with evaluation source code that can directly utilize these embeddings. The code is available on "this https URL.

Title: Towards Evaluating Large Language Models on Sarcasm Understanding

• Authors: Yazhou Zhang, Chunwang Zou, Zheng Lian, Prayag Tiwari, Jing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11319
• Pdf URL: https://arxiv.org/pdf/2408.11319
• Copy Paste: [[2408.11319]] Towards Evaluating Large Language Models on Sarcasm Understanding(https://arxiv.org/abs/2408.11319)
• Keywords: large language model
• Abstract: In the era of large language models (LLMs), the task of ``System I''~-~the fast, unconscious, and intuitive tasks, e.g., sentiment analysis, text classification, etc., have been argued to be successfully solved. However, sarcasm, as a subtle linguistic phenomenon, often employs rhetorical devices like hyperbole and figuration to convey true sentiments and intentions, involving a higher level of abstraction than sentiment analysis. There is growing concern that the argument about LLMs' success may not be fully tenable when considering sarcasm understanding. To address this question, we select eleven SOTA LLMs and eight SOTA pre-trained language models (PLMs) and present comprehensive evaluations on six widely used benchmark datasets through different prompting approaches, i.e., zero-shot input/output (IO) prompting, few-shot IO prompting, chain of thought (CoT) prompting. Our results highlight three key findings: (1) current LLMs underperform supervised PLMs based sarcasm detection baselines across six sarcasm benchmarks. This suggests that significant efforts are still required to improve LLMs' understanding of human sarcasm. (2) GPT-4 consistently and significantly outperforms other LLMs across various prompting methods, with an average improvement of 14.0\%$\uparrow$. Claude 3 and ChatGPT demonstrate the next best performance after GPT-4. (3) Few-shot IO prompting method outperforms the other two methods: zero-shot IO and few-shot CoT. The reason is that sarcasm detection, being a holistic, intuitive, and non-rational cognitive process, is argued not to adhere to step-by-step logical reasoning, making CoT less effective in understanding sarcasm compared to its effectiveness in mathematical reasoning tasks.

Title: Design Principle Transfer in Neural Architecture Search via Large Language Models

• Authors: Xun Zhou, Liang Feng, Xingyu Wu, Zhichao Lu, Kay Chen Tan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11330
• Pdf URL: https://arxiv.org/pdf/2408.11330
• Copy Paste: [[2408.11330]] Design Principle Transfer in Neural Architecture Search via Large Language Models(https://arxiv.org/abs/2408.11330)
• Keywords: large language model
• Abstract: Transferable neural architecture search (TNAS) has been introduced to design efficient neural architectures for multiple tasks, to enhance the practical applicability of NAS in real-world scenarios. In TNAS, architectural knowledge accumulated in previous search processes is reused to warm up the architecture search for new tasks. However, existing TNAS methods still search in an extensive search space, necessitating the evaluation of numerous architectures. To overcome this challenge, this work proposes a novel transfer paradigm, i.e., design principle transfer. In this work, the linguistic description of various structural components' effects on architectural performance is termed design principles. They are learned from established architectures and then can be reused to reduce the search space by discarding unpromising architectures. Searching in the refined search space can boost both the search performance and efficiency for new NAS tasks. To this end, a large language model (LLM)-assisted design principle transfer (LAPT) framework is devised. In LAPT, LLM is applied to automatically reason the design principles from a set of given architectures, and then a principle adaptation method is applied to refine these principles progressively based on the new search results. Experimental results show that LAPT can beat the state-of-the-art TNAS methods on most tasks and achieve comparable performance on others.

Title: BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports

• Authors: Yuxuan Chen, Haoyan Yang, Hengkai Pan, Fardeen Siddiqui, Antonio Verdone, Qingyang Zhang, Sumit Chopra, Chen Zhao, Yiqiu Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11334
• Pdf URL: https://arxiv.org/pdf/2408.11334
• Copy Paste: [[2408.11334]] BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports(https://arxiv.org/abs/2408.11334)
• Keywords: privacy, protect, extraction
• Abstract: Breast ultrasound is essential for detecting and diagnosing abnormalities, with radiology reports summarizing key findings like lesion characteristics and malignancy assessments. Extracting this critical information is challenging due to the unstructured nature of these reports, with varied linguistic styles and inconsistent formatting. While proprietary LLMs like GPT-4 are effective, they are costly and raise privacy concerns when handling protected health information. This study presents a pipeline for developing an in-house LLM to extract clinical information from radiology reports. We first use GPT-4 to create a small labeled dataset, then fine-tune a Llama3-8B model on it. Evaluated on clinician-annotated reports, our model achieves an average F1 score of 84.6%, which is on par with GPT-4. Our findings demonstrate the feasibility of developing an in-house LLM that not only matches GPT-4's performance but also offers cost reductions and enhanced data privacy.

Title: FATE: Focal-modulated Attention Encoder for Temperature Prediction

• Authors: Tajamul Ashraf, Janibul Bashir
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11336
• Pdf URL: https://arxiv.org/pdf/2408.11336
• Copy Paste: [[2408.11336]] FATE: Focal-modulated Attention Encoder for Temperature Prediction(https://arxiv.org/abs/2408.11336)
• Keywords: transformer
• Abstract: One of the major challenges of the twenty-first century is climate change, evidenced by rising sea levels, melting glaciers, and increased storm frequency. Accurate temperature forecasting is vital for understanding and mitigating these impacts. Traditional data-driven models often use recurrent neural networks (RNNs) but face limitations in parallelization, especially with longer sequences. To address this, we introduce a novel approach based on the FocalNet Transformer architecture. Our Focal modulation Attention Encoder (FATE) framework operates in a multi-tensor format, utilizing tensorized modulation to capture spatial and temporal nuances in meteorological data. Comparative evaluations against existing transformer encoders, 3D CNNs, LSTM, and ConvLSTM models show that FATE excels at identifying complex patterns in temperature data. Additionally, we present a new labeled dataset, the Climate Change Parameter dataset (CCPD), containing 40 years of data from Jammu and Kashmir on seven climate-related parameters. Experiments with real-world temperature datasets from the USA, Canada, and Europe show accuracy improvements of 12\%, 23\%, and 28\%, respectively, over current state-of-the-art models. Our CCPD dataset also achieved a 24\% improvement in accuracy. To support reproducible research, we have released the source code and pre-trained FATE model at \href{this https URL}{this https URL}.

Title: Clinical Context-aware Radiology Report Generation from Medical Images using Transformers

• Authors: Sonit Singh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11344
• Pdf URL: https://arxiv.org/pdf/2408.11344
• Copy Paste: [[2408.11344]] Clinical Context-aware Radiology Report Generation from Medical Images using Transformers(https://arxiv.org/abs/2408.11344)
• Keywords: robust, transformer
• Abstract: Recent developments in the field of Natural Language Processing, especially language models such as the transformer have brought state-of-the-art results in language understanding and language generation. In this work, we investigate the use of the transformer model for radiology report generation from chest X-rays. We also highlight limitations in evaluating radiology report generation using only the standard language generation metrics. We then applied a transformer based radiology report generation architecture, and also compare the performance of a transformer based decoder with the recurrence based decoder. Experiments were performed using the IU-CXR dataset, showing superior results to its LSTM counterpart and being significantly faster. Finally, we identify the need of evaluating radiology report generation system using both language generation metrics and classification metrics, which helps to provide robust measure of generated reports in terms of their coherence and diagnostic value.

Title: Image Score: Learning and Evaluating Human Preferences for Mercari Search

• Authors: Chingis Oinar, Miao Cao, Shanshan Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11349
• Pdf URL: https://arxiv.org/pdf/2408.11349
• Copy Paste: [[2408.11349]] Image Score: Learning and Evaluating Human Preferences for Mercari Search(https://arxiv.org/abs/2408.11349)
• Keywords: explainability, large language model
• Abstract: Mercari is the largest C2C e-commerce marketplace in Japan, having more than 20 million active monthly users. Search being the fundamental way to discover desired items, we have always had a substantial amount of data with implicit feedback. Although we actively take advantage of that to provide the best service for our users, the correlation of implicit feedback for such tasks as image quality assessment is not trivial. Many traditional lines of research in Machine Learning (ML) are similarly motivated by the insatiable appetite of Deep Learning (DL) models for well-labelled training data. Weak supervision is about leveraging higher-level and/or noisier supervision over unlabeled data. Large Language Models (LLMs) are being actively studied and used for data labelling tasks. We present how we leverage a Chain-of-Thought (CoT) to enable LLM to produce image aesthetics labels that correlate well with human behavior in e-commerce settings. Leveraging LLMs is more cost-effective compared to explicit human judgment, while significantly improving the explainability of deep image quality evaluation which is highly important for customer journey optimization at Mercari. We propose a cost-efficient LLM-driven approach for assessing and predicting image quality in e-commerce settings, which is very convenient for proof-of-concept testing. We show that our LLM-produced labels correlate with user behavior on Mercari. Finally, we show our results from an online experimentation, where we achieved a significant growth in sales on the web platform.

Title: HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model

• Authors: Yi Wang, Jian Ma, Ruizhi Shao, Qiao Feng, Yu-kun Lai, Kun Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11357
• Pdf URL: https://arxiv.org/pdf/2408.11357
• Copy Paste: [[2408.11357]] HumanCoser: Layered 3D Human Generation via Semantic-Aware Diffusion Model(https://arxiv.org/abs/2408.11357)
• Keywords: diffusion
• Abstract: This paper aims to generate physically-layered 3D humans from text prompts. Existing methods either generate 3D clothed humans as a whole or support only tight and simple clothing generation, which limits their applications to virtual try-on and part-level editing. To achieve physically-layered 3D human generation with reusable and complex clothing, we propose a novel layer-wise dressed human representation based on a physically-decoupled diffusion model. Specifically, to achieve layer-wise clothing generation, we propose a dual-representation decoupling framework for generating clothing decoupled from the human body, in conjunction with an innovative multi-layer fusion volume rendering method. To match the clothing with different body shapes, we propose an SMPL-driven implicit field deformation network that enables the free transfer and reuse of clothing. Extensive experiments demonstrate that our approach not only achieves state-of-the-art layered 3D human generation with complex clothing but also supports virtual try-on and layered human animation.

Title: Hypergraph Learning based Recommender System for Anomaly Detection, Control and Optimization

• Authors: Sakhinana Sagar Srinivas, Rajat Kumar Sarkar, Venkataramana Runkana
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11359
• Pdf URL: https://arxiv.org/pdf/2408.11359
• Copy Paste: [[2408.11359]] Hypergraph Learning based Recommender System for Anomaly Detection, Control and Optimization(https://arxiv.org/abs/2408.11359)
• Keywords: fair
• Abstract: Anomaly detection is fundamental yet, challenging problem with practical applications in industry. The current approaches neglect the higher-order dependencies within the networks of interconnected sensors in the high-dimensional time series(multisensor data) for anomaly detection. To this end, we present a self-adapting anomaly detection framework for joint learning of (a) discrete hypergraph structure and (b) modeling the temporal trends and spatial relations among the interdependent sensors using the hierarchical encoder-decoder architecture to overcome the challenges. The hypergraph representation learning-based framework exploits the relational inductive biases in the hypergraph-structured data to learn the pointwise single-step-ahead forecasts through the self-supervised autoregressive task and predicts the anomalies based on the forecast error. Furthermore, our framework incentivizes learning the anomaly-diagnosis ontology through a differentiable approach. It derives the anomaly information propagation-based computational hypergraphs for root cause analysis and provides recommendations through an offline, optimal predictive control policy to remedy an anomaly. We conduct extensive experiments to evaluate the proposed method on the benchmark datasets for fair and rigorous comparison with the popular baselines. The proposed method outperforms the baseline models and achieves SOTA performance. We report the ablation studies to support the efficacy of the framework.

Title: Current Status and Trends in Image Anti-Forensics Research: A Bibliometric Analysis

• Authors: Yihong Lu, Jianyi Liu, Ru Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11365
• Pdf URL: https://arxiv.org/pdf/2408.11365
• Copy Paste: [[2408.11365]] Current Status and Trends in Image Anti-Forensics Research: A Bibliometric Analysis(https://arxiv.org/abs/2408.11365)
• Keywords: security, privacy
• Abstract: Image anti-forensics is a critical topic in the field of image privacy and security research. With the increasing ease of manipulating or generating human faces in images, the potential misuse of such forged images is a growing concern. This study aims to comprehensively review the knowledge structure and research hotspots related to image anti-forensics by analyzing publications in the Web of Science Core Collection (WoSCC) database. The bibliometric analysis conducted using VOSViewer software has revealed the research trends, major research institutions, most influential publications, top publishing venues, and most active contributors in this field. This is the first comprehensive bibliometric study summarizing research trends and developments in image anti-forensics. The information highlights recent and primary research directions, serving as a reference for future research in image anti-forensics.

Title: GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding

• Authors: Yibo Yan, Joey Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11366
• Pdf URL: https://arxiv.org/pdf/2408.11366
• Copy Paste: [[2408.11366]] GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding(https://arxiv.org/abs/2408.11366)
• Keywords: large language model
• Abstract: In human reading and communication, individuals tend to engage in geospatial reasoning, which involves recognizing geographic entities and making informed inferences about their interrelationships. To mimic such cognitive process, current methods either utilize conventional natural language understanding toolkits, or directly apply models pretrained on geo-related natural language corpora. However, these methods face two significant challenges: i) they do not generalize well to unseen geospatial scenarios, and ii) they overlook the importance of integrating geospatial context from geographical databases with linguistic information from the Internet. To handle these challenges, we propose GeoReasoner, a language model capable of reasoning on geospatially grounded natural language. Specifically, it first leverages Large Language Models (LLMs) to generate a comprehensive location description based on linguistic and geospatial information. It also encodes direction and distance information into spatial embedding via treating them as pseudo-sentences. Consequently, the model is trained on both anchor-level and neighbor-level inputs to learn geo-entity representation. Extensive experimental results demonstrate GeoReasoner's superiority in three tasks: toponym recognition, toponym linking, and geo-entity typing, compared to the state-of-the-art baselines.

Title: RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

• Authors: Xuanwang Zhang, Yunze Song, Yidong Wang, Shuyun Tang, Xinfeng Li, Zhengran Zeng, Zhen Wu, Wei Ye, Wenyuan Xu, Yue Zhang, Xinyu Dai, Shikun Zhang, Qingsong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11381
• Pdf URL: https://arxiv.org/pdf/2408.11381
• Copy Paste: [[2408.11381]] RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation(https://arxiv.org/abs/2408.11381)
• Keywords: fair, large language model
• Abstract: Large Language Models (LLMs) demonstrate human-level capabilities in dialogue, reasoning, and knowledge retention. However, even the most advanced LLMs face challenges such as hallucinations and real-time updating of their knowledge. Current research addresses this bottleneck by equipping LLMs with external knowledge, a technique known as Retrieval Augmented Generation (RAG). However, two key issues constrained the development of RAG. First, there is a growing lack of comprehensive and fair comparisons between novel RAG algorithms. Second, open-source tools such as LlamaIndex and LangChain employ high-level abstractions, which results in a lack of transparency and limits the ability to develop novel algorithms and evaluation metrics. To close this gap, we introduce RAGLAB, a modular and research-oriented open-source library. RAGLAB reproduces 6 existing algorithms and provides a comprehensive ecosystem for investigating RAG algorithms. Leveraging RAGLAB, we conduct a fair comparison of 6 RAG algorithms across 10 benchmarks. With RAGLAB, researchers can efficiently compare the performance of various algorithms and develop novel algorithms.

Title: On the Interchangeability of Positional Embeddings in Multilingual Neural Machine Translation Models

• Authors: Varun Gumma, Pranjal A. Chitale, Kalika Bali
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11382
• Pdf URL: https://arxiv.org/pdf/2408.11382
• Copy Paste: [[2408.11382]] On the Interchangeability of Positional Embeddings in Multilingual Neural Machine Translation Models(https://arxiv.org/abs/2408.11382)
• Keywords: large language model
• Abstract: Standard Neural Machine Translation (NMT) models have traditionally been trained with Sinusoidal Positional Embeddings (PEs), which are inadequate for capturing long-range dependencies and are inefficient for long-context or document-level translation. In contrast, state-of-the-art large language models (LLMs) employ relative PEs, demonstrating superior length generalization. This work explores the potential for efficiently switching the Positional Embeddings of pre-trained NMT models from absolute sinusoidal PEs to relative approaches such as RoPE and ALiBi. Our findings reveal that sinusoidal PEs can be effectively replaced with RoPE and ALiBi with negligible or no performance loss, achieved by fine-tuning on a small fraction of high-quality data. Additionally, models trained without Positional Embeddings (NoPE) are not a viable solution for Encoder-Decoder architectures, as they consistently under-perform compared to models utilizing any form of Positional Embedding. Furthermore, even a model trained from scratch with these relative PEs slightly under-performs a fine-tuned model, underscoring the efficiency and validity of our hypothesis.

Title: Fairness measures for biometric quality assessment

• Authors: André Dörsch, Torsten Schlett, Peter Munch, Christian Rathgeb, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11392
• Pdf URL: https://arxiv.org/pdf/2408.11392
• Copy Paste: [[2408.11392]] Fairness measures for biometric quality assessment(https://arxiv.org/abs/2408.11392)
• Keywords: biometric, fair
• Abstract: Quality assessment algorithms measure the quality of a captured biometric sample. Since the sample quality strongly affects the recognition performance of a biometric system, it is essential to only process samples of sufficient quality and discard samples of low-quality. Even though quality assessment algorithms are not intended to yield very different quality scores across demographic groups, quality score discrepancies are possible, resulting in different discard ratios. To ensure that quality assessment algorithms do not take demographic characteristics into account when assessing sample quality and consequently to ensure that the quality algorithms perform equally for all individuals, it is crucial to develop a fairness measure. In this work we propose and compare multiple fairness measures for evaluating quality components across demographic groups. Proposed measures, could be used as potential candidates for an upcoming standard in this important field.

Title: First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

• Authors: Chi Ma, Mincong Huang, Ying Zhang, Chao Wang, Yujie Wang, Lei Yu, Chuan Liu, Wei Lin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11393
• Pdf URL: https://arxiv.org/pdf/2408.11393
• Copy Paste: [[2408.11393]] First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models(https://arxiv.org/abs/2408.11393)
• Keywords: robust, large language model
• Abstract: Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25\% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness.

Title: MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

• Authors: Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, Jiajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11396
• Pdf URL: https://arxiv.org/pdf/2408.11396
• Copy Paste: [[2408.11396]] MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing(https://arxiv.org/abs/2408.11396)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at this https URL.

Title: EAGLE: Elevating Geometric Reasoning through LLM-empowered Visual Instruction Tuning

• Authors: Zhihao Li, Yao Du, Yang Liu, Yan Zhang, Yufang Liu, Mengdi Zhang, Xunliang Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11397
• Pdf URL: https://arxiv.org/pdf/2408.11397
• Copy Paste: [[2408.11397]] EAGLE: Elevating Geometric Reasoning through LLM-empowered Visual Instruction Tuning(https://arxiv.org/abs/2408.11397)
• Keywords: large language model
• Abstract: Multi-modal Large Language Models have recently experienced rapid developments and excel in various multi-modal tasks. However, they still struggle with mathematical geometric problem solving, which requires exceptional visual perception proficiency. Existing MLLMs mostly optimize the LLM backbone to acquire geometric reasoning capabilities, while rarely emphasizing improvements in visual comprehension. In this paper, we first investigate the visual perception performance of MLLMs when facing geometric diagrams. Our findings reveal that current MLLMs severely suffer from inaccurate geometric perception and hallucinations. To address these limitations, we propose EAGLE, a novel two-stage end-to-end visual enhancement MLLM framework designed to ElevAte Geometric reasoning through LLM-Empowered visual instruction tuning. Specifically, in the preliminary stage, we feed geometric image-caption pairs into our MLLM that contains a fully fine-tuning CLIP ViT and a frozen LLM, aiming to endow our model with basic geometric knowledge. In the subsequent advanced stage, we incorporate LoRA modules into the vision encoder and unfreeze the LLM backbone. This enables the model to leverage the inherent CoT rationales within question-answer pairs, guiding the MLLM to focus on nuanced visual cues and enhancing its overall perceptual capacity. Moreover, we optimize the cross-modal projector in both stages to foster adaptive visual-linguistic alignments. After the two-stage visual enhancement, we develop the geometry expert model EAGLE-7B. Extensive experiments on popular benchmarks demonstrate the effectiveness of our model. For example, on the GeoQA benchmark, EAGLE-7B not only surpasses the exemplary G-LLaVA 7B model by 2.9%, but also marginally outperforms the larger G-LLaVA 13B model. On the MathVista benchmark, EAGLE-7B achieves remarkable 3.8% improvements compared with the proprietary model GPT-4V.

Title: Revisiting FunnyBirds evaluation framework for prototypical parts networks

• Authors: Szymon Opłatek, Dawid Rymarczyk, Bartosz Zieliński
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11401
• Pdf URL: https://arxiv.org/pdf/2408.11401
• Copy Paste: [[2408.11401]] Revisiting FunnyBirds evaluation framework for prototypical parts networks(https://arxiv.org/abs/2408.11401)
• Keywords: explainability
• Abstract: Prototypical parts networks, such as ProtoPNet, became popular due to their potential to produce more genuine explanations than post-hoc methods. However, for a long time, this potential has been strictly theoretical, and no systematic studies have existed to support it. That changed recently with the introduction of the FunnyBirds benchmark, which includes metrics for evaluating different aspects of explanations. However, this benchmark employs attribution maps visualization for all explanation techniques except for the ProtoPNet, for which the bounding boxes are used. This choice significantly influences the metric scores and questions the conclusions stated in FunnyBirds publication. In this study, we comprehensively compare metric scores obtained for two types of ProtoPNet visualizations: bounding boxes and similarity maps. Our analysis indicates that employing similarity maps aligns better with the essence of ProtoPNet, as evidenced by different metric scores obtained from FunnyBirds. Therefore, we advocate using similarity maps as a visualization technique for prototypical parts networks in explainability evaluation benchmarks.

Title: Video Diffusion Models are Strong Video Inpainter

• Authors: Minhyeok Lee, Suhwan Cho, Chajin Shin, Jungho Lee, Sunghun Yang, Sangyoun Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11402
• Pdf URL: https://arxiv.org/pdf/2408.11402
• Copy Paste: [[2408.11402]] Video Diffusion Models are Strong Video Inpainter(https://arxiv.org/abs/2408.11402)
• Keywords: robust, diffusion
• Abstract: Propagation-based video inpainting using optical flow at the pixel or feature level has recently garnered significant attention. However, it has limitations such as the inaccuracy of optical flow prediction and the propagation of noise over time. These issues result in non-uniform noise and time consistency problems throughout the video, which are particularly pronounced when the removed area is large and involves substantial movement. To address these issues, we propose a novel First Frame Filling Video Diffusion Inpainting model (FFF-VDI). We design FFF-VDI inspired by the capabilities of pre-trained image-to-video diffusion models that can transform the first frame image into a highly natural video. To apply this to the video inpainting task, we propagate the noise latent information of future frames to fill the masked areas of the first frame's noise latent code. Next, we fine-tune the pre-trained image-to-video diffusion model to generate the inpainted video. The proposed model addresses the limitations of existing methods that rely on optical flow quality, producing much more natural and temporally consistent videos. This proposed approach is the first to effectively integrate image-to-video diffusion models into video inpainting tasks. Through various comparative experiments, we demonstrate that the proposed model can robustly handle diverse inpainting types with high quality.

Title: Latent Feature and Attention Dual Erasure Attack against Multi-View Diffusion Models for 3D Assets Protection

• Authors: Jingwei Sun, Xuchong Zhang, Changfeng Sun, Qicheng Bai, Hongbin Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11408
• Pdf URL: https://arxiv.org/pdf/2408.11408
• Copy Paste: [[2408.11408]] Latent Feature and Attention Dual Erasure Attack against Multi-View Diffusion Models for 3D Assets Protection(https://arxiv.org/abs/2408.11408)
• Keywords: protect, defense, attack, robust, diffusion
• Abstract: Multi-View Diffusion Models (MVDMs) enable remarkable improvements in the field of 3D geometric reconstruction, but the issue regarding intellectual property has received increasing attention due to unauthorized imitation. Recently, some works have utilized adversarial attacks to protect copyright. However, all these works focus on single-image generation tasks which only need to consider the inner feature of images. Previous methods are inefficient in attacking MVDMs because they lack the consideration of disrupting the geometric and visual consistency among the generated multi-view images. This paper is the first to address the intellectual property infringement issue arising from MVDMs. Accordingly, we propose a novel latent feature and attention dual erasure attack to disrupt the distribution of latent feature and the consistency across the generated images from multi-view and multi-domain simultaneously. The experiments conducted on SOTA MVDMs indicate that our approach achieves superior performances in terms of attack effectiveness, transferability, and robustness against defense methods. Therefore, this paper provides an efficient solution to protect 3D assets from MVDMs-based 3D geometry reconstruction.

Title: Linear-time One-Class Classification with Repeated Element-wise Folding

• Authors: Jenni Raitoharju
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11412
• Pdf URL: https://arxiv.org/pdf/2408.11412
• Copy Paste: [[2408.11412]] Linear-time One-Class Classification with Repeated Element-wise Folding(https://arxiv.org/abs/2408.11412)
• Keywords: robust
• Abstract: This paper proposes an easy-to-use method for one-class classification: Repeated Element-wise Folding (REF). The algorithm consists of repeatedly standardizing and applying an element-wise folding operation on the one-class training data. Equivalent mappings are performed on unknown test items and the classification prediction is based on the item's distance to the origin of the final distribution. As all the included operations have linear time complexity, the proposed algorithm provides a linear-time alternative for the commonly used computationally much more demanding approaches. Furthermore, REF can avoid the challenges of hyperparameter setting in one-class classification by providing robust default settings. The experiments show that the proposed method can produce similar classification performance or even outperform the more complex algorithms on various benchmark datasets. Matlab codes for REF are publicly available at this https URL.

Title: Pano2Room: Novel View Synthesis from a Single Indoor Panorama

• Authors: Guo Pu, Yiming Zhao, Zhouhui Lian
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.11413
• Pdf URL: https://arxiv.org/pdf/2408.11413
• Copy Paste: [[2408.11413]] Pano2Room: Novel View Synthesis from a Single Indoor Panorama(https://arxiv.org/abs/2408.11413)
• Keywords: generative
• Abstract: Recent single-view 3D generative methods have made significant advancements by leveraging knowledge distilled from extensive 3D object datasets. However, challenges persist in the synthesis of 3D scenes from a single view, primarily due to the complexity of real-world environments and the limited availability of high-quality prior resources. In this paper, we introduce a novel approach called Pano2Room, designed to automatically reconstruct high-quality 3D indoor scenes from a single panoramic image. These panoramic images can be easily generated using a panoramic RGBD inpainter from captures at a single location with any camera. The key idea is to initially construct a preliminary mesh from the input panorama, and iteratively refine this mesh using a panoramic RGBD inpainter while collecting photo-realistic 3D-consistent pseudo novel views. Finally, the refined mesh is converted into a 3D Gaussian Splatting field and trained with the collected pseudo novel views. This pipeline enables the reconstruction of real-world 3D scenes, even in the presence of large occlusions, and facilitates the synthesis of photo-realistic novel views with detailed geometry. Extensive qualitative and quantitative experiments have been conducted to validate the superiority of our method in single-panorama indoor novel synthesis compared to the state-of-the-art. Our code and data are available at \url{this https URL}.

Title: Towards "Differential AI Psychology" and in-context Value-driven Statement Alignment with Moral Foundations Theory

• Authors: Simon Münker
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11415
• Pdf URL: https://arxiv.org/pdf/2408.11415
• Copy Paste: [[2408.11415]] Towards "Differential AI Psychology" and in-context Value-driven Statement Alignment with Moral Foundations Theory(https://arxiv.org/abs/2408.11415)
• Keywords: generative
• Abstract: Contemporary research in social sciences is increasingly utilizing state-of-the-art statistical language models to annotate or generate content. While these models perform benchmark-leading on common language tasks and show exemplary task-independent emergent abilities, transferring them to novel out-of-domain tasks is only insufficiently explored. The implications of the statistical black-box approach - stochastic parrots - are prominently criticized in the language model research community; however, the significance for novel generative tasks is not. This work investigates the alignment between personalized language models and survey participants on a Moral Foundation Theory questionnaire. We adapt text-to-text models to different political personas and survey the questionnaire repetitively to generate a synthetic population of persona and model combinations. Analyzing the intra-group variance and cross-alignment shows significant differences across models and personas. Our findings indicate that adapted models struggle to represent the survey-captured assessment of political ideologies. Thus, using language models to mimic social interactions requires measurable improvements in in-context optimization or parameter manipulation to align with psychological and sociological stereotypes. Without quantifiable alignment, generating politically nuanced content remains unfeasible. To enhance these representations, we propose a testable framework to generate agents based on moral value statements for future research.

Title: EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning

• Authors: Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, Heikki Kälviäinen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11424
• Pdf URL: https://arxiv.org/pdf/2408.11424
• Copy Paste: [[2408.11424]] EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning(https://arxiv.org/abs/2408.11424)
• Keywords: large language model
• Abstract: Facial expression recognition (FER) is an important research topic in emotional artificial intelligence. In recent decades, researchers have made remarkable progress. However, current FER paradigms face challenges in generalization, lack semantic information aligned with natural language, and struggle to process both images and videos within a unified framework, making their application in multimodal emotion understanding and human-computer interaction difficult. Multimodal Large Language Models (MLLMs) have recently achieved success, offering advantages in addressing these issues and potentially overcoming the limitations of current FER paradigms. However, directly applying pre-trained MLLMs to FER still faces several challenges. Our zero-shot evaluations of existing open-source MLLMs on FER indicate a significant performance gap compared to GPT-4V and current supervised state-of-the-art (SOTA) methods. In this paper, we aim to enhance MLLMs' capabilities in understanding facial expressions. We first generate instruction data for five FER datasets with Gemini. We then propose a novel MLLM, named EMO-LLaMA, which incorporates facial priors from a pretrained facial analysis network to enhance human facial information. Specifically, we design a Face Info Mining module to extract both global and local facial information. Additionally, we utilize a handcrafted prompt to introduce age-gender-race attributes, considering the emotional differences across different human groups. Extensive experiments show that EMO-LLaMA achieves SOTA-comparable or competitive results across both static and dynamic FER datasets. The instruction dataset and code are available at this https URL.

Title: Diagnosing and Remedying Knowledge Deficiencies in LLMs via Label-free Curricular Meaningful Learning

• Authors: Kai Xiong, Xiao Ding, Li Du, Jiahao Ying, Ting Liu, Bing Qin, Yixin Cao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11431
• Pdf URL: https://arxiv.org/pdf/2408.11431
• Copy Paste: [[2408.11431]] Diagnosing and Remedying Knowledge Deficiencies in LLMs via Label-free Curricular Meaningful Learning(https://arxiv.org/abs/2408.11431)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are versatile and demonstrate impressive generalization ability by mining and learning information from extensive unlabeled text. However, they still exhibit reasoning mistakes, often stemming from knowledge deficiencies, which can affect their trustworthiness and reliability. Although users can provide diverse and comprehensive queries, obtaining sufficient and effective feedback is demanding. Furthermore, evaluating LLMs comprehensively with limited labeled samples is difficult. This makes it a challenge to diagnose and remedy the deficiencies of LLMs through rich label-free user queries. To tackle this challenge, we propose a label-free curricular meaningful learning framework (LaMer). LaMer first employs relative entropy to automatically diagnose and quantify the knowledge deficiencies of LLMs in a label-free setting. Next, to remedy the diagnosed knowledge deficiencies, we apply curricular meaningful learning: first, we adopt meaningful learning to adaptively synthesize augmentation data according to the severity of the deficiencies, and then design a curricular deficiency remedy strategy to remedy the knowledge deficiencies of LLMs progressively. Experiments show that LaMer efficiently and effectively diagnoses and remedies knowledge deficiencies in LLMs, improving various LLMs across seven out-of-distribution (OOD) reasoning and language understanding benchmarks, achieving comparable results to baselines with just 40\% training data. LaMer even surpasses methods that rely on labeled datasets for deficiency diagnosis. In application, our label-free method can offer an effective knowledge deficiency diagnostic tool for efficient LLM development.

Title: T2VIndexer: A Generative Video Indexer for Efficient Text-Video Retrieval

• Authors: Yili Li, Jing Yu, Keke Gai, Bang Liu, Gang Xiong, Qi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11432
• Pdf URL: https://arxiv.org/pdf/2408.11432
• Copy Paste: [[2408.11432]] T2VIndexer: A Generative Video Indexer for Efficient Text-Video Retrieval(https://arxiv.org/abs/2408.11432)
• Keywords: generative
• Abstract: Current text-video retrieval methods mainly rely on cross-modal matching between queries and videos to calculate their similarity scores, which are then sorted to obtain retrieval results. This method considers the matching between each candidate video and the query, but it incurs a significant time cost and will increase notably with the increase of candidates. Generative models are common in natural language processing and computer vision, and have been successfully applied in document retrieval, but their application in multimodal retrieval remains unexplored. To enhance retrieval efficiency, in this paper, we introduce a model-based video indexer named T2VIndexer, which is a sequence-to-sequence generative model directly generating video identifiers and retrieving candidate videos with constant time complexity. T2VIndexer aims to reduce retrieval time while maintaining high accuracy. To achieve this goal, we propose video identifier encoding and query-identifier augmentation approaches to represent videos as short sequences while preserving their semantic information. Our method consistently enhances the retrieval efficiency of current state-of-the-art models on four standard datasets. It enables baselines with only 30\%-50\% of the original retrieval time to achieve better retrieval performance on MSR-VTT (+1.0%), MSVD (+1.8%), ActivityNet (+1.5%), and DiDeMo (+0.2%). The code is available at this https URL.

Title: Towards Aligned Data Removal via Twin Machine Unlearning

• Authors: Yuyao Sun, Zhenxing Niu, Gang hua, Rong jin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11433
• Pdf URL: https://arxiv.org/pdf/2408.11433
• Copy Paste: [[2408.11433]] Towards Aligned Data Removal via Twin Machine Unlearning(https://arxiv.org/abs/2408.11433)
• Keywords: privacy
• Abstract: Modern privacy regulations have spurred the evolution of machine unlearning, a technique that enables the removal of data from an already trained ML model without requiring retraining from scratch. Previous unlearning methods tend to induce the model to achieve lowest classification accuracy on the removal data. Nonetheless, the authentic objective of machine unlearning is to align the unlearned model with the gold model, i.e., achieving the same classification accuracy as the gold model. For this purpose, we present a Twin Machine Unlearning (TMU) approach, where a twin unlearning problem is defined corresponding to the original unlearning problem. As a results, the generalization-label predictor trained on the twin problem can be transferred to the original problem, facilitating aligned data removal. Comprehensive empirical experiments illustrate that our approach significantly enhances the alignment between the unlearned model and the gold model. Meanwhile, our method allows data removal without compromising the model accuracy.

Title: DABench: A Benchmark Dataset for Data-Driven Weather Data Assimilation

• Authors: Wuxin Wang, Weicheng Ni, Tao Han, Lei Bai, Boheng Duan, Kaijun Ren
• Subjects: cs.LG, cs.CV, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2408.11438
• Pdf URL: https://arxiv.org/pdf/2408.11438
• Copy Paste: [[2408.11438]] DABench: A Benchmark Dataset for Data-Driven Weather Data Assimilation(https://arxiv.org/abs/2408.11438)
• Keywords: fair, transformer
• Abstract: Recent advancements in deep learning (DL) have led to the development of several Large Weather Models (LWMs) that rival state-of-the-art (SOTA) numerical weather prediction (NWP) systems. Up to now, these models still rely on traditional NWP-generated analysis fields as input and are far from being an autonomous system. While researchers are exploring data-driven data assimilation (DA) models to generate accurate initial fields for LWMs, the lack of a standard benchmark impedes the fair evaluation among different data-driven DA algorithms. Here, we introduce DABench, a benchmark dataset utilizing ERA5 data as ground truth to guide the development of end-to-end data-driven weather prediction systems. DABench contributes four standard features: (1) sparse and noisy simulated observations under the guidance of the observing system simulation experiment method; (2) a skillful pre-trained weather prediction model to generate background fields while fairly evaluating the impact of assimilation outcomes on predictions; (3) standardized evaluation metrics for model comparison; (4) a strong baseline called the DA Transformer (DaT). DaT integrates the four-dimensional variational DA prior knowledge into the Transformer model and outperforms the SOTA in physical state reconstruction, named 4DVarNet. Furthermore, we exemplify the development of an end-to-end data-driven weather prediction system by integrating DaT with the prediction model. Researchers can leverage DABench to develop their models and compare performance against established baselines, which will benefit the future advancements of data-driven weather prediction systems. The code is available on this Github repository and the dataset is available at the Baidu Drive.

Title: BAdd: Bias Mitigation through Bias Addition

• Authors: Ioannis Sarridis, Christos Koutlis, Symeon Papadopoulos, Christos Diou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11439
• Pdf URL: https://arxiv.org/pdf/2408.11439
• Copy Paste: [[2408.11439]] BAdd: Bias Mitigation through Bias Addition(https://arxiv.org/abs/2408.11439)
• Keywords: fair
• Abstract: Computer vision (CV) datasets often exhibit biases that are perpetuated by deep learning models. While recent efforts aim to mitigate these biases and foster fair representations, they fail in complex real-world scenarios. In particular, existing methods excel in controlled experiments involving benchmarks with single-attribute injected biases, but struggle with multi-attribute biases being present in well-established CV datasets. Here, we introduce BAdd, a simple yet effective method that allows for learning fair representations invariant to the attributes introducing bias by incorporating features representing these attributes into the backbone. BAdd is evaluated on seven benchmarks and exhibits competitive performance, surpassing state-of-the-art methods on both single- and multi-attribute benchmarks. Notably, BAdd achieves +27.5% and +5.5% absolute accuracy improvements on the challenging multi-attribute benchmarks, FB-Biased-MNIST and CelebA, respectively.

Title: LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems

• Authors: Tahir Javed, Janki Nawale, Sakshi Joshi, Eldho George, Kaushal Bhogale, Deovrat Mehendale, Mitesh M. Khapra
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11440
• Pdf URL: https://arxiv.org/pdf/2408.11440
• Copy Paste: [[2408.11440]] LAHAJA: A Robust Multi-accent Benchmark for Evaluating Hindi ASR Systems(https://arxiv.org/abs/2408.11440)
• Keywords: robust
• Abstract: Hindi, one of the most spoken language of India, exhibits a diverse array of accents due to its usage among individuals from diverse linguistic origins. To enable a robust evaluation of Hindi ASR systems on multiple accents, we create a benchmark, LAHAJA, which contains read and extempore speech on a diverse set of topics and use cases, with a total of 12.5 hours of Hindi audio, sourced from 132 speakers spanning 83 districts of India. We evaluate existing open-source and commercial models on LAHAJA and find their performance to be poor. We then train models using different datasets and find that our model trained on multilingual data with good speaker diversity outperforms existing models by a significant margin. We also present a fine-grained analysis which shows that the performance declines for speakers from North-East and South India, especially with content heavy in named entities and specialized terminology.

Title: A Practical Trigger-Free Backdoor Attack on Neural Networks

• Authors: Jiahao Wang, Xianglong Zhang, Xiuzhen Cheng, Pengfei Hu, Guoming Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11444
• Pdf URL: https://arxiv.org/pdf/2408.11444
• Copy Paste: [[2408.11444]] A Practical Trigger-Free Backdoor Attack on Neural Networks(https://arxiv.org/abs/2408.11444)
• Keywords: security, attack, steal
• Abstract: Backdoor attacks on deep neural networks have emerged as significant security threats, especially as DNNs are increasingly deployed in security-critical applications. However, most existing works assume that the attacker has access to the original training data. This limitation restricts the practicality of launching such attacks in real-world scenarios. Additionally, using a specified trigger to activate the injected backdoor compromises the stealthiness of the attacks. To address these concerns, we propose a trigger-free backdoor attack that does not require access to any training data. Specifically, we design a novel fine-tuning approach that incorporates the concept of malicious data into the concept of the attacker-specified class, resulting the misclassification of trigger-free malicious data into the attacker-specified class. Furthermore, instead of relying on training data to preserve the model's knowledge, we employ knowledge distillation methods to maintain the performance of the infected model on benign samples, and introduce a parameter importance evaluation mechanism based on elastic weight constraints to facilitate the fine-tuning of the infected model. The effectiveness, practicality, and stealthiness of the proposed attack are comprehensively evaluated on three real-world datasets. Furthermore, we explore the potential for enhancing the attack through the use of auxiliary datasets and model inversion.

Title: Lookism: The overlooked bias in computer vision

• Authors: Aditya Gulati, Bruno Lepri, Nuria Oliver
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.11448
• Pdf URL: https://arxiv.org/pdf/2408.11448
• Copy Paste: [[2408.11448]] Lookism: The overlooked bias in computer vision(https://arxiv.org/abs/2408.11448)
• Keywords: security, fair
• Abstract: In recent years, there have been significant advancements in computer vision which have led to the widespread deployment of image recognition and generation systems in socially relevant applications, from hiring to security screening. However, the prevalence of biases within these systems has raised significant ethical and social concerns. The most extensively studied biases in this context are related to gender, race and age. Yet, other biases are equally pervasive and harmful, such as lookism, i.e., the preferential treatment of individuals based on their physical appearance. Lookism remains under-explored in computer vision but can have profound implications not only by perpetuating harmful societal stereotypes but also by undermining the fairness and inclusivity of AI technologies. Thus, this paper advocates for the systematic study of lookism as a critical bias in computer vision models. Through a comprehensive review of existing literature, we identify three areas of intersection between lookism and computer vision. We illustrate them by means of examples and a user study. We call for an interdisciplinary approach to address lookism, urging researchers, developers, and policymakers to prioritize the development of equitable computer vision systems that respect and reflect the diversity of human appearances.

Title: Using Part-based Representations for Explainable Deep Reinforcement Learning

• Authors: Manos Kirtas, Konstantinos Tsampazis, Loukia Avramelou, Nikolaos Passalis, Nikolaos Passalis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11455
• Pdf URL: https://arxiv.org/pdf/2408.11455
• Copy Paste: [[2408.11455]] Using Part-based Representations for Explainable Deep Reinforcement Learning(https://arxiv.org/abs/2408.11455)
• Keywords: extraction, interpretability
• Abstract: Utilizing deep learning models to learn part-based representations holds significant potential for interpretable-by-design approaches, as these models incorporate latent causes obtained from feature representations through simple addition. However, training a part-based learning model presents challenges, particularly in enforcing non-negative constraints on the model's parameters, which can result in training difficulties such as instability and convergence issues. Moreover, applying such approaches in Deep Reinforcement Learning (RL) is even more demanding due to the inherent instabilities that impact many optimization methods. In this paper, we propose a non-negative training approach for actor models in RL, enabling the extraction of part-based representations that enhance interpretability while adhering to non-negative constraints. To this end, we employ a non-negative initialization technique, as well as a modified sign-preserving training method, which can ensure better gradient flow compared to existing approaches. We demonstrate the effectiveness of the proposed approach using the well-known Cartpole benchmark.

Title: MambaOcc: Visual State Space Model for BEV-based Occupancy Prediction with Local Adaptive Reordering

• Authors: Yonglin Tian, Songlin Bai, Zhiyao Luo, Yutong Wang, Yisheng Lv, Fei-Yue Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11464
• Pdf URL: https://arxiv.org/pdf/2408.11464
• Copy Paste: [[2408.11464]] MambaOcc: Visual State Space Model for BEV-based Occupancy Prediction with Local Adaptive Reordering(https://arxiv.org/abs/2408.11464)
• Keywords: transformer
• Abstract: Occupancy prediction has attracted intensive attention and shown great superiority in the development of autonomous driving systems. The fine-grained environmental representation brought by occupancy prediction in terms of both geometry and semantic information has facilitated the general perception and safe planning under open scenarios. However, it also brings high computation costs and heavy parameters in existing works that utilize voxel-based 3d dense representation and Transformer-based quadratic attention. To address these challenges, in this paper, we propose a Mamba-based occupancy prediction method (MambaOcc) adopting BEV features to ease the burden of 3D scenario representation, and linear Mamba-style attention to achieve efficient long-range perception. Besides, to address the sensitivity of Mamba to sequence order, we propose a local adaptive reordering (LAR) mechanism with deformable convolution and design a hybrid BEV encoder comprised of convolution layers and Mamba. Extensive experiments on the Occ3D-nuScenes dataset demonstrate that MambaOcc achieves state-of-the-art performance in terms of both accuracy and computational efficiency. For example, compared to FlashOcc, MambaOcc delivers superior results while reducing the number of parameters by 42\% and computational costs by 39\%. Code will be available at this https URL.

Title: MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation

• Authors: Kim Yu-Ji, Hyunwoo Ha, Kim Youwang, Jaeheung Surh, Hyowon Ha, Tae-Hyun Oh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11465
• Pdf URL: https://arxiv.org/pdf/2408.11465
• Copy Paste: [[2408.11465]] MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation(https://arxiv.org/abs/2408.11465)
• Keywords: generative
• Abstract: Reconstructing 3D from a single view image is a long-standing challenge. One of the popular approaches to tackle this problem is learning-based methods, but dealing with the test cases unfamiliar with training data (Out-of-distribution; OoD) introduces an additional challenge. To adapt for unseen samples in test time, we propose MeTTA, a test-time adaptation (TTA) exploiting generative prior. We design joint optimization of 3D geometry, appearance, and pose to handle OoD cases with only a single view image. However, the alignment between the reference image and the 3D shape via the estimated viewpoint could be erroneous, which leads to ambiguity. To address this ambiguity, we carefully design learnable virtual cameras and their self-calibration. In our experiments, we demonstrate that MeTTA effectively deals with OoD scenarios at failure cases of existing learning-based 3D reconstruction models and enables obtaining a realistic appearance with physically based rendering (PBR) textures.

Title: TrackGo: A Flexible and Efficient Method for Controllable Video Generation

• Authors: Haitao Zhou, Chuang Wang, Rui Nie, Jinxiao Lin, Dongdong Yu, Qian Yu, Changhu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11475
• Pdf URL: https://arxiv.org/pdf/2408.11475
• Copy Paste: [[2408.11475]] TrackGo: A Flexible and Efficient Method for Controllable Video Generation(https://arxiv.org/abs/2408.11475)
• Keywords: diffusion
• Abstract: Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation. This method offers users with a flexible and precise mechanism for manipulating video content. We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers of a pretrained video generation model. This design leverages our observation that the attention map of these layers can accurately activate regions corresponding to motion in videos. Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo can be found at: this https URL

Title: LAKD-Activation Mapping Distillation Based on Local Learning

• Authors: Yaoze Zhang, Yuming Zhang, Yu Zhao, Yue Zhang, Feiyu Zhu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11478
• Pdf URL: https://arxiv.org/pdf/2408.11478
• Copy Paste: [[2408.11478]] LAKD-Activation Mapping Distillation Based on Local Learning(https://arxiv.org/abs/2408.11478)
• Keywords: interpretability
• Abstract: Knowledge distillation is widely applied in various fundamental vision models to enhance the performance of compact models. Existing knowledge distillation methods focus on designing different distillation targets to acquire knowledge from teacher models. However, these methods often overlook the efficient utilization of distilled information, crudely coupling different types of information, making it difficult to explain how the knowledge from the teacher network aids the student network in learning. This paper proposes a novel knowledge distillation framework, Local Attention Knowledge Distillation (LAKD), which more efficiently utilizes the distilled information from teacher networks, achieving higher interpretability and competitive performance. The framework establishes an independent interactive training mechanism through a separation-decoupling mechanism and non-directional activation mapping. LAKD decouples the teacher's features and facilitates progressive interaction training from simple to complex. Specifically, the student network is divided into local modules with independent gradients to decouple the knowledge transferred from the teacher. The non-directional activation mapping helps the student network integrate knowledge from different local modules by learning coarse-grained feature knowledge. We conducted experiments on the CIFAR-10, CIFAR-100, and ImageNet datasets, and the results show that our LAKD method significantly outperforms existing methods, consistently achieving state-of-the-art performance across different datasets.

Title: Learning Deep Dissipative Dynamics

• Authors: Yuji Okamoto, Ryosuke Kojima
• Subjects: cs.LG, eess.SY, math.DS
• Abstract URL: https://arxiv.org/abs/2408.11479
• Pdf URL: https://arxiv.org/pdf/2408.11479
• Copy Paste: [[2408.11479]] Learning Deep Dissipative Dynamics(https://arxiv.org/abs/2408.11479)
• Keywords: robust
• Abstract: This study challenges strictly guaranteeing ``dissipativity'' of a dynamical system represented by neural networks learned from given time-series data. Dissipativity is a crucial indicator for dynamical systems that generalizes stability and input-output stability, known to be valid across various systems including robotics, biological systems, and molecular dynamics. By analytically proving the general solution to the nonlinear Kalman-Yakubovich-Popov (KYP) lemma, which is the necessary and sufficient condition for dissipativity, we propose a differentiable projection that transforms any dynamics represented by neural networks into dissipative ones and a learning method for the transformed dynamics. Utilizing the generality of dissipativity, our method strictly guarantee stability, input-output stability, and energy conservation of trained dynamical systems. Finally, we demonstrate the robustness of our method against out-of-domain input through applications to robotic arms and fluid dynamics. Code here this https URL

Title: Security Evaluation in Software-Defined Networks

• Authors: Igor Ivkić, Dominik Thiede, Nicholas Race, Matthew Broadbent, Antonios Gouglidis
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.11486
• Pdf URL: https://arxiv.org/pdf/2408.11486
• Copy Paste: [[2408.11486]] Security Evaluation in Software-Defined Networks(https://arxiv.org/abs/2408.11486)
• Keywords: security
• Abstract: Cloud computing has grown in importance in recent years which has led to a significant increase in Data Centre (DC) network requirements. A major driver of this change is virtualisation, which allows computing resources to be deployed on a large scale. However, traditional DCs, with their network topology and proliferation of network endpoints, are struggling to meet the flexible, centrally managed requirements of cloud computing applications. Software-Defined Networks (SDN) promise to offer a solution to these growing networking requirements by separating control functions from data routing. This shift adds more flexibility to networks but also introduces new security issues. This article presents a framework for evaluating security of SDN architectures. In addition, through an experimental study, we demonstrate how this framework can identify the threats and vulnerabilities, calculate their risks and severity, and provide the necessary measures to mitigate them. The proposed framework helps administrators to evaluate SDN security, address identified threats and meet network security requirements.

Title: DocTabQA: Answering Questions from Long Documents Using Tables

• Authors: Haochen Wang, Kai Hu, Haoyu Dong, Liangcai Gao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11490
• Pdf URL: https://arxiv.org/pdf/2408.11490
• Copy Paste: [[2408.11490]] DocTabQA: Answering Questions from Long Documents Using Tables(https://arxiv.org/abs/2408.11490)
• Keywords: large language model
• Abstract: We study a new problem setting of question answering (QA), referred to as DocTabQA. Within this setting, given a long document, the goal is to respond to questions by organizing the answers into structured tables derived directly from the document's content. Unlike traditional QA approaches which predominantly rely on unstructured text to formulate responses, DocTabQA aims to leverage structured tables as answers to convey information clearly and systematically, thereby enhancing user comprehension and highlighting relationships between data points. To the best of our knowledge, this problem has not been previously explored. In this paper, we introduce the QTabA dataset, encompassing 300 financial documents, accompanied by manually annotated 1.5k question-table pairs. Initially, we leverage Large Language Models (LLMs) such as GPT-4 to establish a baseline. However, it is widely acknowledged that LLMs encounter difficulties when tasked with generating intricate, structured outputs from long input sequences. To overcome these challenges, we present a two-stage framework, called DocTabTalk, which initially retrieves relevant sentences from extensive documents and subsequently generates hierarchical tables based on these identified sentences. DocTabTalk incorporates two key technological innovations: AlignLLaMA and TabTalk, which are specifically tailored to assist GPT-4 in tackling DocTabQA, enabling it to generate well-structured, hierarchical tables with improved organization and clarity. Comprehensive experimental evaluations conducted on both QTabA and RotoWire datasets demonstrate that our DocTabTalk significantly enhances the performances of the GPT-4 in our proposed DocTabQA task and the table generation task. The code and dataset are available at this https URL for further research.

Title: MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning

• Authors: Minghao Han, Linhao Qu, Dingkang Yang, Xukun Zhang, Xiaoying Wang, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11505
• Pdf URL: https://arxiv.org/pdf/2408.11505
• Copy Paste: [[2408.11505]] MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning(https://arxiv.org/abs/2408.11505)
• Keywords: large language model
• Abstract: Multiple instance learning (MIL) has become a standard paradigm for weakly supervised classification of whole slide images (WSI). However, this paradigm relies on the use of a large number of labelled WSIs for training. The lack of training data and the presence of rare diseases present significant challenges for these methods. Prompt tuning combined with the pre-trained Vision-Language models (VLMs) is an effective solution to the Few-shot Weakly Supervised WSI classification (FSWC) tasks. Nevertheless, applying prompt tuning methods designed for natural images to WSIs presents three significant challenges: 1) These methods fail to fully leverage the prior knowledge from the VLM's text modality; 2) They overlook the essential multi-scale and contextual information in WSIs, leading to suboptimal results; and 3) They lack exploration of instance aggregation methods. To address these problems, we propose a Multi-Scale and Context-focused Prompt Tuning (MSCPT) method for FSWC tasks. Specifically, MSCPT employs the frozen large language model to generate pathological visual language prior knowledge at multi-scale, guiding hierarchical prompt tuning. Additionally, we design a graph prompt tuning module to learn essential contextual information within WSI, and finally, a non-parametric cross-guided instance aggregation module has been introduced to get the WSI-level features. Based on two VLMs, extensive experiments and visualizations on three datasets demonstrated the powerful performance of our MSCPT.

Title: IKUN for WMT24 General MT Task: LLMs Are here for Multilingual Machine Translation

• Authors: Baohao Liao, Christian Herold, Shahram Khadivi, Christof Monz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11512
• Pdf URL: https://arxiv.org/pdf/2408.11512
• Copy Paste: [[2408.11512]] IKUN for WMT24 General MT Task: LLMs Are here for Multilingual Machine Translation(https://arxiv.org/abs/2408.11512)
• Keywords: secure, large language model
• Abstract: This paper introduces two multilingual systems, IKUN and IKUN-C, developed for the general machine translation task in WMT24. IKUN and IKUN-C represent an open system and a constrained system, respectively, built on Llama-3-8b and Mistral-7B-v0.3. Both systems are designed to handle all 11 language directions using a single model. According to automatic evaluation metrics, IKUN-C achieved 6 first-place and 3 second-place finishes among all constrained systems, while IKUN secured 1 first-place and 2 second-place finishes across both open and constrained systems. These encouraging results suggest that large language models (LLMs) are nearing the level of proficiency required for effective multilingual machine translation. The systems are based on a two-stage approach: first, continuous pre-training on monolingual data in 10 languages, followed by fine-tuning on high-quality parallel data for 11 language directions. The primary difference between IKUN and IKUN-C lies in their monolingual pre-training strategy. IKUN-C is pre-trained using constrained monolingual data, whereas IKUN leverages monolingual data from the OSCAR dataset. In the second phase, both systems are fine-tuned on parallel data sourced from NTREX, Flores, and WMT16-23 for all 11 language pairs.

Title: Imagining from Images with an AI Storytelling Tool

• Authors: Edirlei Soares de Lima, Marco A. Casanova, Antonio L. Furtado
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11517
• Pdf URL: https://arxiv.org/pdf/2408.11517
• Copy Paste: [[2408.11517]] Imagining from Images with an AI Storytelling Tool(https://arxiv.org/abs/2408.11517)
• Keywords: diffusion
• Abstract: A method for generating narratives by analyzing single images or image sequences is presented, inspired by the time immemorial tradition of Narrative Art. The proposed method explores the multimodal capabilities of GPT-4o to interpret visual content and create engaging stories, which are illustrated by a Stable Diffusion XL model. The method is supported by a fully implemented tool, called ImageTeller, which accepts images from diverse sources as input. Users can guide the narrative's development according to the conventions of fundamental genres - such as Comedy, Romance, Tragedy, Satire or Mystery -, opt to generate data-driven stories, or to leave the prototype free to decide how to handle the narrative structure. User interaction is provided along the generation process, allowing the user to request alternative chapters or illustrations, and even reject and restart the story generation based on the same input. Additionally, users can attach captions to the input images, influencing the system's interpretation of the visual content. Examples of generated stories are provided, along with details on how to access the prototype.

Title: The Vizier Gaussian Process Bandit Algorithm

• Authors: Xingyou Song, Qiuyi Zhang, Chansoo Lee, Emily Fertig, Tzu-Kuo Huang, Lior Belenki, Greg Kochanski, Setareh Ariafar, Srinivas Vasudevan, Sagi Perel, Daniel Golovin
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2408.11527
• Pdf URL: https://arxiv.org/pdf/2408.11527
• Copy Paste: [[2408.11527]] The Vizier Gaussian Process Bandit Algorithm(https://arxiv.org/abs/2408.11527)
• Keywords: robust
• Abstract: Google Vizier has performed millions of optimizations and accelerated numerous research and production systems at Google, demonstrating the success of Bayesian optimization as a large-scale service. Over multiple years, its algorithm has been improved considerably, through the collective experiences of numerous research efforts and user feedback. In this technical report, we discuss the implementation details and design choices of the current default algorithm provided by Open Source Vizier. Our experiments on standardized benchmarks reveal its robustness and versatility against well-established industry baselines on multiple practical modes.

Title: SAM-REF: Rethinking Image-Prompt Synergy for Refinement in Segment Anything

• Authors: Chongkai Yu, Anqi Li, Xiaochao Qu, Luoqi Liu, Ting Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11535
• Pdf URL: https://arxiv.org/pdf/2408.11535
• Copy Paste: [[2408.11535]] SAM-REF: Rethinking Image-Prompt Synergy for Refinement in Segment Anything(https://arxiv.org/abs/2408.11535)
• Keywords: segmentation
• Abstract: The advent of the Segment Anything Model (SAM) marks a significant milestone for interactive segmentation using generalist models. As a late fusion model, SAM extracts image embeddings once and merges them with prompts in later interactions. This strategy limits the models ability to extract detailed information from the prompted target zone. Current specialist models utilize the early fusion strategy that encodes the combination of images and prompts to target the prompted objects, yet repetitive complex computations on the images result in high latency. The key to these issues is efficiently synergizing the images and prompts. We propose SAM-REF, a two-stage refinement framework that fully integrates images and prompts globally and locally while maintaining the accuracy of early fusion and the efficiency of late fusion. The first-stage GlobalDiff Refiner is a lightweight early fusion network that combines the whole image and prompts, focusing on capturing detailed information for the entire object. The second-stage PatchDiff Refiner locates the object detail window according to the mask and prompts, then refines the local details of the object. Experimentally, we demonstrated the high effectiveness and efficiency of our method in tackling complex cases with multiple interactions. Our SAM-REF model outperforms the current state-of-the-art method in most metrics on segmentation quality without compromising efficiency.

Title: UNetMamba: Efficient UNet-Like Mamba for Semantic Segmentation of High-Resolution Remote Sensing Images

• Authors: Enze Zhu, Zhan Chen, Dingkai Wang, Hanru Shi, Xiaoxuan Liu, Lei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11545
• Pdf URL: https://arxiv.org/pdf/2408.11545
• Copy Paste: [[2408.11545]] UNetMamba: Efficient UNet-Like Mamba for Semantic Segmentation of High-Resolution Remote Sensing Images(https://arxiv.org/abs/2408.11545)
• Keywords: transformer, segmentation
• Abstract: The semantic segmentation of high-resolution remote sensing images plays a crucial role in downstream applications such as urban planning and disaster assessment. However, existing Transformer-based methods suffer from the constraint between accuracy and efficiency. To overcome this dilemma, we propose UNetMamba, a novel Mamba-based semantic segmentation model. It incorporates a Mamba Segmentation Decoder (MSD) that can efficiently decode the complex information within high-resolution images, and a Local Supervision Module (LSM), which is train-only but can significantly enhance the perception of local contents. Extensive experiments demonstrate that UNet-Mamba outperforms the state-of-the-art methods with the mIoU increased by 0.87% on LoveDA and 0.36% on ISPRS Vaihingen, while achieving high efficiency through light weight, low memory footprint and low computational cost. The source code will soon be publicly available at this https URL.

Title: Memorization In In-Context Learning

• Authors: Shahriar Golchin, Mihai Surdeanu, Steven Bethard, Eduardo Blanco, Ellen Riloff
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11546
• Pdf URL: https://arxiv.org/pdf/2408.11546
• Copy Paste: [[2408.11546]] Memorization In In-Context Learning(https://arxiv.org/abs/2408.11546)
• Keywords: large language model
• Abstract: In-context learning (ICL) has proven to be an effective strategy for improving the performance of large language models (LLMs) with no additional training. However, the exact mechanism behind these performance improvements remains unclear. This study is the first to show how ICL surfaces memorized training data and to explore the correlation between this memorization and performance across various ICL regimes: zero-shot, few-shot, and many-shot. Our most notable findings include: (1) ICL significantly surfaces memorization compared to zero-shot learning in most cases; (2) demonstrations, without their labels, are the most effective element in surfacing memorization; (3) ICL improves performance when the surfaced memorization in few-shot regimes reaches a high level (about 40%); and (4) there is a very strong correlation between performance and memorization in ICL when it outperforms zero-shot learning. Overall, our study uncovers a hidden phenomenon -- memorization -- at the core of ICL, raising an important question: to what extent do LLMs truly generalize from demonstrations in ICL, and how much of their success is due to memorization?

Title: AnyDesign: Versatile Area Fashion Editing via Mask-Free Diffusion

• Authors: Yunfang Niu, Lingxiang Wu, Dong Yi, Jie Peng, Ning Jiang, Haiying Wu, Jinqiao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11553
• Pdf URL: https://arxiv.org/pdf/2408.11553
• Copy Paste: [[2408.11553]] AnyDesign: Versatile Area Fashion Editing via Mask-Free Diffusion(https://arxiv.org/abs/2408.11553)
• Keywords: diffusion
• Abstract: Fashion image editing aims to modify a person's appearance based on a given instruction. Existing methods require auxiliary tools like segmenters and keypoint extractors, lacking a flexible and unified framework. Moreover, these methods are limited in the variety of clothing types they can handle, as most datasets focus on people in clean backgrounds and only include generic garments such as tops, pants, and dresses. These limitations restrict their applicability in real-world scenarios. In this paper, we first extend an existing dataset for human generation to include a wider range of apparel and more complex backgrounds. This extended dataset features people wearing diverse items such as tops, pants, dresses, skirts, headwear, scarves, shoes, socks, and bags. Additionally, we propose AnyDesign, a diffusion-based method that enables mask-free editing on versatile areas. Users can simply input a human image along with a corresponding prompt in either text or image format. Our approach incorporates Fashion DiT, equipped with a Fashion-Guidance Attention (FGA) module designed to fuse explicit apparel types and CLIP-encoded apparel features. Both Qualitative and quantitative experiments demonstrate that our method delivers high-quality fashion editing and outperforms contemporary text-guided fashion editing methods.

Title: GSTran: Joint Geometric and Semantic Coherence for Point Cloud Segmentation

• Authors: Abiao Li, Chenlei Lv, Guofeng Mei, Yifan Zuo, Jian Zhang, Yuming Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11558
• Pdf URL: https://arxiv.org/pdf/2408.11558
• Copy Paste: [[2408.11558]] GSTran: Joint Geometric and Semantic Coherence for Point Cloud Segmentation(https://arxiv.org/abs/2408.11558)
• Keywords: transformer, segmentation
• Abstract: Learning meaningful local and global information remains a challenge in point cloud segmentation tasks. When utilizing local information, prior studies indiscriminately aggregates neighbor information from different classes to update query points, potentially compromising the distinctive feature of query points. In parallel, inaccurate modeling of long-distance contextual dependencies when utilizing global information can also impact model performance. To address these issues, we propose GSTran, a novel transformer network tailored for the segmentation task. The proposed network mainly consists of two principal components: a local geometric transformer and a global semantic transformer. In the local geometric transformer module, we explicitly calculate the geometric disparity within the local region. This enables amplifying the affinity with geometrically similar neighbor points while suppressing the association with other neighbors. In the global semantic transformer module, we design a multi-head voting strategy. This strategy evaluates semantic similarity across the entire spatial range, facilitating the precise capture of contextual dependencies. Experiments on ShapeNetPart and S3DIS benchmarks demonstrate the effectiveness of the proposed method, showing its superiority over other algorithms. The code is available at this https URL.

Title: Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance

• Authors: Duc-Hai Pham, Duc Dung Nguyen, Hoang-Anh Pham, Ho Lai Tuan, Phong Ha Nguyen, Khoi Nguyen, Rang Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11559
• Pdf URL: https://arxiv.org/pdf/2408.11559
• Copy Paste: [[2408.11559]] Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance(https://arxiv.org/abs/2408.11559)
• Keywords: transformer
• Abstract: Accurate prediction of 3D semantic occupancy from 2D visual images is vital in enabling autonomous agents to comprehend their surroundings for planning and navigation. State-of-the-art methods typically employ fully supervised approaches, necessitating a huge labeled dataset acquired through expensive LiDAR sensors and meticulous voxel-wise labeling by human annotators. The resource-intensive nature of this annotating process significantly hampers the application and scalability of these methods. We introduce a novel semi-supervised framework to alleviate the dependency on densely annotated data. Our approach leverages 2D foundation models to generate essential 3D scene geometric and semantic cues, facilitating a more efficient training process. Our framework exhibits notable properties: (1) Generalizability, applicable to various 3D semantic scene completion approaches, including 2D-3D lifting and 3D-2D transformer methods. (2) Effectiveness, as demonstrated through experiments on SemanticKITTI and NYUv2, wherein our method achieves up to 85% of the fully-supervised performance using only 10% labeled data. This approach not only reduces the cost and labor associated with data annotation but also demonstrates the potential for broader adoption in camera-based systems for 3D semantic occupancy prediction.

Title: Self-Supervised Iterative Refinement for Anomaly Detection in Industrial Quality Control

• Authors: Muhammad Aqeel, Shakiba Sharifi, Marco Cristani, Francesco Setti
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11561
• Pdf URL: https://arxiv.org/pdf/2408.11561
• Copy Paste: [[2408.11561]] Self-Supervised Iterative Refinement for Anomaly Detection in Industrial Quality Control(https://arxiv.org/abs/2408.11561)
• Keywords: robust
• Abstract: This study introduces the Iterative Refinement Process (IRP), a robust anomaly detection methodology designed for high-stakes industrial quality control. The IRP enhances defect detection accuracy through a cyclic data refinement strategy, iteratively removing misleading data points to improve model performance and robustness. We validate the IRP's effectiveness using two benchmark datasets, Kolektor SDD2 (KSDD2) and MVTec AD, covering a wide range of industrial products and defect types. Our experimental results demonstrate that the IRP consistently outperforms traditional anomaly detection models, particularly in environments with high noise levels. This study highlights the IRP's potential to significantly enhance anomaly detection processes in industrial settings, effectively managing the challenges of sparse and noisy data.

Title: AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition

• Authors: Minheng Ni, Chenfei Wu, Huaying Yuan, Zhengyuan Yang, Ming Gong, Lijuan Wang, Zicheng Liu, Wangmeng Zuo, Nan Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11564
• Pdf URL: https://arxiv.org/pdf/2408.11564
• Copy Paste: [[2408.11564]] AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition(https://arxiv.org/abs/2408.11564)
• Keywords: generative
• Abstract: With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to generate multi-sensory outputs has not been fully explored, limiting the application on high-value scenarios such as of directing a film. Developing a movie director agent faces two major challenges: (1) Lack of parallelism and online scheduling with production steps: In the production of multi-sensory films, there are complex dependencies between different sensory elements, and the production time for each element varies. (2) Diverse needs and clear communication demands with users: Users often cannot clearly express their needs until they see a draft, which requires human-computer interaction and iteration to continually adjust and optimize the film content based on user feedback. To address these issues, we introduce AutoDirector, an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. This framework improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs. AutoDirector not only expands the application scope of human-machine collaboration but also demonstrates the potential of AI in collaborating with humans in the role of a film director to complete multi-sensory films.

Title: Positional Prompt Tuning for Efficient 3D Representation Learning

• Authors: Shaochen Zhang, Zekun Qi, Runpei Dong, Xiuxiu Bai, Xing Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11567
• Pdf URL: https://arxiv.org/pdf/2408.11567
• Copy Paste: [[2408.11567]] Positional Prompt Tuning for Efficient 3D Representation Learning(https://arxiv.org/abs/2408.11567)
• Keywords: transformer, segmentation
• Abstract: Point cloud analysis has achieved significant development and is well-performed in multiple downstream tasks like point cloud classification and segmentation, etc. Being conscious of the simplicity of the position encoding structure in Transformer-based architectures, we attach importance to the position encoding as a high-dimensional part and the patch encoder to offer multi-scale information. Together with the sequential Transformer, the whole module with position encoding comprehensively constructs a multi-scale feature abstraction module that considers both the local parts from the patch and the global parts from center points as position encoding. With only a few parameters, the position embedding module fits the setting of PEFT (Parameter-Efficient Fine-Tuning) tasks pretty well. Thus we unfreeze these parameters as a fine-tuning part. At the same time, we review the existing prompt and adapter tuning methods, proposing a fresh way of prompts and synthesizing them with adapters as dynamic adjustments. Our Proposed method of PEFT tasks, namely PPT, with only 1.05% of parameters for training, gets state-of-the-art results in several mainstream datasets, such as 95.01% accuracy in the ScanObjectNN OBJ_BG dataset. Codes will be released at this https URL.

Title: CHOTA: A Higher Order Accuracy Metric for Cell Tracking

• Authors: Timo Kaiser, Vladimir Ulman, Bodo Rosenhahn
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11571
• Pdf URL: https://arxiv.org/pdf/2408.11571
• Copy Paste: [[2408.11571]] CHOTA: A Higher Order Accuracy Metric for Cell Tracking(https://arxiv.org/abs/2408.11571)
• Keywords: robust
• Abstract: The evaluation of cell tracking results steers the development of tracking methods, significantly impacting biomedical research. This is quantitatively achieved by means of evaluation metrics. Unfortunately, current metrics favor local correctness and weakly reward global coherence, impeding high-level biological analysis. To also foster global coherence, we propose the CHOTA metric (Cell-specific Higher Order Tracking Accuracy) which unifies the evaluation of all relevant aspects of cell tracking: cell detections and local associations, global coherence, and lineage tracking. We achieve this by introducing a new definition of the term 'trajectory' that includes the entire cell lineage and by including this into the well-established HOTA metric from general multiple object tracking. Furthermore, we provide a detailed survey of contemporary cell tracking metrics to compare our novel CHOTA metric and to show its advantages. All metrics are extensively evaluated on state-of-the-art real-data cell tracking results and synthetic results that simulate specific tracking errors. We show that CHOTA is sensitive to all tracking errors and gives a good indication of the biologically relevant capability of a method to reconstruct the full lineage of cells. It introduces a robust and comprehensive alternative to the currently used metrics in cell tracking. Python code is available at this https URL .

Title: Constructions of Efficiently Implementable Boolean functions Possessing High Nonlinearity and Good Resistance to Algebraic Attacks

• Authors: Claude Carlet, Palash Sarkar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11583
• Pdf URL: https://arxiv.org/pdf/2408.11583
• Copy Paste: [[2408.11583]] Constructions of Efficiently Implementable Boolean functions Possessing High Nonlinearity and Good Resistance to Algebraic Attacks(https://arxiv.org/abs/2408.11583)
• Keywords: attack
• Abstract: We describe two new classes of functions which provide the presently best known trade-offs between low computational complexity, nonlinearity and (fast) algebraic immunity. The nonlinearity and (fast) algebraic immunity of the new functions substantially improve upon those properties of all previously known efficiently implementable functions. Appropriately chosen functions from the two new classes provide excellent solutions to the problem of designing filtering functions for use in the nonlinear filter model of stream ciphers, or in any other stream ciphers using Boolean functions for ensuring confusion. In particular, for $n\leq 20$, we show that there are functions in our first family whose implementation efficiences are significantly lower than all previously known functions achieving a comparable combination of nonlinearity and (fast) algebraic immunity. Given positive integers $\ell$ and $\delta$, it is possible to choose a function from our second family whose linear bias is provably at most $2^{-\ell}$, fast algebraic immunity is at least $\delta$ (based on conjecture which is well supported by experimental results), and which can be implemented in time and space which is linear in $\ell$ and $\delta$. Further, the functions in our second family are built using homomorphic friendly operations, making these functions well suited for the application of transciphering.

Title: Characterizing the Evolution of Psychological Factors Exploited by Malicious Emails

• Authors: Theodore Longtchi, Shouhuai Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11584
• Pdf URL: https://arxiv.org/pdf/2408.11584
• Copy Paste: [[2408.11584]] Characterizing the Evolution of Psychological Factors Exploited by Malicious Emails(https://arxiv.org/abs/2408.11584)
• Keywords: defense, attack, steal
• Abstract: Cyber attacks, including cyber social engineering attacks, such as malicious emails, are always evolving with time. Thus, it is important to understand their evolution. In this paper we characterize the evolution of malicious emails through the lens of Psychological Factors, PFs, which are humans psychological attributes that can be exploited by malicious emails. That is, attackers who send them. For this purpose, we propose a methodology and apply it to conduct a case study on 1,260 malicious emails over a span of 21 years, 2004 to 2024. Our findings include attackers have been constantly seeking to exploit many PFs, especially the ones that reflect human traits. Attackers have been increasingly exploiting 9 PFs and mostly in an implicit or stealthy fashion. Some PFs are often exploited together. These insights shed light on how to design future defenses against malicious emails.

Title: Characterizing the Evolution of Psychological Tactics and Techniques Exploited by Malicious Emails

• Authors: Theodore Longtchi, Shouhuai Xu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11586
• Pdf URL: https://arxiv.org/pdf/2408.11586
• Copy Paste: [[2408.11586]] Characterizing the Evolution of Psychological Tactics and Techniques Exploited by Malicious Emails(https://arxiv.org/abs/2408.11586)
• Keywords: defense, attack
• Abstract: The landscape of malicious emails and cyber social engineering attacks in general are constantly evolving. In order to design effective defenses against these attacks, we must deeply understand the Psychological Tactics, PTacs, and Psychological Techniques, PTechs, that are exploited by these attacks. In this paper we present a methodology for characterizing the evolution of PTacs and PTechs exploited by malicious emails. As a case study, we apply the methodology to a real-world dataset. This leads to a number insights, such as which PTacs or PTechs are more often exploited than others. These insights shed light on directions for future research towards designing psychologically-principled solutions to effectively counter malicious emails.

Title: Large Language Models are Good Attackers: Efficient and Stealthy Textual Backdoor Attacks

• Authors: Ziqiang Li, Yueqi Zeng, Pengfei Xia, Lei Liu, Zhangjie Fu, Bin Li
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11587
• Pdf URL: https://arxiv.org/pdf/2408.11587
• Copy Paste: [[2408.11587]] Large Language Models are Good Attackers: Efficient and Stealthy Textual Backdoor Attacks(https://arxiv.org/abs/2408.11587)
• Keywords: attack, steal, large language model
• Abstract: With the burgeoning advancements in the field of natural language processing (NLP), the demand for training data has increased significantly. To save costs, it has become common for users and businesses to outsource the labor-intensive task of data collection to third-party entities. Unfortunately, recent research has unveiled the inherent risk associated with this practice, particularly in exposing NLP systems to potential backdoor attacks. Specifically, these attacks enable malicious control over the behavior of a trained model by poisoning a small portion of the training data. Unlike backdoor attacks in computer vision, textual backdoor attacks impose stringent requirements for attack stealthiness. However, existing attack methods meet significant trade-off between effectiveness and stealthiness, largely due to the high information entropy inherent in textual data. In this paper, we introduce the Efficient and Stealthy Textual backdoor attack method, EST-Bad, leveraging Large Language Models (LLMs). Our EST-Bad encompasses three core strategies: optimizing the inherent flaw of models as the trigger, stealthily injecting triggers with LLMs, and meticulously selecting the most impactful samples for backdoor injection. Through the integration of these techniques, EST-Bad demonstrates an efficient achievement of competitive attack performance while maintaining superior stealthiness compared to prior methods across various text classifier datasets.

Title: Cause-Aware Empathetic Response Generation via Chain-of-Thought Fine-Tuning

• Authors: Xinhao Chen, Chong Yang, Man Lan, Li Cai, Yang Chen, Tu Hu, Xinlin Zhuang, Aimin Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11599
• Pdf URL: https://arxiv.org/pdf/2408.11599
• Copy Paste: [[2408.11599]] Cause-Aware Empathetic Response Generation via Chain-of-Thought Fine-Tuning(https://arxiv.org/abs/2408.11599)
• Keywords: large language model
• Abstract: Empathetic response generation endows agents with the capability to comprehend dialogue contexts and react to expressed emotions. Previous works predominantly focus on leveraging the speaker's emotional labels, but ignore the importance of emotion cause reasoning in empathetic response generation, which hinders the model's capacity for further affective understanding and cognitive inference. In this paper, we propose a cause-aware empathetic generation approach by integrating emotions and causes through a well-designed Chain-of-Thought (CoT) prompt on Large Language Models (LLMs). Our approach can greatly promote LLMs' performance of empathy by instruction tuning and enhancing the role awareness of an empathetic listener in the prompt. Additionally, we propose to incorporate cause-oriented external knowledge from COMET into the prompt, which improves the diversity of generation and alleviates conflicts between internal and external knowledge at the same time. Experimental results on the benchmark dataset demonstrate that our approach on LLaMA-7b achieves state-of-the-art performance in both automatic and human evaluations.

Title: Confidential Computing on Heterogeneous Systems: Survey and Implications

• Authors: Qifan Wang, David Oswald
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2408.11601
• Pdf URL: https://arxiv.org/pdf/2408.11601
• Copy Paste: [[2408.11601]] Confidential Computing on Heterogeneous Systems: Survey and Implications(https://arxiv.org/abs/2408.11601)
• Keywords: secure, security, privacy, protect, attack
• Abstract: In recent years, the widespread informatization and rapid data explosion have increased the demand for high-performance heterogeneous systems that integrate multiple computing cores such as CPUs, Graphics Processing Units (GPU s), Application Specific Integrated Circuits ( ASICs), Field Programmable Gate Arrays (FPGAs), and Neural Processing Units (NPU s). The combination of CPU and GPU is particularly popular due to its versatility. However, these heterogeneous systems face significant security and privacy risks. Advances in privacy-preserving techniques, especially hardware-based Trusted Execution Environments ( TEE s), offer effective protection for GPU applications. Nonetheless, the potential security risks involved in extending TEE s to GPUs in heterogeneous systems remain uncertain and need further investigation. To investigate these risks in depth, we study the existing popular GPU TEE designs and summarize and compare their key implications. Additionally, we review existing powerful attacks on GPUs and traditional TEE s deployed on CPUs, along with the efforts to mitigate these threats. We identify potential attack surfaces introduced by GPU TEE s and provide insights into key considerations for designing secure GPU TEEs. This survey is timely as new TEE s for heterogeneous systems, particularly GPUs, are being developed, highlighting the need to understand potential security threats and build both efficient and secure systems.

Title: Xinyu: An Efficient LLM-based System for Commentary Generation

• Authors: Yiquan Wu, Bo Tang, Chenyang Xi, Yu Yu, Pengyu Wang, Yifei Liu, Kun Kuang, Haiying Deng, Zhiyu Li, Feiyu Xiong, Jie Hu, Peng Cheng, Zhonghao Wang, Yi Wang, Yi Luo, Mingchuan Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11609
• Pdf URL: https://arxiv.org/pdf/2408.11609
• Copy Paste: [[2408.11609]] Xinyu: An Efficient LLM-based System for Commentary Generation(https://arxiv.org/abs/2408.11609)
• Keywords: fair, large language model
• Abstract: Commentary provides readers with a deep understanding of events by presenting diverse arguments and evidence. However, creating commentary is a time-consuming task, even for skilled commentators. Large language models (LLMs) have simplified the process of natural language generation, but their direct application in commentary creation still faces challenges due to unique task requirements. These requirements can be categorized into two levels: 1) fundamental requirements, which include creating well-structured and logically consistent narratives, and 2) advanced requirements, which involve generating quality arguments and providing convincing evidence. In this paper, we introduce Xinyu, an efficient LLM-based system designed to assist commentators in generating Chinese commentaries. To meet the fundamental requirements, we deconstruct the generation process into sequential steps, proposing targeted strategies and supervised fine-tuning (SFT) for each step. To address the advanced requirements, we present an argument ranking model for arguments and establish a comprehensive evidence database that includes up-to-date events and classic books, thereby strengthening the substantiation of the evidence with retrieval augmented generation (RAG) technology. To evaluate the generated commentaries more fairly, corresponding to the two-level requirements, we introduce a comprehensive evaluation metric that considers five distinct perspectives in commentary generation. Our experiments confirm the effectiveness of our proposed system. We also observe a significant increase in the efficiency of commentators in real-world scenarios, with the average time spent on creating a commentary dropping from 4 hours to 20 minutes. Importantly, such an increase in efficiency does not compromise the quality of the commentaries.

Title: Optimizing Interpretable Decision Tree Policies for Reinforcement Learning

• Authors: Daniël Vos, Sicco Verwer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11632
• Pdf URL: https://arxiv.org/pdf/2408.11632
• Copy Paste: [[2408.11632]] Optimizing Interpretable Decision Tree Policies for Reinforcement Learning(https://arxiv.org/abs/2408.11632)
• Keywords: interpretability
• Abstract: Reinforcement learning techniques leveraging deep learning have made tremendous progress in recent years. However, the complexity of neural networks prevents practitioners from understanding their behavior. Decision trees have gained increased attention in supervised learning for their inherent interpretability, enabling modelers to understand the exact prediction process after learning. This paper considers the problem of optimizing interpretable decision tree policies to replace neural networks in reinforcement learning settings. Previous works have relaxed the tree structure, restricted to optimizing only tree leaves, or applied imitation learning techniques to approximately copy the behavior of a neural network policy with a decision tree. We propose the Decision Tree Policy Optimization (DTPO) algorithm that directly optimizes the complete decision tree using policy gradients. Our technique uses established decision tree heuristics for regression to perform policy optimization. We empirically show that DTPO is a competitive algorithm compared to imitation learning algorithms for optimizing decision tree policies in reinforcement learning.

Title: Video-to-Text Pedestrian Monitoring (VTPM): Leveraging Computer Vision and Large Language Models for Privacy-Preserve Pedestrian Activity Monitoring at Intersections

• Authors: Ahmed S. Abdelrahman, Mohamed Abdel-Aty, Dongdong Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11649
• Pdf URL: https://arxiv.org/pdf/2408.11649
• Copy Paste: [[2408.11649]] Video-to-Text Pedestrian Monitoring (VTPM): Leveraging Computer Vision and Large Language Models for Privacy-Preserve Pedestrian Activity Monitoring at Intersections(https://arxiv.org/abs/2408.11649)
• Keywords: privacy, large language model
• Abstract: Computer vision has advanced research methodologies, enhancing system services across various fields. It is a core component in traffic monitoring systems for improving road safety; however, these monitoring systems don't preserve the privacy of pedestrians who appear in the videos, potentially revealing their identities. Addressing this issue, our paper introduces Video-to-Text Pedestrian Monitoring (VTPM), which monitors pedestrian movements at intersections and generates real-time textual reports, including traffic signal and weather information. VTPM uses computer vision models for pedestrian detection and tracking, achieving a latency of 0.05 seconds per video frame. Additionally, it detects crossing violations with 90.2% accuracy by incorporating traffic signal data. The proposed framework is equipped with Phi-3 mini-4k to generate real-time textual reports of pedestrian activity while stating safety concerns like crossing violations, conflicts, and the impact of weather on their behavior with latency of 0.33 seconds. To enhance comprehensive analysis of the generated textual reports, Phi-3 medium is fine-tuned for historical analysis of these generated textual reports. This fine-tuning enables more reliable analysis about the pedestrian safety at intersections, effectively detecting patterns and safety critical events. The proposed VTPM offers a more efficient alternative to video footage by using textual reports reducing memory usage, saving up to 253 million percent, eliminating privacy issues, and enabling comprehensive interactive historical analysis.

Title: CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researcher

• Authors: Derry Pratama, Naufal Suryanto, Andro Aprila Adiputra, Thi-Thu-Huong Le, Ahmada Yusril Kadiptya, Muhammad Iqbal, Howon Kim
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11650
• Pdf URL: https://arxiv.org/pdf/2408.11650
• Copy Paste: [[2408.11650]] CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researcher(https://arxiv.org/abs/2408.11650)
• Keywords: security, large language model
• Abstract: Penetration testing, a critical component of cybersecurity, typically requires extensive time and effort to find vulnerabilities. Beginners in this field often benefit from collaborative approaches with the community or experts. To address this, we develop CIPHER (Cybersecurity Intelligent Penetration-testing Helper for Ethical Researchers), a large language model specifically trained to assist in penetration testing tasks. We trained CIPHER using over 300 high-quality write-ups of vulnerable machines, hacking techniques, and documentation of open-source penetration testing tools. Additionally, we introduced the Findings, Action, Reasoning, and Results (FARR) Flow augmentation, a novel method to augment penetration testing write-ups to establish a fully automated pentesting simulation benchmark tailored for large language models. This approach fills a significant gap in traditional cybersecurity Q\&A benchmarks and provides a realistic and rigorous standard for evaluating AI's technical knowledge, reasoning capabilities, and practical utility in dynamic penetration testing scenarios. In our assessments, CIPHER achieved the best overall performance in providing accurate suggestion responses compared to other open-source penetration testing models of similar size and even larger state-of-the-art models like Llama 3 70B and Qwen1.5 72B Chat, particularly on insane difficulty machine setups. This demonstrates that the current capabilities of general LLMs are insufficient for effectively guiding users through the penetration testing process. We also discuss the potential for improvement through scaling and the development of better benchmarks using FARR Flow augmentation results. Our benchmark will be released publicly at this https URL.

Title: Macformer: Transformer with Random Maclaurin Feature Attention

• Authors: Yuhan Guo, Lizhong Ding, Ye Yuan, Guoren Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11656
• Pdf URL: https://arxiv.org/pdf/2408.11656
• Copy Paste: [[2408.11656]] Macformer: Transformer with Random Maclaurin Feature Attention(https://arxiv.org/abs/2408.11656)
• Keywords: transformer
• Abstract: Random feature attention (RFA) adopts random fourier feature (RFF) methods to approximate the softmax function, resulting in a linear time and space attention mechanism that enables the construction of an efficient Transformer. Inspired by RFA, we propose Macformer, a Transformer architecture that employs random Maclaurin features (RMF) to approximate various dot-product kernels, thereby accelerating attention computations for long sequence. Macformer consists of Random Maclaurin Feature Attention (RMFA) and pre-post Scaling Batch Normalization (ppSBN), the former is an unbiased approximation for dot-product kernelized attention and the later is a two-stage regularization mechanism guaranteeing the error of RMFA. We conducted toy experiments to demonstrate the efficiency of RMFA and ppSBN, and experiments on long range arena (LRA) benchmark to validate the acceleration and accuracy of Macformer with different dot-product kernels. Experiment results of Macformer are consistent with our theoretical analysis.

Title: Optimizing Federated Graph Learning with Inherent Structural Knowledge and Dual-Densely Connected GNNs

• Authors: Longwen Wang, Jianchun Liu, Zhi Liu, Jinyang Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11662
• Pdf URL: https://arxiv.org/pdf/2408.11662
• Copy Paste: [[2408.11662]] Optimizing Federated Graph Learning with Inherent Structural Knowledge and Dual-Densely Connected GNNs(https://arxiv.org/abs/2408.11662)
• Keywords: federate
• Abstract: Federated Graph Learning (FGL) is an emerging technology that enables clients to collaboratively train powerful Graph Neural Networks (GNNs) in a distributed manner without exposing their private data. Nevertheless, FGL still faces the challenge of the severe non-Independent and Identically Distributed (non-IID) nature of graphs, which possess diverse node and edge structures, especially across varied domains. Thus, exploring the knowledge inherent in these structures becomes significantly crucial. Existing methods, however, either overlook the inherent structural knowledge in graph data or capture it at the cost of significantly increased resource demands (e.g., FLOPs and communication bandwidth), which can be detrimental to distributed paradigms. Inspired by this, we propose FedDense, a novel FGL framework that optimizes the utilization efficiency of inherent structural knowledge. To better acquire knowledge of diverse and underexploited structures, FedDense first explicitly encodes the structural knowledge inherent within graph data itself alongside node features. Besides, FedDense introduces a Dual-Densely Connected (DDC) GNN architecture that exploits the multi-scale (i.e., one-hop to multi-hop) feature and structure insights embedded in the aggregated feature maps at each layer. In addition to the exploitation of inherent structures, we consider resource limitations in FGL, devising exceedingly narrow layers atop the DDC architecture and adopting a selective parameter sharing strategy to reduce resource costs substantially. We conduct extensive experiments using 15 datasets across 4 different domains, demonstrating that FedDense consistently surpasses baselines by a large margin in training performance, while demanding minimal resources.

Title: Exploring Robustness of Visual State Space model against Backdoor Attacks

• Authors: Cheng-Yi Lee, Cheng-Chang Tsai, Chia-Mu Yu, Chun-Shien Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11679
• Pdf URL: https://arxiv.org/pdf/2408.11679
• Copy Paste: [[2408.11679]] Exploring Robustness of Visual State Space model against Backdoor Attacks(https://arxiv.org/abs/2408.11679)
• Keywords: security, attack, robust, transformer
• Abstract: Visual State Space Model (VSS) has demonstrated remarkable performance in various computer vision tasks. However, in the process of development, backdoor attacks have brought severe challenges to security. Such attacks cause an infected model to predict target labels when a specific trigger is activated, while the model behaves normally on benign samples. In this paper, we conduct systematic experiments to comprehend on robustness of VSS through the lens of backdoor attacks, specifically how the state space model (SSM) mechanism affects robustness. We first investigate the vulnerability of VSS to different backdoor triggers and reveal that the SSM mechanism, which captures contextual information within patches, makes the VSS model more susceptible to backdoor triggers compared to models without SSM. Furthermore, we analyze the sensitivity of the VSS model to patch processing techniques and discover that these triggers are effectively disrupted. Based on these observations, we consider an effective backdoor for the VSS model that recurs in each patch to resist patch perturbations. Extensive experiments across three datasets and various backdoor attacks reveal that the VSS model performs comparably to Transformers (ViTs) but is less robust than the Gated CNNs, which comprise only stacked Gated CNN blocks without SSM.

Title: First line of defense: A robust first layer mitigates adversarial attacks

• Authors: Janani Suresh, Nancy Nayak, Sheetal Kalyani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11680
• Pdf URL: https://arxiv.org/pdf/2408.11680
• Copy Paste: [[2408.11680]] First line of defense: A robust first layer mitigates adversarial attacks(https://arxiv.org/abs/2408.11680)
• Keywords: defense, attack, robust
• Abstract: Adversarial training (AT) incurs significant computational overhead, leading to growing interest in designing inherently robust architectures. We demonstrate that a carefully designed first layer of the neural network can serve as an implicit adversarial noise filter (ANF). This filter is created using a combination of large kernel size, increased convolution filters, and a maxpool operation. We show that integrating this filter as the first layer in architectures such as ResNet, VGG, and EfficientNet results in adversarially robust networks. Our approach achieves higher adversarial accuracies than existing natively robust architectures without AT and is competitive with adversarial-trained architectures across a wide range of datasets. Supporting our findings, we show that (a) the decision regions for our method have better margins, (b) the visualized loss surfaces are smoother, (c) the modified peak signal-to-noise ratio (mPSNR) values at the output of the ANF are higher, (d) high-frequency components are more attenuated, and (e) architectures incorporating ANF exhibit better denoising in Gaussian noise compared to baseline architectures. Code for all our experiments are available at \url{this https URL}.

Title: Interpretable Long-term Action Quality Assessment

• Authors: Xu Dong, Xinran Liu, Wanqing Li, Anthony Adeyemi-Ejeye, Andrew Gilbert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11687
• Pdf URL: https://arxiv.org/pdf/2408.11687
• Copy Paste: [[2408.11687]] Interpretable Long-term Action Quality Assessment(https://arxiv.org/abs/2408.11687)
• Keywords: interpretability, transformer
• Abstract: Long-term Action Quality Assessment (AQA) evaluates the execution of activities in videos. However, the length presents challenges in fine-grained interpretability, with current AQA methods typically producing a single score by averaging clip features, lacking detailed semantic meanings of individual clips. Long-term videos pose additional difficulty due to the complexity and diversity of actions, exacerbating interpretability challenges. While query-based transformer networks offer promising long-term modeling capabilities, their interpretability in AQA remains unsatisfactory due to a phenomenon we term Temporal Skipping, where the model skips self-attention layers to prevent output degradation. To address this, we propose an attention loss function and a query initialization method to enhance performance and interpretability. Additionally, we introduce a weight-score regression module designed to approximate the scoring patterns observed in human judgments and replace conventional single-score regression, improving the rationality of interpretability. Our approach achieves state-of-the-art results on three real-world, long-term AQA benchmarks. Our code is available at: this https URL

Title: Robust 3D Gaussian Splatting for Novel View Synthesis in Presence of Distractors

• Authors: Paul Ungermann, Armin Ettenhofer, Matthias Nießner, Barbara Roessle
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11697
• Pdf URL: https://arxiv.org/pdf/2408.11697
• Copy Paste: [[2408.11697]] Robust 3D Gaussian Splatting for Novel View Synthesis in Presence of Distractors(https://arxiv.org/abs/2408.11697)
• Keywords: robust, segmentation
• Abstract: 3D Gaussian Splatting has shown impressive novel view synthesis results; nonetheless, it is vulnerable to dynamic objects polluting the input data of an otherwise static scene, so called distractors. Distractors have severe impact on the rendering quality as they get represented as view-dependent effects or result in floating artifacts. Our goal is to identify and ignore such distractors during the 3D Gaussian optimization to obtain a clean reconstruction. To this end, we take a self-supervised approach that looks at the image residuals during the optimization to determine areas that have likely been falsified by a distractor. In addition, we leverage a pretrained segmentation network to provide object awareness, enabling more accurate exclusion of distractors. This way, we obtain segmentation masks of distractors to effectively ignore them in the loss formulation. We demonstrate that our approach is robust to various distractors and strongly improves rendering quality on distractor-polluted scenes, improving PSNR by 1.86dB compared to 3D Gaussian Splatting.

Title: Supervised Representation Learning towards Generalizable Assembly State Recognition

• Authors: Tim J. Schoonbeek, Goutham Balachandran, Hans Onvlee, Tim Houben, Shao-Hsuan Hung, Jacek Kustra, Peter H.N. de With, Fons van der Sommen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11700
• Pdf URL: https://arxiv.org/pdf/2408.11700
• Copy Paste: [[2408.11700]] Supervised Representation Learning towards Generalizable Assembly State Recognition(https://arxiv.org/abs/2408.11700)
• Keywords: robust
• Abstract: Assembly state recognition facilitates the execution of assembly procedures, offering feedback to enhance efficiency and minimize errors. However, recognizing assembly states poses challenges in scalability, since parts are frequently updated, and the robustness to execution errors remains underexplored. To address these challenges, this paper proposes an approach based on representation learning and the novel intermediate-state informed loss function modification (ISIL). ISIL leverages unlabeled transitions between states and demonstrates significant improvements in clustering and classification performance for all tested architectures and losses. Despite being trained exclusively on images without execution errors, thorough analysis on error states demonstrates that our approach accurately distinguishes between correct states and states with various types of execution errors. The integration of the proposed algorithm can offer meaningful assistance to workers and mitigate unexpected losses due to procedural mishaps in industrial settings. The code is available at: this https URL

Title: FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting

• Authors: Liyao Jiang, Negar Hassanpour, Mohammad Salameh, Mohan Sai Singamsetti, Fengyu Sun, Wei Lu, Di Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11706
• Pdf URL: https://arxiv.org/pdf/2408.11706
• Copy Paste: [[2408.11706]] FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting(https://arxiv.org/abs/2408.11706)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models have demonstrated impressive capabilities in generating high-quality images given a text prompt. However, ensuring the prompt-image alignment remains a considerable challenge, i.e., generating images that faithfully align with the prompt's semantics. Recent works attempt to improve the faithfulness by optimizing the latent code, which potentially could cause the latent code to go out-of-distribution and thus produce unrealistic images. In this paper, we propose FRAP, a simple, yet effective approach based on adaptively adjusting the per-token prompt weights to improve prompt-image alignment and authenticity of the generated images. We design an online algorithm to adaptively update each token's weight coefficient, which is achieved by minimizing a unified objective function that encourages object presence and the binding of object-modifier pairs. Through extensive evaluations, we show FRAP generates images with significantly higher prompt-image alignment to prompts from complex datasets, while having a lower average latency compared to recent latent code optimization methods, e.g., 4 seconds faster than D&B on the COCO-Subject dataset. Furthermore, through visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that FRAP not only improves prompt-image alignment but also generates more authentic images with realistic appearances. We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image alignment, where we observe improvements in both prompt-image alignment and image quality.

Title: On Learnable Parameters of Optimal and Suboptimal Deep Learning Models

• Authors: Ziwei Zheng, Huizhi Liang, Vaclav Snasel, Vito Latora, Panos Pardalos, Giuseppe Nicosia, Varun Ojha
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11720
• Pdf URL: https://arxiv.org/pdf/2408.11720
• Copy Paste: [[2408.11720]] On Learnable Parameters of Optimal and Suboptimal Deep Learning Models(https://arxiv.org/abs/2408.11720)
• Keywords: transformer
• Abstract: We scrutinize the structural and operational aspects of deep learning models, particularly focusing on the nuances of learnable parameters (weight) statistics, distribution, node interaction, and visualization. By establishing correlations between variance in weight patterns and overall network performance, we investigate the varying (optimal and suboptimal) performances of various deep-learning models. Our empirical analysis extends across widely recognized datasets such as MNIST, Fashion-MNIST, and CIFAR-10, and various deep learning models such as deep neural networks (DNNs), convolutional neural networks (CNNs), and vision transformer (ViT), enabling us to pinpoint characteristics of learnable parameters that correlate with successful networks. Through extensive experiments on the diverse architectures of deep learning models, we shed light on the critical factors that influence the functionality and efficiency of DNNs. Our findings reveal that successful networks, irrespective of datasets or models, are invariably similar to other successful networks in their converged weights statistics and distribution, while poor-performing networks vary in their weights. In addition, our research shows that the learnable parameters of widely varied deep learning models such as DNN, CNN, and ViT exhibit similar learning characteristics.

Title: Iterative Object Count Optimization for Text-to-image Diffusion Models

• Authors: Oz Zafar, Lior Wolf, Idan Schwartz
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11721
• Pdf URL: https://arxiv.org/pdf/2408.11721
• Copy Paste: [[2408.11721]] Iterative Object Count Optimization for Text-to-image Diffusion Models(https://arxiv.org/abs/2408.11721)
• Keywords: diffusion
• Abstract: We address a persistent challenge in text-to-image models: accurately generating a specified number of objects. Current models, which learn from image-text pairs, inherently struggle with counting, as training data cannot depict every possible number of objects for any given object. To solve this, we propose optimizing the generated image based on a counting loss derived from a counting model that aggregates an objectś potential. Employing an out-of-the-box counting model is challenging for two reasons: first, the model requires a scaling hyperparameter for the potential aggregation that varies depending on the viewpoint of the objects, and second, classifier guidance techniques require modified models that operate on noisy intermediate diffusion steps. To address these challenges, we propose an iterated online training mode that improves the accuracy of inferred images while altering the text conditioning embedding and dynamically adjusting hyperparameters. Our method offers three key advantages: (i) it can consider non-derivable counting techniques based on detection models, (ii) it is a zero-shot plug-and-play solution facilitating rapid changes to the counting techniques and image generation methods, and (iii) the optimized counting token can be reused to generate accurate images without additional optimization. We evaluate the generation of various objects and show significant improvements in accuracy. The project page is available at this https URL.

Title: Efficient Detection of Toxic Prompts in Large Language Models

• Authors: Yi Liu, Junzhe Yu, Huijia Sun, Ling Shi, Gelei Deng, Yuqi Chen, Yang Liu
• Subjects: cs.CR, cs.AI, cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.11727
• Pdf URL: https://arxiv.org/pdf/2408.11727
• Copy Paste: [[2408.11727]] Efficient Detection of Toxic Prompts in Large Language Models(https://arxiv.org/abs/2408.11727)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) like ChatGPT and Gemini have significantly advanced natural language processing, enabling various applications such as chatbots and automated content generation. However, these models can be exploited by malicious individuals who craft toxic prompts to elicit harmful or unethical responses. These individuals often employ jailbreaking techniques to bypass safety mechanisms, highlighting the need for robust toxic prompt detection methods. Existing detection techniques, both blackbox and whitebox, face challenges related to the diversity of toxic prompts, scalability, and computational efficiency. In response, we propose ToxicDetector, a lightweight greybox method designed to efficiently detect toxic prompts in LLMs. ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP) classifier for prompt classification. Our evaluation on various versions of the LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%, outperforming state-of-the-art methods. Additionally, ToxicDetector's processing time of 0.0780 seconds per prompt makes it highly suitable for real-time applications. ToxicDetector achieves high accuracy, efficiency, and scalability, making it a practical method for toxic prompt detection in LLMs.

Title: Enhancing Cross-Modal Medical Image Segmentation through Compositionality

• Authors: Aniek Eijpe, Valentina Corbetta, Kalina Chupetlovska, Regina Beets-Tan, Wilson Silva
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11733
• Pdf URL: https://arxiv.org/pdf/2408.11733
• Copy Paste: [[2408.11733]] Enhancing Cross-Modal Medical Image Segmentation through Compositionality(https://arxiv.org/abs/2408.11733)
• Keywords: interpretability, segmentation
• Abstract: Cross-modal medical image segmentation presents a significant challenge, as different imaging modalities produce images with varying resolutions, contrasts, and appearances of anatomical structures. We introduce compositionality as an inductive bias in a cross-modal segmentation network to improve segmentation performance and interpretability while reducing complexity. The proposed network is an end-to-end cross-modal segmentation framework that enforces compositionality on the learned representations using learnable von Mises-Fisher kernels. These kernels facilitate content-style disentanglement in the learned representations, resulting in compositional content representations that are inherently interpretable and effectively disentangle different anatomical structures. The experimental results demonstrate enhanced segmentation performance and reduced computational costs on multiple medical datasets. Additionally, we demonstrate the interpretability of the learned compositional features. Code and checkpoints will be publicly available at: this https URL.

Title: MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

• Authors: Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler, Dan Alistarh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11743
• Pdf URL: https://arxiv.org/pdf/2408.11743
• Copy Paste: [[2408.11743]] MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models(https://arxiv.org/abs/2408.11743)
• Keywords: large language model
• Abstract: As inference on Large Language Models (LLMs) emerges as an important workload in machine learning applications, weight quantization has become a standard technique for efficient GPU deployment. Quantization not only reduces model size, but has also been shown to yield substantial speedups for single-user inference, due to reduced memory movement, with low accuracy impact. Yet, it remains open whether speedups are achievable also in \emph{batched} settings with multiple parallel clients, which are highly relevant for practical serving. It is unclear whether GPU kernels can be designed to remain practically memory-bound, while supporting the substantially increased compute requirements of batched workloads. This paper resolves this question positively by describing the design of Mixed-precision Auto-Regressive LINear kernels, called MARLIN. Concretely, given a model whose weights are compressed via quantization to, e.g., 4 bits per element, MARLIN shows that batchsizes up to 16-32 can be supported with close to maximum ($4\times$) quantization speedup, and larger batchsizes up to 64-128 with gradually decreasing, but still significant, acceleration. MARLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining, and bespoke quantization support. Our experiments show that MARLIN's near-optimal performance on individual LLM layers across different scenarios can also lead to end-to-end LLM inference speedups (of up to $2.8\times$) when integrated with the popular vLLM serving engine. Finally, MARLIN is extensible to further compression techniques, like NVIDIA 2:4 sparsity, leading to additional speedups.

Title: FocusLLM: Scaling LLM's Context by Parallel Decoding

• Authors: Zhenyu Li, Yike Zhang, Tengyu Pan, Yutao Sun, Zhichao Duan, Junjie Fang, Rong Han, Zixuan Wang, Jianyong Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11745
• Pdf URL: https://arxiv.org/pdf/2408.11745
• Copy Paste: [[2408.11745]] FocusLLM: Scaling LLM's Context by Parallel Decoding(https://arxiv.org/abs/2408.11745)
• Keywords: transformer
• Abstract: Empowering LLMs with the ability to utilize useful information from a long context is crucial for many downstream applications. However, achieving long context lengths with the conventional transformer architecture requires substantial training and inference resources. In this paper, we present FocusLLM, a framework designed to extend the context length of any decoder-only LLM, enabling the model to focus on relevant information from very long sequences. FocusLLM processes long text inputs by dividing them into chunks based on the model's original context length to alleviate the issue of attention distraction. Then, it appends the local context to each chunk as a prompt to extract essential information from each chunk based on a novel parallel decoding mechanism, and ultimately integrates the extracted information into the local context. FocusLLM stands out for great training efficiency and versatility: trained with an 8K input length with much less training cost than previous methods, FocusLLM exhibits superior performance across downstream long-context tasks and maintains strong language modeling ability when handling extensive long texts, even up to 400K tokens. Our code is available at this https URL.

Title: Mixed Sparsity Training: Achieving 4$\times$ FLOP Reduction for Transformer Pretraining

• Authors: Pihe Hu, Shaolong Li, Longbo Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11746
• Pdf URL: https://arxiv.org/pdf/2408.11746
• Copy Paste: [[2408.11746]] Mixed Sparsity Training: Achieving 4$\times$ FLOP Reduction for Transformer Pretraining(https://arxiv.org/abs/2408.11746)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have made significant strides in complex tasks, yet their widespread adoption is impeded by substantial computational demands. With hundreds of billion parameters, transformer-based LLMs necessitate months of pretraining across a high-end GPU cluster. However, this paper reveals a compelling finding: transformers exhibit considerable redundancy in pretraining computations, which motivates our proposed solution, Mixed Sparsity Training (MST), an efficient pretraining method that can reduce about $75\%$ of Floating Point Operations (FLOPs) while maintaining performance. MST integrates dynamic sparse training (DST) with Sparsity Variation (SV) and Hybrid Sparse Attention (HSA) during pretraining, involving three distinct phases: warm-up, ultra-sparsification, and restoration. The warm-up phase transforms the dense model into a sparse one, and the restoration phase reinstates connections. Throughout these phases, the model is trained with a dynamically evolving sparse topology and an HSA mechanism to maintain performance and minimize training FLOPs concurrently. Our experiment on GPT-2 showcases a FLOP reduction of $4\times$ without compromising performance.

Title: Open-Ended 3D Point Cloud Instance Segmentation

• Authors: Phuc D.A. Nguyen, Minh Luu, Anh Tran, Cuong Pham, Khoi Nguyen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11747
• Pdf URL: https://arxiv.org/pdf/2408.11747
• Copy Paste: [[2408.11747]] Open-Ended 3D Point Cloud Instance Segmentation(https://arxiv.org/abs/2408.11747)
• Keywords: large language model, segmentation
• Abstract: Open-Vocab 3D Instance Segmentation methods (OV-3DIS) have recently demonstrated their ability to generalize to unseen objects. However, these methods still depend on predefined class names during testing, restricting the autonomy of agents. To mitigate this constraint, we propose a novel problem termed Open-Ended 3D Instance Segmentation (OE-3DIS), which eliminates the necessity for predefined class names during testing. Moreover, we contribute a comprehensive set of strong baselines, derived from OV-3DIS approaches and leveraging 2D Multimodal Large Language Models. To assess the performance of our OE-3DIS system, we introduce a novel Open-Ended score, evaluating both the semantic and geometric quality of predicted masks and their associated class names, alongside the standard AP score. Our approach demonstrates significant performance improvements over the baselines on the ScanNet200 and ScanNet++ datasets. Remarkably, our method surpasses the performance of Open3DIS, the current state-of-the-art method in OV-3DIS, even in the absence of ground-truth object class names.

Title: Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks

• Authors: Yiyi Chen, Russa Biswas, Heather Lent, Johannes Bjerva
• Subjects: cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.11749
• Pdf URL: https://arxiv.org/pdf/2408.11749
• Copy Paste: [[2408.11749]] Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks(https://arxiv.org/abs/2408.11749)
• Keywords: security, attack, large language model
• Abstract: Large Language Models (LLMs) are susceptible to malicious influence by cyber attackers through intrusions such as adversarial, backdoor, and embedding inversion attacks. In response, the burgeoning field of LLM Security aims to study and defend against such threats. Thus far, the majority of works in this area have focused on monolingual English models, however, emerging research suggests that multilingual LLMs may be more vulnerable to various attacks than their monolingual counterparts. While previous work has investigated embedding inversion over a small subset of European languages, it is challenging to extrapolate these findings to languages from different linguistic families and with differing scripts. To this end, we explore the security of multilingual LLMs in the context of embedding inversion attacks and investigate cross-lingual and cross-script inversion across 20 languages, spanning over 8 language families and 12 scripts. Our findings indicate that languages written in Arabic script and Cyrillic script are particularly vulnerable to embedding inversion, as are languages within the Indo-Aryan language family. We further observe that inversion models tend to suffer from language confusion, sometimes greatly reducing the efficacy of an attack. Accordingly, we systematically explore this bottleneck for inversion models, uncovering predictable patterns which could be leveraged by attackers. Ultimately, this study aims to further the field's understanding of the outstanding security vulnerabilities facing multilingual LLMs and raise awareness for the languages most at risk of negative impact from these attacks.

Title: SBDet: A Symmetry-Breaking Object Detector via Relaxed Rotation-Equivariance

• Authors: Zhiqiang Wu, Yingjie Liu, Hanlin Dong, Xuan Tang, Jian Yang, Bo Jin, Mingsong Chen, Xian Wei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11760
• Pdf URL: https://arxiv.org/pdf/2408.11760
• Copy Paste: [[2408.11760]] SBDet: A Symmetry-Breaking Object Detector via Relaxed Rotation-Equivariance(https://arxiv.org/abs/2408.11760)
• Keywords: robust
• Abstract: Introducing Group Equivariant Convolution (GConv) empowers models to explore symmetries hidden in visual data, improving their performance. However, in real-world scenarios, objects or scenes often exhibit perturbations of a symmetric system, specifically a deviation from a symmetric architecture, which can be characterized by a non-trivial action of a symmetry group, known as Symmetry-Breaking. Traditional GConv methods are limited by the strict operation rules in the group space, only ensuring features remain strictly equivariant under limited group transformations, making it difficult to adapt to Symmetry-Breaking or non-rigid transformations. Motivated by this, we introduce a novel Relaxed Rotation GConv (R2GConv) with our defined Relaxed Rotation-Equivariant group $\mathbf{R}_4$. Furthermore, we propose a Relaxed Rotation-Equivariant Network (R2Net) as the backbone and further develop the Symmetry-Breaking Object Detector (SBDet) for 2D object detection built upon it. Experiments demonstrate the effectiveness of our proposed R2GConv in natural image classification tasks, and SBDet achieves excellent performance in object detection tasks with improved generalization capabilities and robustness.

Title: Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards

• Authors: Omar Erak, Nouf Alabbasi, Omar Alhussein, Ismail Lotfi, Amr Hussein, Sami Muhaidat, Merouane Debbah
• Subjects: cs.CL, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.11775
• Pdf URL: https://arxiv.org/pdf/2408.11775
• Copy Paste: [[2408.11775]] Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards(https://arxiv.org/abs/2408.11775)
• Keywords: large language model
• Abstract: Recent studies show that large language models (LLMs) struggle with technical standards in telecommunications. We propose a fine-tuned retrieval-augmented generation (RAG) system based on the Phi-2 small language model (SLM) to serve as an oracle for communication networks. Our developed system leverages forward-looking semantic chunking to adaptively determine parsing breakpoints based on embedding similarity, enabling effective processing of diverse document formats. To handle the challenge of multiple similar contexts in technical standards, we employ a re-ranking algorithm to prioritize the most relevant retrieved chunks. Recognizing the limitations of Phi-2's small context window, we implement a recent technique, namely SelfExtend, to expand the context window during inference, which not only boosts the performance but also can accommodate a wider range of user queries and design requirements from customers to specialized technicians. For fine-tuning, we utilize the low-rank adaptation (LoRA) technique to enhance computational efficiency during training and enable effective fine-tuning on small datasets. Our comprehensive experiments demonstrate substantial improvements over existing question-answering approaches in the telecom domain, achieving performance that exceeds larger language models such as GPT-4 (which is about 880 times larger in size). This work presents a novel approach to leveraging SLMs for communication networks, offering a balance of efficiency and performance. This work can serve as a foundation towards agentic language models for networks.

Title: Sum of Squares Circuits

• Authors: Lorenzo Loconte, Stefan Mengel, Antonio Vergari
• Subjects: cs.LG, cs.AI, cs.CC, math.AG
• Abstract URL: https://arxiv.org/abs/2408.11778
• Pdf URL: https://arxiv.org/pdf/2408.11778
• Copy Paste: [[2408.11778]] Sum of Squares Circuits(https://arxiv.org/abs/2408.11778)
• Keywords: generative
• Abstract: Designing expressive generative models that support exact and efficient inference is a core question in probabilistic ML. Probabilistic circuits (PCs) offer a framework where this tractability-vs-expressiveness trade-off can be analyzed theoretically. Recently, squared PCs encoding subtractive mixtures via negative parameters have emerged as tractable models that can be exponentially more expressive than monotonic PCs, i.e., PCs with positive parameters only. In this paper, we provide a more precise theoretical characterization of the expressiveness relationships among these models. First, we prove that squared PCs can be less expressive than monotonic ones. Second, we formalize a novel class of PCs -- sum of squares PCs -- that can be exponentially more expressive than both squared and monotonic PCs. Around sum of squares PCs, we build an expressiveness hierarchy that allows us to precisely unify and separate different tractable model classes such as Born Machines and PSD models, and other recently introduced tractable probabilistic models by using complex parameters. Finally, we empirically show the effectiveness of sum of squares circuits in performing distribution estimation.

Title: Personality Alignment of Large Language Models

• Authors: Minjun Zhu, Linyi Yang, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11779
• Pdf URL: https://arxiv.org/pdf/2408.11779
• Copy Paste: [[2408.11779]] Personality Alignment of Large Language Models(https://arxiv.org/abs/2408.11779)
• Keywords: large language model
• Abstract: Current methods for aligning large language models (LLMs) typically aim to reflect general human values and behaviors, but they often fail to capture the unique characteristics and preferences of individual users. To address this gap, we introduce the concept of Personality Alignment. This approach tailors LLMs' responses and decisions to match the specific preferences of individual users or closely related groups. Inspired by psychometrics, we created the Personality Alignment with Personality Inventories (PAPI) dataset, which includes data from 300,000 real subjects, each providing behavioral preferences based on the Big Five Personality Factors. This dataset allows us to quantitatively evaluate the extent to which LLMs can align with each subject's behavioral patterns. Recognizing the challenges of personality alignments: such as limited personal data, diverse preferences, and scalability requirements: we developed an activation intervention optimization method. This method enhances LLMs' ability to efficiently align with individual behavioral preferences using minimal data and computational resources. Remarkably, our method, PAS, achieves superior performance while requiring only 1/5 of the optimization time compared to DPO, offering practical value for personality alignment. Our work paves the way for future AI systems to make decisions and reason in truly personality ways, enhancing the relevance and meaning of AI interactions for each user and advancing human-centered artificial intelligence.The code has released in \url{this https URL}.

Title: RFID based Health Adherence Medicine Case Using Fair Federated Learning

• Authors: Ali Kamrani khodaei, Sina Hajer Ahmadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11782
• Pdf URL: https://arxiv.org/pdf/2408.11782
• Copy Paste: [[2408.11782]] RFID based Health Adherence Medicine Case Using Fair Federated Learning(https://arxiv.org/abs/2408.11782)
• Keywords: secure, privacy, extraction, federate, fair
• Abstract: Medication nonadherence significantly reduces the effectiveness of therapies, yet it remains prevalent among patients. Nonadherence has been linked to adverse outcomes, including increased risks of mortality and hospitalization. Although various methods exist to help patients track medication schedules, such as the Intelligent Drug Administration System (IDAS) and Smart Blister, these tools often face challenges that hinder their commercial viability. Building on the principles of dosage measurement and information communication in IoT, we introduce the Smart Pill Case a smart health adherence tool that leverages RFID-based data recording and NFC-based data extraction. This system incorporates a load cell for precise dosage measurement and features an Android app to monitor medication intake, offer suggestions, and issue warnings. To enhance the effectiveness and personalization of the Smart Pill Case, we propose integrating federated learning into the system. Federated learning allows the Smart Pill Case to learn from medication adherence patterns across multiple users without compromising individual privacy. By training machine learning models on decentralized data collected from various Smart Pill Cases, the system can continuously improve its recommendations and warnings, adapting to the diverse needs and behaviors of users. This approach not only enhances the tools ability to support medication adherence but also ensures that sensitive user data remains secure and private.

Title: Timeline and Boundary Guided Diffusion Network for Video Shadow Detection

• Authors: Haipeng Zhou, Honqiu Wang, Tian Ye, Zhaohu Xing, Jun Ma, Ping Li, Qiong Wang, Lei Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11785
• Pdf URL: https://arxiv.org/pdf/2408.11785
• Copy Paste: [[2408.11785]] Timeline and Boundary Guided Diffusion Network for Video Shadow Detection(https://arxiv.org/abs/2408.11785)
• Keywords: diffusion
• Abstract: Video Shadow Detection (VSD) aims to detect the shadow masks with frame sequence. Existing works suffer from inefficient temporal learning. Moreover, few works address the VSD problem by considering the characteristic (i.e., boundary) of shadow. Motivated by this, we propose a Timeline and Boundary Guided Diffusion (TBGDiff) network for VSD where we take account of the past-future temporal guidance and boundary information jointly. In detail, we design a Dual Scale Aggregation (DSA) module for better temporal understanding by rethinking the affinity of the long-term and short-term frames for the clipped video. Next, we introduce Shadow Boundary Aware Attention (SBAA) to utilize the edge contexts for capturing the characteristics of shadows. Moreover, we are the first to introduce the Diffusion model for VSD in which we explore a Space-Time Encoded Embedding (STEE) to inject the temporal guidance for Diffusion to conduct shadow detection. Benefiting from these designs, our model can not only capture the temporal information but also the shadow property. Extensive experiments show that the performance of our approach overtakes the state-of-the-art methods, verifying the effectiveness of our components. We release the codes, weights, and results at \url{this https URL}.

Title: Critique-out-Loud Reward Models

• Authors: Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11791
• Pdf URL: https://arxiv.org/pdf/2408.11791
• Copy Paste: [[2408.11791]] Critique-out-Loud Reward Models(https://arxiv.org/abs/2408.11791)
• Keywords: large language model
• Abstract: Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabilities of the underlying large language model (LLM). This limits the capabilities of reward models as they must reason implicitly about the quality of a response, i.e., preference modeling must be performed in a single forward pass through the model. To enable reward models to reason explicitly about the quality of a response, we introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response that is then used to predict a scalar reward for the quality of the response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models: compared to classic reward models CLoud reward models improve pairwise preference classification accuracy on RewardBench by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively. Furthermore, CLoud reward models lead to a Pareto improvement for win rate on ArenaHard when used as the scoring model for Best-of-N. Finally, we explore how to exploit the dynamic inference compute capabilities of CLoud reward models by performing self-consistency decoding for reward prediction.

Title: EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

• Authors: Feipeng Ma, Yizhou Zhou, Hebei Li, Zilong He, Siying Wu, Fengyun Rao, Yueyi Zhang, Xiaoyan Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11795
• Pdf URL: https://arxiv.org/pdf/2408.11795
• Copy Paste: [[2408.11795]] EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model(https://arxiv.org/abs/2408.11795)
• Keywords: large language model
• Abstract: In the realm of multimodal research, numerous studies leverage substantial image-text pairs to conduct modal alignment learning, transforming Large Language Models (LLMs) into Multimodal LLMs and excelling in a variety of visual-language tasks. The prevailing methodologies primarily fall into two categories: self-attention-based and cross-attention-based methods. While self-attention-based methods offer superior data efficiency due to their simple MLP architecture, they often suffer from lower computational efficiency due to concatenating visual and textual tokens as input for LLM. Conversely, cross-attention-based methods, although less data-efficient due to additional learnable parameters, exhibit higher computational efficiency by avoiding long sequence input for LLM. To address these trade-offs, we introduce the Data-Efficient and Compute-Efficient Multimodal Large Language Model (EE-MLLM). Without introducing additional modules or learnable parameters, EE-MLLM achieves both data and compute efficiency. Specifically, we modify the original self-attention mechanism in MLLM to a composite attention mechanism. This mechanism has two key characteristics: 1) Eliminating the computational overhead of self-attention within visual tokens to achieve compute efficiency, and 2) Reusing the weights on each layer of LLM to facilitate effective modality alignment between vision and language for data efficiency. Experimental results demonstrate the effectiveness of EE-MLLM across a range of benchmarks, including general-purpose datasets like MMBench and SeedBench, as well as fine-grained tasks such as TextVQA and DocVQA.

Title: Practical token pruning for foundation models in few-shot conversational virtual assistant systems

• Authors: Haode Qi, Cheng Qian, Jian Ni, Pratyush Singh, Reza Fazeli, Gengyu Wang, Zhongzheng Shu, Eric Wayne, Juergen Bross
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11799
• Pdf URL: https://arxiv.org/pdf/2408.11799
• Copy Paste: [[2408.11799]] Practical token pruning for foundation models in few-shot conversational virtual assistant systems(https://arxiv.org/abs/2408.11799)
• Keywords: transformer
• Abstract: In an enterprise Virtual Assistant (VA) system, intent classification is the crucial component that determines how a user input is handled based on what the user wants. The VA system is expected to be a cost-efficient SaaS service with low training and inference time while achieving high accuracy even with a small number of training samples. We pretrain a transformer-based sentence embedding model with a contrastive learning objective and leverage the embedding of the model as features when training intent classification models. Our approach achieves the state-of-the-art results for few-shot scenarios and performs better than other commercial solutions on popular intent classification benchmarks. However, generating features via a transformer-based model increases the inference time, especially for longer user inputs, due to the quadratic runtime of the transformer's attention mechanism. On top of model distillation, we introduce a practical multi-task adaptation approach that configures dynamic token pruning without the need for task-specific training for intent classification. We demonstrate that this approach improves the inference speed of popular sentence transformer models without affecting model performance.

Title: PermitQA: A Benchmark for Retrieval Augmented Generation in Wind Siting and Permitting domain

• Authors: Rounak Meyur, Hung Phan, Sridevi Wagle, Jan Strube, Mahantesh Halappanavar, Sameera Horawalavithana, Anurag Acharya, Sai Munikoti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11800
• Pdf URL: https://arxiv.org/pdf/2408.11800
• Copy Paste: [[2408.11800]] PermitQA: A Benchmark for Retrieval Augmented Generation in Wind Siting and Permitting domain(https://arxiv.org/abs/2408.11800)
• Keywords: large language model
• Abstract: In the rapidly evolving landscape of Natural Language Processing (NLP) and text generation, the emergence of Retrieval Augmented Generation (RAG) presents a promising avenue for improving the quality and reliability of generated text by leveraging information retrieved from user specified database. Benchmarking is essential to evaluate and compare the performance of the different RAG configurations in terms of retriever and generator, providing insights into their effectiveness, scalability, and suitability for the specific domain and applications. In this paper, we present a comprehensive framework to generate a domain relevant RAG benchmark. Our framework is based on automatic question-answer generation with Human (domain experts)-AI Large Language Model (LLM) teaming. As a case study, we demonstrate the framework by introducing PermitQA, a first-of-its-kind benchmark on the wind siting and permitting domain which comprises of multiple scientific documents/reports related to environmental impact of wind energy projects. Our framework systematically evaluates RAG performance using diverse metrics and multiple question types with varying complexity level. We also demonstrate the performance of different models on our benchmark.

Title: Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models

• Authors: Yuzhou Huang, Yiran Qin, Shunlin Lu, Xintao Wang, Rui Huang, Ying Shan, Ruimao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11801
• Pdf URL: https://arxiv.org/pdf/2408.11801
• Copy Paste: [[2408.11801]] Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models(https://arxiv.org/abs/2408.11801)
• Keywords: large language model
• Abstract: Traditional visual storytelling is complex, requiring specialized knowledge and substantial resources, yet often constrained by human creativity and creation precision. While Large Language Models (LLMs) enhance visual storytelling, current approaches often limit themselves to 2D visuals or oversimplify stories through motion synthesis and behavioral simulation, failing to create comprehensive, multi-dimensional narratives. To this end, we present Story3D-Agent, a pioneering approach that leverages the capabilities of LLMs to transform provided narratives into 3D-rendered visualizations. By integrating procedural modeling, our approach enables precise control over multi-character actions and motions, as well as diverse decorative elements, ensuring the long-range and dynamic 3D representation. Furthermore, our method supports narrative extension through logical reasoning, ensuring that generated content remains consistent with existing conditions. We have thoroughly evaluated our Story3D-Agent to validate its effectiveness, offering a basic framework to advance 3D story representation.

Title: Approaching Deep Learning through the Spectral Dynamics of Weights

• Authors: David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, Matthew R. Walter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11804
• Pdf URL: https://arxiv.org/pdf/2408.11804
• Copy Paste: [[2408.11804]] Approaching Deep Learning through the Spectral Dynamics of Weights(https://arxiv.org/abs/2408.11804)
• Keywords: transformer
• Abstract: We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale ``grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers. We also demonstrate that weight decay enhances this bias beyond its role as a norm regularizer, even in practical systems. Moreover, we show that these spectral dynamics distinguish memorizing networks from generalizing ones, offering a novel perspective on this longstanding conundrum. Additionally, we leverage spectral dynamics to explore the emergence of well-performing sparse subnetworks (lottery tickets) and the structure of the loss surface through linear mode connectivity. Our findings suggest that spectral dynamics provide a coherent framework to better understand the behavior of neural networks across diverse settings.

Title: Pixel Is Not A Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models

• Authors: Chun-Yen Shih, Li-Xuan Peng, Jia-Wei Liao, Ernie Chu, Cheng-Fu Chou, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11810
• Pdf URL: https://arxiv.org/pdf/2408.11810
• Copy Paste: [[2408.11810]] Pixel Is Not A Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models(https://arxiv.org/abs/2408.11810)
• Keywords: protect, defense, attack, robust, diffusion, generative
• Abstract: Diffusion Models have emerged as powerful generative models for high-quality image synthesis, with many subsequent image editing techniques based on them. However, the ease of text-based image editing introduces significant risks, such as malicious editing for scams or intellectual property infringement. Previous works have attempted to safeguard images from diffusion-based editing by adding imperceptible perturbations. These methods are costly and specifically target prevalent Latent Diffusion Models (LDMs), while Pixel-domain Diffusion Models (PDMs) remain largely unexplored and robust against such attacks. Our work addresses this gap by proposing a novel attacking framework with a feature representation attack loss that exploits vulnerabilities in denoising UNets and a latent optimization strategy to enhance the naturalness of protected images. Extensive experiments demonstrate the effectiveness of our approach in attacking dominant PDM-based editing methods (e.g., SDEdit) while maintaining reasonable protection fidelity and robustness against common defense methods. Additionally, our framework is extensible to LDMs, achieving comparable performance to existing approaches.

Title: EmbodiedSAM: Online Segment Any 3D Thing in Real Time

• Authors: Xiuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, Jiwen Lu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.11811
• Pdf URL: https://arxiv.org/pdf/2408.11811
• Copy Paste: [[2408.11811]] EmbodiedSAM: Online Segment Any 3D Thing in Real Time(https://arxiv.org/abs/2408.11811)
• Keywords: segmentation
• Abstract: Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration, so an online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed. Since high-quality 3D data is limited, directly training such a model in 3D is almost infeasible. Meanwhile, vision foundation models (VFM) has revolutionized the field of 2D computer vision with superior performance, which makes the use of VFM to assist embodied 3D perception a promising direction. However, most existing VFM-assisted 3D perception methods are either offline or too slow that cannot be applied in practical embodied tasks. In this paper, we aim to leverage Segment Anything Model (SAM) for real-time 3D instance segmentation in an online setting. This is a challenging problem since future frames are not available in the input streaming RGB-D video, and an instance may be observed in several frames so object matching between frames is required. To address these challenges, we first propose a geometric-aware query lifting module to represent the 2D masks generated by SAM by 3D-aware queries, which is then iteratively refined by a dual-level query decoder. In this way, the 2D masks are transferred to fine-grained shapes on 3D point clouds. Benefit from the query representation for 3D masks, we can compute the similarity matrix between the 3D masks from different views by efficient matrix operation, which enables real-time inference. Experiments on ScanNet, ScanNet200, SceneNN and 3RScan show our method achieves leading performance even compared with offline methods. Our method also demonstrates great generalization ability in several zero-shot dataset transferring experiments and show great potential in open-vocabulary and data-efficient setting. Code and demo are available at this https URL, with only one RTX 3090 GPU required for training and evaluation.

Title: SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs

• Authors: Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Di Zhang, Baoqun Yin, Wentao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11813
• Pdf URL: https://arxiv.org/pdf/2408.11813
• Copy Paste: [[2408.11813]] SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs(https://arxiv.org/abs/2408.11813)
• Keywords: interpretability, large language model
• Abstract: Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable perceptual and reasoning abilities, typically comprising a Vision Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as the critical bridge between the visual and language components. However, training adapters with image-level supervision often results in significant misalignment, undermining the LLMs' capabilities and limiting the potential of Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment (SEA), a token-level alignment method that leverages vision-language pre-trained models, such as CLIP, to align visual tokens with the LLM's embedding space through contrastive learning. This approach ensures a more coherent integration of visual and language representations, enhancing the performance and interpretability of multimodal LLMs while preserving their inherent capabilities. Extensive experiments show that SEA effectively improves MLLMs, particularly for smaller models, without adding extra data or inference computation. SEA also lays the groundwork for developing more general and adaptable solutions to enhance multimodal systems.

Title: SynPlay: Importing Real-world Diversity for a Synthetic Human Dataset

• Authors: Jinsub Yim, Hyungtae Lee, Sungmin Eum, Yi-Ting Shen, Yan Zhang, Heesung Kwon, Shuvra S. Bhattacharyya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.11814
• Pdf URL: https://arxiv.org/pdf/2408.11814
• Copy Paste: [[2408.11814]] SynPlay: Importing Real-world Diversity for a Synthetic Human Dataset(https://arxiv.org/abs/2408.11814)
• Keywords: segmentation
• Abstract: We introduce Synthetic Playground (SynPlay), a new synthetic human dataset that aims to bring out the diversity of human appearance in the real world. We focus on two factors to achieve a level of diversity that has not yet been seen in previous works: i) realistic human motions and poses and ii) multiple camera viewpoints towards human instances. We first use a game engine and its library-provided elementary motions to create games where virtual players can take less-constrained and natural movements while following the game rules (i.e., rule-guided motion design as opposed to detail-guided design). We then augment the elementary motions with real human motions captured with a motion capture device. To render various human appearances in the games from multiple viewpoints, we use seven virtual cameras encompassing the ground and aerial views, capturing abundant aerial-vs-ground and dynamic-vs-static attributes of the scene. Through extensive and carefully-designed experiments, we show that using SynPlay in model training leads to enhanced accuracy over existing synthetic datasets for human detection and segmentation. The benefit of SynPlay becomes even greater for tasks in the data-scarce regime, such as few-shot and cross-domain learning tasks. These results clearly demonstrate that SynPlay can be used as an essential dataset with rich attributes of complex human appearances and poses suitable for model pretraining. SynPlay dataset comprising over 73k images and 6.5M human instances, is available for download at this https URL.