secure

Title: Functional Encryption in the Bounded Storage Models. (arXiv:2309.06702v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06702
• Code URL: null
• Copy Paste: [[2309.06702]] Functional Encryption in the Bounded Storage Models(http://arxiv.org/abs/2309.06702)
• Summary:

  Functional encryption is a powerful paradigm for public-key encryption which allows for controlled access to encrypted data. This primitive is generally impossible in the standard setting so we investigate possibilities in the bounded quantum storage model (BQSM) and the bounded classical storage model (BCSM). In these models, ciphertexts potentially disappear which nullifies impossibility results and allows us to obtain positive outcomes.

  Firstly, in the BQSM, we construct information-theoretically secure functional encryption with $\texttt{q}=O(\sqrt{\texttt{s}/\texttt{r}})$ where $\texttt{r}$ can be set to any value less than $\texttt{s}$. Here $\texttt{r}$ denotes the number of times that an adversary is restricted to $\texttt{s}$--qubits of quantum memory in the protocol and $\texttt{q}$ denotes the required quantum memory to run the protocol honestly. We then show that our scheme is optimal by proving that it is impossible to attain information-theoretically secure functional encryption with $\texttt{q} < \sqrt{\texttt{s}/\texttt{r}}$. However, by assuming the existence of post-quantum one-way functions, we can do far better and achieve functional encryption with classical keys and with $\texttt{q}=0$ and $\texttt{r}=1$.

  Secondly, in the BCSM, we construct $(O(\texttt{n}),\texttt{n}^2)$ functional encryption assuming the existence of $(\texttt{n},\texttt{n}^2)$ virtual weak grey-box obfuscation. Here, the pair $(\texttt{n},\texttt{n}^2)$ indicates the required memory to run honestly and the needed memory to break security, respectively. This memory gap is optimal and the assumption is minimal. In particular, we also construct $(O(\texttt{n}),\texttt{n}^2)$ virtual weak grey-box obfuscation assuming $(\texttt{n},\texttt{n}^2)$ functional encryption.

security

Title: Evaluating Homomorphic Operations on a Real-World Processing-In-Memory System. (arXiv:2309.06545v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06545
• Code URL: null
• Copy Paste: [[2309.06545]] Evaluating Homomorphic Operations on a Real-World Processing-In-Memory System(http://arxiv.org/abs/2309.06545)
• Summary:

  Computing on encrypted data is a promising approach to reduce data security and privacy risks, with homomorphic encryption serving as a facilitator in achieving this goal. In this work, we accelerate homomorphic operations using the Processing-in- Memory (PIM) paradigm to mitigate the large memory capacity and frequent data movement requirements. Using a real-world PIM system, we accelerate the Brakerski-Fan-Vercauteren (BFV) scheme for homomorphic addition and multiplication. We evaluate the PIM implementations of these homomorphic operations with statistical workloads (arithmetic mean, variance, linear regression) and compare to CPU and GPU implementations. Our results demonstrate 50-100x speedup with a real PIM system (UPMEM) over the CPU and 2-15x over the GPU in vector addition. For vector multiplication, the real PIM system outperforms the CPU by 40-50x. However, it lags 10-15x behind the GPU due to the lack of native sufficiently wide multiplication support in the evaluated first-generation real PIM system. For mean, variance, and linear regression, the real PIM system performance improvements vary between 30x and 300x over the CPU and between 10x and 30x over the GPU, uncovering real PIM system trade-offs in terms of scalability of homomorphic operations for varying amounts of data. We plan to make our implementation open-source in the future.

privacy

Title: Level Up: Private Non-Interactive Decision Tree Evaluation using Levelled Homomorphic Encryption. (arXiv:2309.06496v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06496
• Code URL: null
• Copy Paste: [[2309.06496]] Level Up: Private Non-Interactive Decision Tree Evaluation using Levelled Homomorphic Encryption(http://arxiv.org/abs/2309.06496)
• Summary:

  As machine learning as a service continues gaining popularity, concerns about privacy and intellectual property arise. Users often hesitate to disclose their private information to obtain a service, while service providers aim to protect their proprietary models. Decision trees, a widely used machine learning model, are favoured for their simplicity, interpretability, and ease of training. In this context, Private Decision Tree Evaluation (PDTE) enables a server holding a private decision tree to provide predictions based on a client's private attributes. The protocol is such that the server learns nothing about the client's private attributes. Similarly, the client learns nothing about the server's model besides the prediction and some hyperparameters.

  In this paper, we propose two novel non-interactive PDTE protocols, XXCMP-PDTE and RCC-PDTE, based on two new non-interactive comparison protocols, XXCMP and RCC. Our evaluation of these comparison operators demonstrates that our proposed constructions can efficiently evaluate high-precision numbers. Specifically, RCC can compare 32-bit numbers in under 10 milliseconds.

  We assess our proposed PDTE protocols on decision trees trained over UCI datasets and compare our results with existing work in the field. Moreover, we evaluate synthetic decision trees to showcase scalability, revealing that RCC-PDTE can evaluate a decision tree with over 1000 nodes and 16 bits of precision in under 2 seconds. In contrast, the current state-of-the-art requires over 10 seconds to evaluate such a tree with only 11 bits of precision.

Title: Deciding Differential Privacy of Online Algorithms with Multiple Variables. (arXiv:2309.06615v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06615
• Code URL: null
• Copy Paste: [[2309.06615]] Deciding Differential Privacy of Online Algorithms with Multiple Variables(http://arxiv.org/abs/2309.06615)
• Summary:

  We consider the problem of checking the differential privacy of online randomized algorithms that process a stream of inputs and produce outputs corresponding to each input. This paper generalizes an automaton model called DiP automata (See arXiv:2104.14519) to describe such algorithms by allowing multiple real-valued storage variables. A DiP automaton is a parametric automaton whose behavior depends on the privacy budget $\epsilon$. An automaton $A$ will be said to be differentially private if, for some $\mathfrak{D}$, the automaton is $\mathfrak{D}\epsilon$-differentially private for all values of $\epsilon>0$. We identify a precise characterization of the class of all differentially private DiP automata. We show that the problem of determining if a given DiP automaton belongs to this class is PSPACE-complete. Our PSPACE algorithm also computes a value for $\mathfrak{D}$ when the given automaton is differentially private. The algorithm has been implemented, and experiments demonstrating its effectiveness are presented.

Title: DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass. (arXiv:2309.06746v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06746
• Code URL: null
• Copy Paste: [[2309.06746]] DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass(http://arxiv.org/abs/2309.06746)
• Summary:

  Differentially private stochastic gradient descent (DP-SGD) adds noise to gradients in back-propagation, safeguarding training data from privacy leakage, particularly membership inference. It fails to cover (inference-time) threats like embedding inversion and sensitive attribute inference. It is also costly in storage and computation when used to fine-tune large pre-trained language models (LMs).

  We propose DP-Forward, which directly perturbs embedding matrices in the forward pass of LMs. It satisfies stringent local DP requirements for training and inference data. To instantiate it using the smallest matrix-valued noise, we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possibly non-i.i.d. noise from a matrix Gaussian distribution. We then investigate perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises. Its utility on three typical tasks almost hits the non-private baseline and outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves 3$\times$ time and memory costs compared to DP-SGD with the latest high-speed library. It also reduces the average success rates of embedding inversion and sensitive attribute inference by up to 88pp and 41pp, respectively, whereas DP-SGD fails.

Title: ZKROWNN: Zero Knowledge Right of Ownership for Neural Networks. (arXiv:2309.06779v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06779
• Code URL: null
• Copy Paste: [[2309.06779]] ZKROWNN: Zero Knowledge Right of Ownership for Neural Networks(http://arxiv.org/abs/2309.06779)
• Summary:

  Training contemporary AI models requires investment in procuring learning data and computing resources, making the models intellectual property of the owners. Popular model watermarking solutions rely on key input triggers for detection; the keys have to be kept private to prevent discovery, forging, and removal of the hidden signatures. We present ZKROWNN, the first automated end-to-end framework utilizing Zero-Knowledge Proofs (ZKP) that enable an entity to validate their ownership of a model, while preserving the privacy of the watermarks. ZKROWNN permits a third party client to verify model ownership in less than a second, requiring as little as a few KBs of communication.

Title: Robustness for Spectral Clustering of General Graphs under Local Differential Privacy. (arXiv:2309.06867v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06867
• Code URL: null
• Copy Paste: [[2309.06867]] Robustness for Spectral Clustering of General Graphs under Local Differential Privacy(http://arxiv.org/abs/2309.06867)
• Summary:

  Spectral clustering is a widely used algorithm to find clusters in networks. Several researchers have studied the stability of spectral clustering under local differential privacy with the additional assumption that the underlying networks are generated from the stochastic block model (SBM). However, we argue that this assumption is too restrictive since social networks do not originate from the SBM. Thus, delve into an analysis for general graphs in this work. Our primary focus is the edge flipping method -- a common technique for protecting local differential privacy. On a positive side, our findings suggest that even when the edges of an $n$-vertex graph satisfying some reasonable well-clustering assumptions are flipped with a probability of $O(\log n/n)$, the clustering outcomes are largely consistent. Empirical tests further corroborate these theoretical findings. Conversely, although clustering outcomes have been stable for dense and well-clustered graphs produced from the SBM, we show that in general, spectral clustering may yield highly erratic results on certain dense and well-clustered graphs when the flipping probability is $\omega(\log n/n)$. This indicates that the best privacy budget obtainable for general graphs is $\Theta(\log n)$.

Title: Communication-Efficient Laplace Mechanism for Differential Privacy via Random Quantization. (arXiv:2309.06982v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06982
• Code URL: null
• Copy Paste: [[2309.06982]] Communication-Efficient Laplace Mechanism for Differential Privacy via Random Quantization(http://arxiv.org/abs/2309.06982)
• Summary:

  We propose the first method that realizes the Laplace mechanism exactly (i.e., a Laplace noise is added to the data) that requires only a finite amount of communication (whereas the original Laplace mechanism requires the transmission of a real number) while guaranteeing privacy against the server and database. Our mechanism can serve as a drop-in replacement for local or centralized differential privacy applications where the Laplace mechanism is used. Our mechanism is constructed using a random quantization technique. Unlike the simple and prevalent Laplace-mechanism-then-quantize approach, the quantization in our mechanism does not result in any distortion or degradation of utility. Unlike existing dithered quantization and channel simulation schemes for simulating additive Laplacian noise, our mechanism guarantees privacy not only against the database and downstream, but also against the honest but curious server which attempts to decode the data using the dither signals.

Title: Chained-DP: Can We Recycle Privacy Budget?. (arXiv:2309.07075v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.07075
• Code URL: null
• Copy Paste: [[2309.07075]] Chained-DP: Can We Recycle Privacy Budget?(http://arxiv.org/abs/2309.07075)
• Summary:

  Privacy-preserving vector mean estimation is a crucial primitive in federated analytics. Existing practices usually resort to Local Differentiated Privacy (LDP) mechanisms that inject random noise into users' vectors when communicating with users and the central server. Due to the privacy-utility trade-off, the privacy budget has been widely recognized as the bottleneck resource that requires well-provisioning. In this paper, we explore the possibility of privacy budget recycling and propose a novel Chained-DP framework enabling users to carry out data aggregation sequentially to recycle the privacy budget. We establish a sequential game to model the user interactions in our framework. We theoretically show the mathematical nature of the sequential game, solve its Nash Equilibrium, and design an incentive mechanism with provable economic properties. We further derive a differentially privacy-guaranteed protocol to alleviate potential privacy collusion attacks to avoid holistic exposure. Our numerical simulation validates the effectiveness of Chained-DP, showing that it can significantly save privacy budget and lower estimation error compared to the traditional LDP mechanism.

protect

Title: SHARM: Segmented Head Anatomical Reference Models. (arXiv:2309.06677v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06677
• Code URL: null
• Copy Paste: [[2309.06677]] SHARM: Segmented Head Anatomical Reference Models(http://arxiv.org/abs/2309.06677)
• Summary:

  Reliable segmentation of anatomical tissues of human head is a major step in several clinical applications such as brain mapping, surgery planning and associated computational simulation studies. Segmentation is based on identifying different anatomical structures through labeling different tissues through medical imaging modalities. The segmentation of brain structures is commonly feasible with several remarkable contributions mainly for medical perspective; however, non-brain tissues are of less interest due to anatomical complexity and difficulties to be observed using standard medical imaging protocols. The lack of whole head segmentation methods and unavailability of large human head segmented datasets limiting the variability studies, especially in the computational evaluation of electrical brain stimulation (neuromodulation), human protection from electromagnetic field, and electroencephalography where non-brain tissues are of great importance.

  To fill this gap, this study provides an open-access Segmented Head Anatomical Reference Models (SHARM) that consists of 196 subjects. These models are segmented into 15 different tissues; skin, fat, muscle, skull cancellous bone, skull cortical bone, brain white matter, brain gray matter, cerebellum white matter, cerebellum gray matter, cerebrospinal fluid, dura, vitreous humor, lens, mucous tissue and blood vessels. The segmented head models are generated using open-access IXI MRI dataset through convolutional neural network structure named ForkNet+. Results indicate a high consistency in statistical characteristics of different tissue distribution in age scale with real measurements. SHARM is expected to be a useful benchmark not only for electromagnetic dosimetry studies but also for different human head segmentation applications.

defense

Title: Deep Nonparametric Convexified Filtering for Computational Photography, Image Synthesis and Adversarial Defense. (arXiv:2309.06724v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06724
• Code URL: null
• Copy Paste: [[2309.06724]] Deep Nonparametric Convexified Filtering for Computational Photography, Image Synthesis and Adversarial Defense(http://arxiv.org/abs/2309.06724)
• Summary:

  We aim to provide a general framework of for computational photography that recovers the real scene from imperfect images, via the Deep Nonparametric Convexified Filtering (DNCF). It is consists of a nonparametric deep network to resemble the physical equations behind the image formation, such as denoising, super-resolution, inpainting, and flash. DNCF has no parameterization dependent on training data, therefore has a strong generalization and robustness to adversarial image manipulation. During inference, we also encourage the network parameters to be nonnegative and create a bi-convex function on the input and parameters, and this adapts to second-order optimization algorithms with insufficient running time, having 10X acceleration over Deep Image Prior. With these tools, we empirically verify its capability to defend image classification deep networks against adversary attack algorithms in real-time.

Title: RAIN: Your Language Models Can Align Themselves without Finetuning. (arXiv:2309.07124v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.07124
• Code URL: null
• Copy Paste: [[2309.07124]] RAIN: Your Language Models Can Align Themselves without Finetuning(http://arxiv.org/abs/2309.07124)
• Summary:

  Large language models (LLMs) often demonstrate inconsistencies with human preferences. Previous research gathered human preference data and then aligned the pre-trained models using reinforcement learning or instruction tuning, the so-called finetuning step. In contrast, aligning frozen LLMs without any extra data is more appealing. This work explores the potential of the latter setting. We discover that by integrating self-evaluation and rewind mechanisms, unaligned LLMs can directly produce responses consistent with human preferences via self-boosting. We introduce a novel inference method, Rewindable Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate their own generation and use the evaluation results to guide backward rewind and forward generation for AI safety. Notably, RAIN operates without the need of extra data for model alignment and abstains from any training, gradient computation, or parameter updates; during the self-evaluation phase, the model receives guidance on which human preference to align with through a fixed-template prompt, eliminating the need to modify the initial prompt. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate of LLaMA 30B over vanilla inference from 82% to 97%, while maintaining the helpfulness rate. Under the leading adversarial attack llm-attacks on Vicuna 33B, RAIN establishes a new defense baseline by reducing the attack success rate from 94% to 19%.

attack

Title: Differentiable JPEG: The Devil is in the Details. (arXiv:2309.06978v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06978
• Code URL: https://github.com/necla-ml/diff-jpeg
• Copy Paste: [[2309.06978]] Differentiable JPEG: The Devil is in the Details(http://arxiv.org/abs/2309.06978)
• Summary:

  JPEG remains one of the most widespread lossy image coding methods. However, the non-differentiable nature of JPEG restricts the application in deep learning pipelines. Several differentiable approximations of JPEG have recently been proposed to address this issue. This paper conducts a comprehensive review of existing diff. JPEG approaches and identifies critical details that have been missed by previous methods. To this end, we propose a novel diff. JPEG approach, overcoming previous limitations. Our approach is differentiable w.r.t. the input image, the JPEG quality, the quantization tables, and the color conversion parameters. We evaluate the forward and backward performance of our diff. JPEG approach against existing methods. Additionally, extensive ablations are performed to evaluate crucial design choices. Our proposed diff. JPEG resembles the (non-diff.) reference implementation best, significantly surpassing the recent-best diff. approach by $3.47$dB (PSNR) on average. For strong compression rates, we can even improve PSNR by $9.51$dB. Strong adversarial attack results are yielded by our diff. JPEG, demonstrating the effective gradient approximation. Our code is available at https://github.com/necla-ml/Diff-JPEG.

Title: Hardening RGB-D Object Recognition Systems against Adversarial Patch Attacks. (arXiv:2309.07106v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.07106
• Code URL: null
• Copy Paste: [[2309.07106]] Hardening RGB-D Object Recognition Systems against Adversarial Patch Attacks(http://arxiv.org/abs/2309.07106)
• Summary:

  RGB-D object recognition systems improve their predictive performances by fusing color and depth information, outperforming neural network architectures that rely solely on colors. While RGB-D systems are expected to be more robust to adversarial examples than RGB-only systems, they have also been proven to be highly vulnerable. Their robustness is similar even when the adversarial examples are generated by altering only the original images' colors. Different works highlighted the vulnerability of RGB-D systems; however, there is a lacking of technical explanations for this weakness. Hence, in our work, we bridge this gap by investigating the learned deep representation of RGB-D systems, discovering that color features make the function learned by the network more complex and, thus, more sensitive to small perturbations. To mitigate this problem, we propose a defense based on a detection mechanism that makes RGB-D systems more robust against adversarial examples. We empirically show that this defense improves the performances of RGB-D systems against adversarial examples even when they are computed ad-hoc to circumvent this detection mechanism, and that is also more effective than adversarial training.

Title: Machine Translation Models Stand Strong in the Face of Adversarial Attacks. (arXiv:2309.06527v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06527
• Code URL: null
• Copy Paste: [[2309.06527]] Machine Translation Models Stand Strong in the Face of Adversarial Attacks(http://arxiv.org/abs/2309.06527)
• Summary:

  Adversarial attacks expose vulnerabilities of deep learning models by introducing minor perturbations to the input, which lead to substantial alterations in the output. Our research focuses on the impact of such adversarial attacks on sequence-to-sequence (seq2seq) models, specifically machine translation models. We introduce algorithms that incorporate basic text perturbation heuristics and more advanced strategies, such as the gradient-based attack, which utilizes a differentiable approximation of the inherently non-differentiable translation metric. Through our investigation, we provide evidence that machine translation models display robustness displayed robustness against best performed known adversarial attacks, as the degree of perturbation in the output is directly proportional to the perturbation in the input. However, among underdogs, our attacks outperform alternatives, providing the best relative performance. Another strong candidate is an attack based on mixing of individual characters.

Title: Pump, Dump, and then What? The Long-Term Impact of Cryptocurrency Pump-and-Dump Schemes. (arXiv:2309.06608v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06608
• Code URL: null
• Copy Paste: [[2309.06608]] Pump, Dump, and then What? The Long-Term Impact of Cryptocurrency Pump-and-Dump Schemes(http://arxiv.org/abs/2309.06608)
• Summary:

  The pump and dump scheme is a form of market manipulation attack in which coordinated actors drive up the price of an asset in order to sell at a higher price. Due in part to a lack of enforcement, these schemes are widespread within the cryptocurrency marketplace, but the negative impact of these events on the coins they target is not yet fully understood. Drawing upon a novel dataset of pump events extracted from Telegram channels, an order of magnitude larger than the nearest comparable dataset in the literature, we explore the differing tactics of pumping channels and the long-term impact of pump and dump schemes across 765 coins. We find that, despite a short-term positive impact in some cases, the long-term impact of pump and dump schemes on the targeted assets is negative, amounting to an average 30% relative drop in price a year after the pump event.

Title: PhantomSound: Black-Box, Query-Efficient Audio Adversarial Attack via Split-Second Phoneme Injection. (arXiv:2309.06960v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06960
• Code URL: null
• Copy Paste: [[2309.06960]] PhantomSound: Black-Box, Query-Efficient Audio Adversarial Attack via Split-Second Phoneme Injection(http://arxiv.org/abs/2309.06960)
• Summary:

  In this paper, we propose PhantomSound, a query-efficient black-box attack toward voice assistants. Existing black-box adversarial attacks on voice assistants either apply substitution models or leverage the intermediate model output to estimate the gradients for crafting adversarial audio samples. However, these attack approaches require a significant amount of queries with a lengthy training stage. PhantomSound leverages the decision-based attack to produce effective adversarial audios, and reduces the number of queries by optimizing the gradient estimation. In the experiments, we perform our attack against 4 different speech-to-text APIs under 3 real-world scenarios to demonstrate the real-time attack impact. The results show that PhantomSound is practical and robust in attacking 5 popular commercial voice controllable devices over the air, and is able to bypass 3 liveness detection mechanisms with >95% success rate. The benchmark result shows that PhantomSound can generate adversarial examples and launch the attack in a few minutes. We significantly enhance the query efficiency and reduce the cost of a successful untargeted and targeted adversarial attack by 93.1% and 65.5% compared with the state-of-the-art black-box attacks, using merely ~300 queries (~5 minutes) and ~1,500 queries (~25 minutes), respectively.

Title: MASTERKEY: Practical Backdoor Attack Against Speaker Verification Systems. (arXiv:2309.06981v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06981
• Code URL: null
• Copy Paste: [[2309.06981]] MASTERKEY: Practical Backdoor Attack Against Speaker Verification Systems(http://arxiv.org/abs/2309.06981)
• Summary:

  Speaker Verification (SV) is widely deployed in mobile systems to authenticate legitimate users by using their voice traits. In this work, we propose a backdoor attack MASTERKEY, to compromise the SV models. Different from previous attacks, we focus on a real-world practical setting where the attacker possesses no knowledge of the intended victim. To design MASTERKEY, we investigate the limitation of existing poisoning attacks against unseen targets. Then, we optimize a universal backdoor that is capable of attacking arbitrary targets. Next, we embed the speaker's characteristics and semantics information into the backdoor, making it imperceptible. Finally, we estimate the channel distortion and integrate it into the backdoor. We validate our attack on 6 popular SV models. Specifically, we poison a total of 53 models and use our trigger to attack 16,430 enrolled speakers, composed of 310 target speakers enrolled in 53 poisoned models. Our attack achieves 100% attack success rate with a 15% poison rate. By decreasing the poison rate to 3%, the attack success rate remains around 50%. We validate our attack in 3 real-world scenarios and successfully demonstrate the attack through both over-the-air and over-the-telephony-line scenarios.

Title: Cryptography: Against AI and QAI Odds. (arXiv:2309.07022v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.07022
• Code URL: null
• Copy Paste: [[2309.07022]] Cryptography: Against AI and QAI Odds(http://arxiv.org/abs/2309.07022)
• Summary:

  Artificial Intelligence (AI) presents prodigious technological prospects for development, however, all that glitters is not gold! The cyber-world faces the worst nightmare with the advent of AI and quantum computers. Together with Quantum Artificial Intelligence (QAI), they pose a catastrophic threat to modern cryptography. It would also increase the capability of cryptanalysts manifold, with its built-in persistent and extensive predictive intelligence. This prediction ability incapacitates the constrained message space in device cryptography. With the comparison of these assumptions and the intercepted ciphertext, the code-cracking process will considerably accelerate. Before the vigorous and robust developments in AI, we have never faced and never had to prepare for such a plaintext-originating attack. The supremacy of AI can be challenged by creating ciphertexts that would give the AI attacker erroneous responses stymied by randomness and misdirect them. AI threat is deterred by deviating from the conventional use of small, known-size keys and pattern-loaded ciphers. The strategy is vested in implementing larger secret size keys, supplemented by ad-hoc unilateral randomness of unbound limitations and a pattern-devoid technique. The very large key size can be handled with low processing and computational burden to achieve desired unicity distances. The strategy against AI odds is feasible by implementing non-algorithmic randomness, large and inexpensive memory chips, and wide-area communication networks. The strength of AI, i.e., randomness and pattern detection can be used to generate highly optimized ciphers and algorithms. These pattern-devoid, randomness-rich ciphers also provide a timely and plausible solution for NIST's proactive approach toward the quantum challenge.

Title: Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss. (arXiv:2309.06774v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06774
• Code URL: null
• Copy Paste: [[2309.06774]] Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss(http://arxiv.org/abs/2309.06774)
• Summary:

  Although deep learning (DL) has led to several breakthroughs in many disciplines as diverse as chemistry, computer science, electrical engineering, mathematics, medicine, neuroscience, and physics, a comprehensive understanding of why and how DL is empirically successful remains fundamentally elusive. To attack this fundamental problem and unravel the mysteries behind DL's empirical successes, significant innovations toward a unified theory of DL have been made. These innovations encompass nearly fundamental advances in optimization, generalization, and approximation. Despite these advances, however, no work to date has offered a way to quantify the testing performance of a DL-based algorithm employed to solve a pattern classification problem. To overcome this fundamental challenge in part, this paper exposes the fundamental testing performance limits of DL-based binary classifiers trained with hinge loss. For binary classifiers that are based on deep rectified linear unit (ReLU) feedforward neural networks (FNNs) and ones that are based on deep FNNs with ReLU and Tanh activation, we derive their respective novel asymptotic testing performance limits. The derived testing performance limits are validated by extensive computer experiments.

robust

Title: Deep Attentive Time Warping. (arXiv:2309.06720v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06720
• Code URL: null
• Copy Paste: [[2309.06720]] Deep Attentive Time Warping(http://arxiv.org/abs/2309.06720)
• Summary:

  Similarity measures for time series are important problems for time series classification. To handle the nonlinear time distortions, Dynamic Time Warping (DTW) has been widely used. However, DTW is not learnable and suffers from a trade-off between robustness against time distortion and discriminative power. In this paper, we propose a neural network model for task-adaptive time warping. Specifically, we use the attention model, called the bipartite attention model, to develop an explicit time warping mechanism with greater distortion invariance. Unlike other learnable models using DTW for warping, our model predicts all local correspondences between two time series and is trained based on metric learning, which enables it to learn the optimal data-dependent warping for the target task. We also propose to induce pre-training of our model by DTW to improve the discriminative power. Extensive experiments demonstrate the superior effectiveness of our model over DTW and its state-of-the-art performance in online signature verification.

Title: Remote Sensing Object Detection Meets Deep Learning: A Meta-review of Challenges and Advances. (arXiv:2309.06751v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06751
• Code URL: null
• Copy Paste: [[2309.06751]] Remote Sensing Object Detection Meets Deep Learning: A Meta-review of Challenges and Advances(http://arxiv.org/abs/2309.06751)
• Summary:

  Remote sensing object detection (RSOD), one of the most fundamental and challenging tasks in the remote sensing field, has received longstanding attention. In recent years, deep learning techniques have demonstrated robust feature representation capabilities and led to a big leap in the development of RSOD techniques. In this era of rapid technical evolution, this review aims to present a comprehensive review of the recent achievements in deep learning based RSOD methods. More than 300 papers are covered in this review. We identify five main challenges in RSOD, including multi-scale object detection, rotated object detection, weak object detection, tiny object detection, and object detection with limited supervision, and systematically review the corresponding methods developed in a hierarchical division manner. We also review the widely used benchmark datasets and evaluation metrics within the field of RSOD, as well as the application scenarios for RSOD. Future research directions are provided for further promoting the research in RSOD.

Title: MagiCapture: High-Resolution Multi-Concept Portrait Customization. (arXiv:2309.06895v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06895
• Code URL: null
• Copy Paste: [[2309.06895]] MagiCapture: High-Resolution Multi-Concept Portrait Customization(http://arxiv.org/abs/2309.06895)
• Summary:

  Large-scale text-to-image models including Stable Diffusion are capable of generating high-fidelity photorealistic portrait images. There is an active research area dedicated to personalizing these models, aiming to synthesize specific subjects or styles using provided sets of reference images. However, despite the plausible results from these personalization methods, they tend to produce images that often fall short of realism and are not yet on a commercially viable level. This is particularly noticeable in portrait image generation, where any unnatural artifact in human faces is easily discernible due to our inherent human bias. To address this, we introduce MagiCapture, a personalization method for integrating subject and style concepts to generate high-resolution portrait images using just a few subject and style references. For instance, given a handful of random selfies, our fine-tuned model can generate high-quality portrait images in specific styles, such as passport or profile photos. The main challenge with this task is the absence of ground truth for the composed concepts, leading to a reduction in the quality of the final output and an identity shift of the source subject. To address these issues, we present a novel Attention Refocusing loss coupled with auxiliary priors, both of which facilitate robust learning within this weakly supervised learning setting. Our pipeline also includes additional post-processing steps to ensure the creation of highly realistic outputs. MagiCapture outperforms other baselines in both quantitative and qualitative evaluations and can also be generalized to other non-human objects.

Title: Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast. (arXiv:2309.06924v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06924
• Code URL: null
• Copy Paste: [[2309.06924]] Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast(http://arxiv.org/abs/2309.06924)
• Summary:

  Video-based remote physiological measurement utilizes facial videos to measure the blood volume change signal, which is also called remote photoplethysmography (rPPG). Supervised methods for rPPG measurements have been shown to achieve good performance. However, the drawback of these methods is that they require facial videos with ground truth (GT) physiological signals, which are often costly and difficult to obtain. In this paper, we propose Contrast-Phys+, a method that can be trained in both unsupervised and weakly-supervised settings. We employ a 3DCNN model to generate multiple spatiotemporal rPPG signals and incorporate prior knowledge of rPPG into a contrastive loss function. We further incorporate the GT signals into contrastive learning to adapt to partial or misaligned labels. The contrastive loss encourages rPPG/GT signals from the same video to be grouped together, while pushing those from different videos apart. We evaluate our methods on five publicly available datasets that include both RGB and Near-infrared videos. Contrast-Phys+ outperforms the state-of-the-art supervised methods, even when using partially available or misaligned GT signals, or no labels at all. Additionally, we highlight the advantages of our methods in terms of computational efficiency, noise robustness, and generalization.

Title: Offline Prompt Evaluation and Optimization with Inverse Reinforcement Learning. (arXiv:2309.06553v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06553
• Code URL: null
• Copy Paste: [[2309.06553]] Offline Prompt Evaluation and Optimization with Inverse Reinforcement Learning(http://arxiv.org/abs/2309.06553)
• Summary:

  The recent advances in the development of Large Language Models (LLMs) like ChatGPT have achieved remarkable performance by leveraging human expertise. Yet, fully eliciting LLMs' potential for complex tasks requires navigating the vast search space of natural language prompts. While prompt engineering has shown promise, the requisite human-crafted prompts in trial-and-error attempts and the associated costs pose significant challenges. Crucially, the efficiency of prompt optimization hinges on the costly procedure of prompt evaluation. This work introduces Prompt-OIRL, an approach rooted in offline inverse reinforcement learning that seeks to bridge the gap between effective prompt evaluation and affordability. Our method draws on offline datasets from expert evaluations, employing Inverse-RL to derive a reward model for offline, query-dependent prompt evaluations. The advantages of Prompt-OIRL are manifold: it predicts prompt performance, is cost-efficient, produces human-readable results, and efficiently navigates the prompt space. We validate our method across four LLMs and three arithmetic datasets, highlighting its potential as a robust and effective tool for offline prompt evaluation and optimization. Our code as well as the offline datasets are released, and we highlight the Prompt-OIRL can be reproduced within a few hours using a single laptop using CPU

Title: Can humans help BERT gain "confidence"?. (arXiv:2309.06580v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06580
• Code URL: null
• Copy Paste: [[2309.06580]] Can humans help BERT gain "confidence"?(http://arxiv.org/abs/2309.06580)
• Summary:

  The advancements in artificial intelligence over the last decade have opened a multitude of avenues for interdisciplinary research. Since the idea of artificial intelligence was inspired by the working of neurons in the brain, it seems pretty practical to combine the two fields and take the help of cognitive data to train AI models. Not only it will help to get a deeper understanding of the technology, but of the brain as well. In this thesis, I conduct novel experiments to integrate cognitive features from the Zurich Cognitive Corpus (ZuCo) (Hollenstein et al., 2018) with a transformer-based encoder model called BERT. I show how EEG and eye-tracking features from ZuCo can help to increase the performance of the NLP model. I confirm the performance increase with the help of a robustness-checking pipeline and derive a word-EEG lexicon to use in benchmarking on an external dataset that does not have any cognitive features associated with it. Further, I analyze the internal working mechanism of BERT and explore a potential method for model explainability by correlating it with a popular model-agnostic explainability framework called LIME (Ribeiro et al., 2016). Finally, I discuss the possible directions to take this research forward.

Title: A Comprehensive Analysis of the Role of Artificial Intelligence and Machine Learning in Modern Digital Forensics and Incident Response. (arXiv:2309.07064v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.07064
• Code URL: null
• Copy Paste: [[2309.07064]] A Comprehensive Analysis of the Role of Artificial Intelligence and Machine Learning in Modern Digital Forensics and Incident Response(http://arxiv.org/abs/2309.07064)
• Summary:

  In the dynamic landscape of digital forensics, the integration of Artificial Intelligence (AI) and Machine Learning (ML) stands as a transformative technology, poised to amplify the efficiency and precision of digital forensics investigations. However, the use of ML and AI in digital forensics is still in its nascent stages. As a result, this paper gives a thorough and in-depth analysis that goes beyond a simple survey and review. The goal is to look closely at how AI and ML techniques are used in digital forensics and incident response. This research explores cutting-edge research initiatives that cross domains such as data collection and recovery, the intricate reconstruction of cybercrime timelines, robust big data analysis, pattern recognition, safeguarding the chain of custody, and orchestrating responsive strategies to hacking incidents. This endeavour digs far beneath the surface to unearth the intricate ways AI-driven methodologies are shaping these crucial facets of digital forensics practice. While the promise of AI in digital forensics is evident, the challenges arising from increasing database sizes and evolving criminal tactics necessitate ongoing collaborative research and refinement within the digital forensics profession. This study examines the contributions, limitations, and gaps in the existing research, shedding light on the potential and limitations of AI and ML techniques. By exploring these different research areas, we highlight the critical need for strategic planning, continual research, and development to unlock AI's full potential in digital forensics and incident response. Ultimately, this paper underscores the significance of AI and ML integration in digital forensics, offering insights into their benefits, drawbacks, and broader implications for tackling modern cyber threats.

Title: Distributionally Robust Transfer Learning. (arXiv:2309.06534v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06534
• Code URL: null
• Copy Paste: [[2309.06534]] Distributionally Robust Transfer Learning(http://arxiv.org/abs/2309.06534)
• Summary:

  Many existing transfer learning methods rely on leveraging information from source data that closely resembles the target data. However, this approach often overlooks valuable knowledge that may be present in different yet potentially related auxiliary samples. When dealing with a limited amount of target data and a diverse range of source models, our paper introduces a novel approach, Distributionally Robust Optimization for Transfer Learning (TransDRO), that breaks free from strict similarity constraints. TransDRO is designed to optimize the most adversarial loss within an uncertainty set, defined as a collection of target populations generated as a convex combination of source distributions that guarantee excellent prediction performances for the target data. TransDRO effectively bridges the realms of transfer learning and distributional robustness prediction models. We establish the identifiability of TransDRO and its interpretation as a weighted average of source models closest to the baseline model. We also show that TransDRO achieves a faster convergence rate than the model fitted with the target data. Our comprehensive numerical studies and analysis of multi-institutional electronic health records data using TransDRO further substantiate the robustness and accuracy of TransDRO, highlighting its potential as a powerful tool in transfer learning applications.

Title: Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources: A Hierarchical Agent-based Approach. (arXiv:2309.06604v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06604
• Code URL: null
• Copy Paste: [[2309.06604]] Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources: A Hierarchical Agent-based Approach(http://arxiv.org/abs/2309.06604)
• Summary:

  Algorithm selection and hyperparameter tuning are critical steps in both academic and applied machine learning. On the other hand, these steps are becoming ever increasingly delicate due to the extensive rise in the number, diversity, and distributedness of machine learning resources. Multi-agent systems, when applied to the design of machine learning platforms, bring about several distinctive characteristics such as scalability, flexibility, and robustness, just to name a few. This paper proposes a fully automatic and collaborative agent-based mechanism for selecting distributedly organized machine learning algorithms and simultaneously tuning their hyperparameters. Our method builds upon an existing agent-based hierarchical machine-learning platform and augments its query structure to support the aforementioned functionalities without being limited to specific learning, selection, and tuning mechanisms. We have conducted theoretical assessments, formal verification, and analytical study to demonstrate the correctness, resource utilization, and computational efficiency of our technique. According to the results, our solution is totally correct and exhibits linear time and space complexity in relation to the size of available resources. To provide concrete examples of how the proposed methodologies can effectively adapt and perform across a range of algorithmic options and datasets, we have also conducted a series of experiments using a system comprised of 24 algorithms and 9 datasets.

Title: Bregman Graph Neural Network. (arXiv:2309.06645v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06645
• Code URL: https://github.com/jiayuzhai1207/bregmangnn
• Copy Paste: [[2309.06645]] Bregman Graph Neural Network(http://arxiv.org/abs/2309.06645)
• Summary:

  Numerous recent research on graph neural networks (GNNs) has focused on formulating GNN architectures as an optimization problem with the smoothness assumption. However, in node classification tasks, the smoothing effect induced by GNNs tends to assimilate representations and over-homogenize labels of connected nodes, leading to adverse effects such as over-smoothing and misclassification. In this paper, we propose a novel bilevel optimization framework for GNNs inspired by the notion of Bregman distance. We demonstrate that the GNN layer proposed accordingly can effectively mitigate the over-smoothing issue by introducing a mechanism reminiscent of the "skip connection". We validate our theoretical results through comprehensive empirical studies in which Bregman-enhanced GNNs outperform their original counterparts in both homophilic and heterophilic graphs. Furthermore, our experiments also show that Bregman GNNs can produce more robust learning accuracy even when the number of layers is high, suggesting the effectiveness of the proposed method in alleviating the over-smoothing issue.

Title: Bias Amplification Enhances Minority Group Performance. (arXiv:2309.06717v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06717
• Code URL: null
• Copy Paste: [[2309.06717]] Bias Amplification Enhances Minority Group Performance(http://arxiv.org/abs/2309.06717)
• Summary:

  Neural networks produced by standard training are known to suffer from poor accuracy on rare subgroups despite achieving high accuracy on average, due to the correlations between certain spurious features and labels. Previous approaches based on worst-group loss minimization (e.g. Group-DRO) are effective in improving worse-group accuracy but require expensive group annotations for all the training samples. In this paper, we focus on the more challenging and realistic setting where group annotations are only available on a small validation set or are not available at all. We propose BAM, a novel two-stage training algorithm: in the first stage, the model is trained using a bias amplification scheme via introducing a learnable auxiliary variable for each training sample; in the second stage, we upweight the samples that the bias-amplified model misclassifies, and then continue training the same model on the reweighted dataset. Empirically, BAM achieves competitive performance compared with existing methods evaluated on spurious correlation benchmarks in computer vision and natural language processing. Moreover, we find a simple stopping criterion based on minimum class accuracy difference that can remove the need for group annotations, with little or no loss in worst-group accuracy. We perform extensive analyses and ablations to verify the effectiveness and robustness of our algorithm in varying class and group imbalance ratios.

Title: Safe Reinforcement Learning with Dual Robustness. (arXiv:2309.06835v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06835
• Code URL: null
• Copy Paste: [[2309.06835]] Safe Reinforcement Learning with Dual Robustness(http://arxiv.org/abs/2309.06835)
• Summary:

  Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or compromise safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. Optimality is only valid inside a feasible region, while identification of maximal feasible region must rely on learning the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including problem formulation, iteration scheme, convergence analysis and practical algorithm design. This unification is built upon constrained two-player zero-sum Markov games. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. The convergence of this iteration scheme is proved. Furthermore, we design a deep RL algorithm for practical implementation, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines significantly.

Title: Supervised Machine Learning and Physics based Machine Learning approach for prediction of peak temperature distribution in Additive Friction Stir Deposition of Aluminium Alloy. (arXiv:2309.06838v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06838
• Code URL: null
• Copy Paste: [[2309.06838]] Supervised Machine Learning and Physics based Machine Learning approach for prediction of peak temperature distribution in Additive Friction Stir Deposition of Aluminium Alloy(http://arxiv.org/abs/2309.06838)
• Summary:

  Additive friction stir deposition (AFSD) is a novel solid-state additive manufacturing technique that circumvents issues of porosity, cracking, and properties anisotropy that plague traditional powder bed fusion and directed energy deposition approaches. However, correlations between process parameters, thermal profiles, and resulting microstructure in AFSD remain poorly understood. This hinders process optimization for properties. This work employs a cutting-edge framework combining supervised machine learning (SML) and physics-informed neural networks (PINNs) to predict peak temperature distribution in AFSD from process parameters. Eight regression algorithms were implemented for SML modeling, while four PINNs leveraged governing equations for transport, wave propagation, heat transfer, and quantum mechanics. Across multiple statistical measures, ensemble techniques like gradient boosting proved superior for SML, with lowest MSE of 165.78. The integrated ML approach was also applied to classify deposition quality from process factors, with logistic regression delivering robust accuracy. By fusing data-driven learning and fundamental physics, this dual methodology provides comprehensive insights into tailoring microstructure through thermal management in AFSD. The work demonstrates the power of bridging statistical and physics-based modeling for elucidating AM process-property relationships.

Title: The Boundaries of Verifiable Accuracy, Robustness, and Generalisation in Deep Learning. (arXiv:2309.07072v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.07072
• Code URL: null
• Copy Paste: [[2309.07072]] The Boundaries of Verifiable Accuracy, Robustness, and Generalisation in Deep Learning(http://arxiv.org/abs/2309.07072)
• Summary:

  In this work, we assess the theoretical limitations of determining guaranteed stability and accuracy of neural networks in classification tasks. We consider classical distribution-agnostic framework and algorithms minimising empirical risks and potentially subjected to some weights regularisation. We show that there is a large family of tasks for which computing and verifying ideal stable and accurate neural networks in the above settings is extremely challenging, if at all possible, even when such ideal solutions exist within the given class of neural architectures.

biometric

Title: Ethnicity and Biometric Uniqueness: Iris Pattern Individuality in a West African Database. (arXiv:2309.06521v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06521
• Code URL: null
• Copy Paste: [[2309.06521]] Ethnicity and Biometric Uniqueness: Iris Pattern Individuality in a West African Database(http://arxiv.org/abs/2309.06521)
• Summary:

  We conducted more than 1.3 million comparisons of iris patterns encoded from images collected at two Nigerian universities, which constitute the newly available African Human Iris (AFHIRIS) database. The purpose was to discover whether ethnic differences in iris structure and appearance such as the textural feature size, as contrasted with an all-Chinese image database or an American database in which only 1.53% were of African-American heritage, made a material difference for iris discrimination. We measured a reduction in entropy for the AFHIRIS database due to the coarser iris features created by the thick anterior layer of melanocytes, and we found stochastic parameters that accurately model the relevant empirical distributions. Quantile-Quantile analysis revealed that a very small change in operational decision thresholds for the African database would compensate for the reduced entropy and generate the same performance in terms of resistance to False Matches. We conclude that despite demographic difference, individuality can be robustly discerned by comparison of iris patterns in this West African population.

steal

extraction

Title: Video Infringement Detection via Feature Disentanglement and Mutual Information Maximization. (arXiv:2309.06877v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06877
• Code URL: https://github.com/yyyooooo/dmi
• Copy Paste: [[2309.06877]] Video Infringement Detection via Feature Disentanglement and Mutual Information Maximization(http://arxiv.org/abs/2309.06877)
• Summary:

  The self-media era provides us tremendous high quality videos. Unfortunately, frequent video copyright infringements are now seriously damaging the interests and enthusiasm of video creators. Identifying infringing videos is therefore a compelling task. Current state-of-the-art methods tend to simply feed high-dimensional mixed video features into deep neural networks and count on the networks to extract useful representations. Despite its simplicity, this paradigm heavily relies on the original entangled features and lacks constraints guaranteeing that useful task-relevant semantics are extracted from the features.

  In this paper, we seek to tackle the above challenges from two aspects: (1) We propose to disentangle an original high-dimensional feature into multiple sub-features, explicitly disentangling the feature into exclusive lower-dimensional components. We expect the sub-features to encode non-overlapping semantics of the original feature and remove redundant information.

  (2) On top of the disentangled sub-features, we further learn an auxiliary feature to enhance the sub-features. We theoretically analyzed the mutual information between the label and the disentangled features, arriving at a loss that maximizes the extraction of task-relevant information from the original feature.

  Extensive experiments on two large-scale benchmark datasets (i.e., SVD and VCSL) demonstrate that our method achieves 90.1% TOP-100 mAP on the large-scale SVD dataset and also sets the new state-of-the-art on the VCSL benchmark dataset. Our code and model have been released at https://github.com/yyyooooo/DMI/, hoping to contribute to the community.

Title: CCSPNet-Joint: Efficient Joint Training Method for Traffic Sihn Detection Under Extreme Conditions. (arXiv:2309.06902v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06902
• Code URL: https://github.com/haoqinhong/ccspnet-joint
• Copy Paste: [[2309.06902]] CCSPNet-Joint: Efficient Joint Training Method for Traffic Sihn Detection Under Extreme Conditions(http://arxiv.org/abs/2309.06902)
• Summary:

  Traffic sign detection is an important research direction in intelligent driving. Unfortunately, existing methods often overlook extreme conditions such as fog, rain, and motion blur. Moreover, the end-to-end training strategy for image denoising and object detection models fails to utilize inter-model information effectively. To address these issues, we propose CCSPNet, an efficient feature extraction module based on Transformers and CNNs, which effectively leverages contextual information, achieves faster inference speed and provides stronger feature enhancement capabilities. Furthermore, we establish the correlation between object detection and image denoising tasks and propose a joint training model, CCSPNet-Joint, to improve data efficiency and generalization. Finally, to validate our approach, we create the CCTSDB-AUG dataset for traffic sign detection in extreme scenarios. Extensive experiments have shown that CCSPNet achieves state-of-the-art performance in traffic sign detection under extreme conditions. Compared to end-to-end methods, CCSPNet-Joint achieves a 5.32% improvement in precision and an 18.09% improvement in mAP@.5.

Title: Unsupervised Bias Detection in College Student Newspapers. (arXiv:2309.06557v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06557
• Code URL: null
• Copy Paste: [[2309.06557]] Unsupervised Bias Detection in College Student Newspapers(http://arxiv.org/abs/2309.06557)
• Summary:

  This paper presents a pipeline with minimal human influence for scraping and detecting bias on college newspaper archives. This paper introduces a framework for scraping complex archive sites that automated tools fail to grab data from, and subsequently generates a dataset of 14 student papers with 23,154 entries. This data can also then be queried by keyword to calculate bias by comparing the sentiment of a large language model summary to the original article. The advantages of this approach are that it is less comparative than reconstruction bias and requires less labelled data than generating keyword sentiment. Results are calculated on politically charged words as well as control words to show how conclusions can be drawn. The complete method facilitates the extraction of nuanced insights with minimal assumptions and categorizations, paving the way for a more objective understanding of bias within student newspaper sources.

Title: Comparative Analysis of Contextual Relation Extraction based on Deep Learning Models. (arXiv:2309.06814v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06814
• Code URL: null
• Copy Paste: [[2309.06814]] Comparative Analysis of Contextual Relation Extraction based on Deep Learning Models(http://arxiv.org/abs/2309.06814)
• Summary:

  Contextual Relation Extraction (CRE) is mainly used for constructing a knowledge graph with a help of ontology. It performs various tasks such as semantic search, query answering, and textual entailment. Relation extraction identifies the entities from raw texts and the relations among them. An efficient and accurate CRE system is essential for creating domain knowledge in the biomedical industry. Existing Machine Learning and Natural Language Processing (NLP) techniques are not suitable to predict complex relations from sentences that consist of more than two relations and unspecified entities efficiently. In this work, deep learning techniques have been used to identify the appropriate semantic relation based on the context from multiple sentences. Even though various machine learning models have been used for relation extraction, they provide better results only for binary relations, i.e., relations occurred exactly between the two entities in a sentence. Machine learning models are not suited for complex sentences that consist of the words that have various meanings. To address these issues, hybrid deep learning models have been used to extract the relations from complex sentence effectively. This paper explores the analysis of various deep learning models that are used for relation extraction.

Title: R\'esum\'e Parsing as Hierarchical Sequence Labeling: An Empirical Study. (arXiv:2309.07015v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.07015
• Code URL: https://github.com/federetyk/resume-parsing
• Copy Paste: [[2309.07015]] R\'esum\'e Parsing as Hierarchical Sequence Labeling: An Empirical Study(http://arxiv.org/abs/2309.07015)
• Summary:

  Extracting information from r\'esum\'es is typically formulated as a two-stage problem, where the document is first segmented into sections and then each section is processed individually to extract the target entities. Instead, we cast the whole problem as sequence labeling in two levels -- lines and tokens -- and study model architectures for solving both tasks simultaneously. We build high-quality r\'esum\'e parsing corpora in English, French, Chinese, Spanish, German, Portuguese, and Swedish. Based on these corpora, we present experimental results that demonstrate the effectiveness of the proposed models for the information extraction task, outperforming approaches introduced in previous work. We conduct an ablation study of the proposed architectures. We also analyze both model performance and resource efficiency, and describe the trade-offs for model deployment in the context of a production environment.

membership infer

federate

Title: Mitigating Group Bias in Federated Learning for Heterogeneous Devices. (arXiv:2309.07085v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.07085
• Code URL: null
• Copy Paste: [[2309.07085]] Mitigating Group Bias in Federated Learning for Heterogeneous Devices(http://arxiv.org/abs/2309.07085)
• Summary:

  Federated Learning is emerging as a privacy-preserving model training approach in distributed edge applications. As such, most edge deployments are heterogeneous in nature i.e., their sensing capabilities and environments vary across deployments. This edge heterogeneity violates the independence and identical distribution (IID) property of local data across clients and produces biased global models i.e. models that contribute to unfair decision-making and discrimination against a particular community or a group. Existing bias mitigation techniques only focus on bias generated from label heterogeneity in non-IID data without accounting for domain variations due to feature heterogeneity and do not address global group-fairness property.

  Our work proposes a group-fair FL framework that minimizes group-bias while preserving privacy and without resource utilization overhead. Our main idea is to leverage average conditional probabilities to compute a cross-domain group \textit{importance weights} derived from heterogeneous training data to optimize the performance of the worst-performing group using a modified multiplicative weights update method. Additionally, we propose regularization techniques to minimize the difference between the worst and best-performing groups while making sure through our thresholding mechanism to strike a balance between bias reduction and group performance degradation. Our evaluation of human emotion recognition and image classification benchmarks assesses the fair decision-making of our framework in real-world heterogeneous settings.

Title: Federated PAC-Bayesian Learning on Non-IID data. (arXiv:2309.06683v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06683
• Code URL: null
• Copy Paste: [[2309.06683]] Federated PAC-Bayesian Learning on Non-IID data(http://arxiv.org/abs/2309.06683)
• Summary:

  Existing research has either adapted the Probably Approximately Correct (PAC) Bayesian framework for federated learning (FL) or used information-theoretic PAC-Bayesian bounds while introducing their theorems, but few considering the non-IID challenges in FL. Our work presents the first non-vacuous federated PAC-Bayesian bound tailored for non-IID local data. This bound assumes unique prior knowledge for each client and variable aggregation weights. We also introduce an objective function and an innovative Gibbs-based algorithm for the optimization of the derived bound. The results are validated on real-world datasets.

Title: Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization. (arXiv:2309.06692v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06692
• Code URL: null
• Copy Paste: [[2309.06692]] Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization(http://arxiv.org/abs/2309.06692)
• Summary:

  Federated learning (FL) is a privacy-preserving paradigm for collaboratively training a global model from decentralized clients. However, the performance of FL is hindered by non-independent and identically distributed (non-IID) data and device heterogeneity. In this work, we revisit this key challenge through the lens of gradient conflicts on the server side. Specifically, we first investigate the gradient conflict phenomenon among multiple clients and reveal that stronger heterogeneity leads to more severe gradient conflicts. To tackle this issue, we propose FedGH, a simple yet effective method that mitigates local drifts through Gradient Harmonization. This technique projects one gradient vector onto the orthogonal plane of the other within conflicting client pairs. Extensive experiments demonstrate that FedGH consistently enhances multiple state-of-the-art FL baselines across diverse benchmarks and non-IID scenarios. Notably, FedGH yields more significant improvements in scenarios with stronger heterogeneity. As a plug-and-play module, FedGH can be seamlessly integrated into any FL framework without requiring hyperparameter tuning.

Title: FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization. (arXiv:2309.06805v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06805
• Code URL: https://github.com/ericloong/feddip
• Copy Paste: [[2309.06805]] FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization(http://arxiv.org/abs/2309.06805)
• Summary:

  Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve \textit{extreme} sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip.

fair

Title: FAIR: Frequency-aware Image Restoration for Industrial Visual Anomaly Detection. (arXiv:2309.07068v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.07068
• Code URL: null
• Copy Paste: [[2309.07068]] FAIR: Frequency-aware Image Restoration for Industrial Visual Anomaly Detection(http://arxiv.org/abs/2309.07068)
• Summary:

  Image reconstruction-based anomaly detection models are widely explored in industrial visual inspection. However, existing models usually suffer from the trade-off between normal reconstruction fidelity and abnormal reconstruction distinguishability, which damages the performance. In this paper, we find that the above trade-off can be better mitigated by leveraging the distinct frequency biases between normal and abnormal reconstruction errors. To this end, we propose Frequency-aware Image Restoration (FAIR), a novel self-supervised image restoration task that restores images from their high-frequency components. It enables precise reconstruction of normal patterns while mitigating unfavorable generalization to anomalies. Using only a simple vanilla UNet, FAIR achieves state-of-the-art performance with higher efficiency on various defect detection datasets. Code: https://github.com/liutongkun/FAIR.

Title: Towards the TopMost: A Topic Modeling System Toolkit. (arXiv:2309.06908v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06908
• Code URL: https://github.com/bobxwu/topmost
• Copy Paste: [[2309.06908]] Towards the TopMost: A Topic Modeling System Toolkit(http://arxiv.org/abs/2309.06908)
• Summary:

  Topic models have been proposed for decades with various applications and recently refreshed by the neural variational inference. However, these topic models adopt totally distinct dataset, implementation, and evaluation settings, which hinders their quick utilization and fair comparisons. This greatly hinders the research progress of topic models. To address these issues, in this paper we propose a Topic Modeling System Toolkit (TopMost). Compared to existing toolkits, TopMost stands out by covering a wider range of topic modeling scenarios including complete lifecycles with dataset pre-processing, model training, testing, and evaluations. The highly cohesive and decoupled modular design of TopMost enables quick utilization, fair comparisons, and flexible extensions of different topic models. This can facilitate the research and applications of topic models. Our code, tutorials, and documentation are available at https://github.com/bobxwu/topmost.

interpretability

Title: Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning. (arXiv:2309.06597v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06597
• Code URL: null
• Copy Paste: [[2309.06597]] Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning(http://arxiv.org/abs/2309.06597)
• Summary:

  The widespread adoption of commercial autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) may largely depend on their acceptance by society, for which their perceived trustworthiness and interpretability to riders are crucial. In general, this task is challenging because modern autonomous systems software relies heavily on black-box artificial intelligence models. Towards this goal, this paper introduces a novel dataset, Rank2Tell, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance. Using various close and open-ended visual question answering, the dataset provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios. The dense annotations and unique attributes of the dataset make it a valuable resource for researchers working on visual scene understanding and related fields. Further, we introduce a joint model for joint importance level ranking and natural language captions generation to benchmark our dataset and demonstrate performance with quantitative evaluations.

Title: MCNS: Mining Causal Natural Structures Inside Time Series via A Novel Internal Causality Scheme. (arXiv:2309.06739v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06739
• Code URL: null
• Copy Paste: [[2309.06739]] MCNS: Mining Causal Natural Structures Inside Time Series via A Novel Internal Causality Scheme(http://arxiv.org/abs/2309.06739)
• Summary:

  Causal inference permits us to discover covert relationships of various variables in time series. However, in most existing works, the variables mentioned above are the dimensions. The causality between dimensions could be cursory, which hinders the comprehension of the internal relationship and the benefit of the causal graph to the neural networks (NNs). In this paper, we find that causality exists not only outside but also inside the time series because it reflects a succession of events in the real world. It inspires us to seek the relationship between internal subsequences. However, the challenges are the hardship of discovering causality from subsequences and utilizing the causal natural structures to improve NNs. To address these challenges, we propose a novel framework called Mining Causal Natural Structure (MCNS), which is automatic and domain-agnostic and helps to find the causal natural structures inside time series via the internal causality scheme. We evaluate the MCNS framework and impregnation NN with MCNS on time series classification tasks. Experimental results illustrate that our impregnation, by refining attention, shape selection classification, and pruning datasets, drives NN, even the data itself preferable accuracy and interpretability. Besides, MCNS provides an in-depth, solid summary of the time series and datasets.

explainability

watermark

diffusion

Title: DreamStyler: Paint by Style Inversion with Text-to-Image Diffusion Models. (arXiv:2309.06933v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06933
• Code URL: null
• Copy Paste: [[2309.06933]] DreamStyler: Paint by Style Inversion with Text-to-Image Diffusion Models(http://arxiv.org/abs/2309.06933)
• Summary:

  Recent progresses in large-scale text-to-image models have yielded remarkable accomplishments, finding various applications in art domain. However, expressing unique characteristics of an artwork (e.g. brushwork, colortone, or composition) with text prompts alone may encounter limitations due to the inherent constraints of verbal description. To this end, we introduce DreamStyler, a novel framework designed for artistic image synthesis, proficient in both text-to-image synthesis and style transfer. DreamStyler optimizes a multi-stage textual embedding with a context-aware text prompt, resulting in prominent image quality. In addition, with content and style guidance, DreamStyler exhibits flexibility to accommodate a range of style references. Experimental results demonstrate its superior performance across multiple scenarios, suggesting its promising potential in artistic product creation.

Title: Reasoning with Latent Diffusion in Offline Reinforcement Learning. (arXiv:2309.06599v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06599
• Code URL: https://github.com/ldcq/ldcq
• Copy Paste: [[2309.06599]] Reasoning with Latent Diffusion in Offline Reinforcement Learning(http://arxiv.org/abs/2309.06599)
• Summary:

  Offline reinforcement learning (RL) holds promise as a means to learn high-reward policies from a static dataset, without the need for further environment interactions. However, a key challenge in offline RL lies in effectively stitching portions of suboptimal trajectories from the static dataset while avoiding extrapolation errors arising due to a lack of support in the dataset. Existing approaches use conservative methods that are tricky to tune and struggle with multi-modal data (as we show) or rely on noisy Monte Carlo return-to-go samples for reward conditioning. In this work, we propose a novel approach that leverages the expressiveness of latent diffusion to model in-support trajectory sequences as compressed latent skills. This facilitates learning a Q-function while avoiding extrapolation error via batch-constraining. The latent space is also expressive and gracefully copes with multi-modal data. We show that the learned temporally-abstract latent space encodes richer task-specific information for offline RL tasks as compared to raw state-actions. This improves credit assignment and facilitates faster reward propagation during Q-learning. Our method demonstrates state-of-the-art performance on the D4RL benchmarks, particularly excelling in long-horizon, sparse-reward tasks.

noise learning

data-free

transformer

Title: DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attention. (arXiv:2309.06511v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06511
• Code URL: null
• Copy Paste: [[2309.06511]] DF-TransFusion: Multimodal Deepfake Detection via Lip-Audio Cross-Attention and Facial Self-Attention(http://arxiv.org/abs/2309.06511)
• Summary:

  With the rise in manipulated media, deepfake detection has become an imperative task for preserving the authenticity of digital content. In this paper, we present a novel multi-modal audio-video framework designed to concurrently process audio and video inputs for deepfake detection tasks. Our model capitalizes on lip synchronization with input audio through a cross-attention mechanism while extracting visual cues via a fine-tuned VGG-16 network. Subsequently, a transformer encoder network is employed to perform facial self-attention. We conduct multiple ablation studies highlighting different strengths of our approach. Our multi-modal methodology outperforms state-of-the-art multi-modal deepfake detection techniques in terms of F-1 and per-video AUC scores.

Title: ShaDocFormer: A Shadow-attentive Threshold Detector with Cascaded Fusion Refiner for document shadow removal' to the ICASSP 2024 online submission system. (arXiv:2309.06670v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06670
• Code URL: null
• Copy Paste: [[2309.06670]] ShaDocFormer: A Shadow-attentive Threshold Detector with Cascaded Fusion Refiner for document shadow removal' to the ICASSP 2024 online submission system(http://arxiv.org/abs/2309.06670)
• Summary:

  Document shadow is a common issue that arise when capturing documents using mobile devices, which significantly impacts the readability. Current methods encounter various challenges including inaccurate detection of shadow masks and estimation of illumination. In this paper, we propose ShaDocFormer, a Transformer-based architecture that integrates traditional methodologies and deep learning techniques to tackle the problem of document shadow removal. The ShaDocFormer architecture comprises two components: the Shadow-attentive Threshold Detector (STD) and the Cascaded Fusion Refiner (CFR). The STD module employs a traditional thresholding technique and leverages the attention mechanism of the Transformer to gather global information, thereby enabling precise detection of shadow masks. The cascaded and aggregative structure of the CFR module facilitates a coarse-to-fine restoration process for the entire image. As a result, ShaDocFormer excels in accurately detecting and capturing variations in both shadow and illumination, thereby enabling effective removal of shadows. Extensive experiments demonstrate that ShaDocFormer outperforms current state-of-the-art methods in both qualitative and quantitative measurements.

Title: Transparent Object Tracking with Enhanced Fusion Module. (arXiv:2309.06701v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06701
• Code URL: https://github.com/kalyan0510/totem
• Copy Paste: [[2309.06701]] Transparent Object Tracking with Enhanced Fusion Module(http://arxiv.org/abs/2309.06701)
• Summary:

  Accurate tracking of transparent objects, such as glasses, plays a critical role in many robotic tasks such as robot-assisted living. Due to the adaptive and often reflective texture of such objects, traditional tracking algorithms that rely on general-purpose learned features suffer from reduced performance. Recent research has proposed to instill transparency awareness into existing general object trackers by fusing purpose-built features. However, with the existing fusion techniques, the addition of new features causes a change in the latent space making it impossible to incorporate transparency awareness on trackers with fixed latent spaces. For example, many of the current days transformer-based trackers are fully pre-trained and are sensitive to any latent space perturbations. In this paper, we present a new feature fusion technique that integrates transparency information into a fixed feature space, enabling its use in a broader range of trackers. Our proposed fusion module, composed of a transformer encoder and an MLP module, leverages key query-based transformations to embed the transparency information into the tracking pipeline. We also present a new two-step training strategy for our fusion module to effectively merge transparency features. We propose a new tracker architecture that uses our fusion techniques to achieve superior results for transparent object tracking. Our proposed method achieves competitive results with state-of-the-art trackers on TOTB, which is the largest transparent object tracking benchmark recently released. Our results and the implementation of code will be made publicly available at https://github.com/kalyan0510/TOTEM.

Title: Dynamic Spectrum Mixer for Visual Recognition. (arXiv:2309.06721v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06721
• Code URL: null
• Copy Paste: [[2309.06721]] Dynamic Spectrum Mixer for Visual Recognition(http://arxiv.org/abs/2309.06721)
• Summary:

  Recently, MLP-based vision backbones have achieved promising performance in several visual recognition tasks. However, the existing MLP-based methods directly aggregate tokens with static weights, leaving the adaptability to different images untouched. Moreover, Recent research demonstrates that MLP-Transformer is great at creating long-range dependencies but ineffective at catching high frequencies that primarily transmit local information, which prevents it from applying to the downstream dense prediction tasks, such as semantic segmentation. To address these challenges, we propose a content-adaptive yet computationally efficient structure, dubbed Dynamic Spectrum Mixer (DSM). The DSM represents token interactions in the frequency domain by employing the Discrete Cosine Transform, which can learn long-term spatial dependencies with log-linear complexity. Furthermore, a dynamic spectrum weight generation layer is proposed as the spectrum bands selector, which could emphasize the informative frequency bands while diminishing others. To this end, the technique can efficiently learn detailed features from visual input that contains both high- and low-frequency information. Extensive experiments show that DSM is a powerful and adaptable backbone for a range of visual recognition tasks. Particularly, DSM outperforms previous transformer-based and MLP-based models, on image classification, object detection, and semantic segmentation tasks, such as 83.8 \% top-1 accuracy on ImageNet, and 49.9 \% mIoU on ADE20K.

Title: Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?. (arXiv:2309.06891v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06891
• Code URL: null
• Copy Paste: [[2309.06891]] Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?(http://arxiv.org/abs/2309.06891)
• Summary:

  Convolutional networks and vision transformers have different forms of pairwise interactions, pooling across layers and pooling at the end of the network. Does the latter really need to be different? As a by-product of pooling, vision transformers provide spatial attention for free, but this is most often of low quality unless self-supervised, which is not well studied. Is supervision really the problem?

  In this work, we develop a generic pooling framework and then we formulate a number of existing methods as instantiations. By discussing the properties of each group of methods, we derive SimPool, a simple attention-based pooling mechanism as a replacement of the default one for both convolutional and transformer encoders. We find that, whether supervised or self-supervised, this improves performance on pre-training and downstream tasks and provides attention maps delineating object boundaries in all cases. One could thus call SimPool universal. To our knowledge, we are the first to obtain attention maps in supervised transformers of at least as good quality as self-supervised, without explicit losses or modifying the architecture. Code at: https://github.com/billpsomas/simpool.

Title: DEFormer: DCT-driven Enhancement Transformer for Low-light Image and Dark Vision. (arXiv:2309.06941v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06941
• Code URL: null
• Copy Paste: [[2309.06941]] DEFormer: DCT-driven Enhancement Transformer for Low-light Image and Dark Vision(http://arxiv.org/abs/2309.06941)
• Summary:

  The goal of low-light image enhancement is to restore the color and details of the image and is of great significance for high-level visual tasks in autonomous driving. However, it is difficult to restore the lost details in the dark area by relying only on the RGB domain. In this paper we introduce frequency as a new clue into the network and propose a novel DCT-driven enhancement transformer (DEFormer). First, we propose a learnable frequency branch (LFB) for frequency enhancement contains DCT processing and curvature-based frequency enhancement (CFE). CFE calculates the curvature of each channel to represent the detail richness of different frequency bands, then we divides the frequency features, which focuses on frequency bands with richer textures. In addition, we propose a cross domain fusion (CDF) for reducing the differences between the RGB domain and the frequency domain. We also adopt DEFormer as a preprocessing in dark detection, DEFormer effectively improves the performance of the detector, bringing 2.1% and 3.4% improvement in ExDark and DARK FACE datasets on mAP respectively.

Title: Neural network-based coronary dominance classification of RCA angiograms. (arXiv:2309.06958v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06958
• Code URL: null
• Copy Paste: [[2309.06958]] Neural network-based coronary dominance classification of RCA angiograms(http://arxiv.org/abs/2309.06958)
• Summary:

  Background. Cardiac dominance classification is essential for SYNTAX score estimation, which is a tool used to determine the complexity of coronary artery disease and guide patient selection toward optimal revascularization strategy. Objectives. Cardiac dominance classification algorithm based on the analysis of right coronary artery (RCA) angiograms using neural network Method. We employed convolutional neural network ConvNext and Swin transformer for 2D image (frames) classification, along with a majority vote for cardio angiographic view classification. An auxiliary network was also used to detect irrelevant images which were then excluded from the data set. Our data set consisted of 828 angiographic studies, 192 of them being patients with left dominance. Results. 5-fold cross validation gave the following dominance classification metrics (p=95%): macro recall=93.1%, accuracy=93.5%, macro F1=89.2%. The most common case in which the model regularly failed was RCA occlusion, as it requires utilization of LCA information. Another cause for false prediction is a small diameter combined with poor quality cardio angiographic view. In such cases, cardiac dominance classification can be complex and may require discussion among specialists to reach an accurate conclusion. Conclusion. The use of machine learning approaches to classify cardiac dominance based on RCA alone has been shown to be successful with satisfactory accuracy. However, for higher accuracy, it is necessary to utilize LCA information in the case of an occluded RCA and detect cases where there is high uncertainty.

Title: Aggregating Long-term Sharp Features via Hybrid Transformers for Video Deblurring. (arXiv:2309.07054v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.07054
• Code URL: https://github.com/shangwei5/stgtn
• Copy Paste: [[2309.07054]] Aggregating Long-term Sharp Features via Hybrid Transformers for Video Deblurring(http://arxiv.org/abs/2309.07054)
• Summary:

  Video deblurring methods, aiming at recovering consecutive sharp frames from a given blurry video, usually assume that the input video suffers from consecutively blurry frames. However, in real-world blurry videos taken by modern imaging devices, sharp frames usually appear in the given video, thus making temporal long-term sharp features available for facilitating the restoration of a blurry frame. In this work, we propose a video deblurring method that leverages both neighboring frames and present sharp frames using hybrid Transformers for feature aggregation. Specifically, we first train a blur-aware detector to distinguish between sharp and blurry frames. Then, a window-based local Transformer is employed for exploiting features from neighboring frames, where cross attention is beneficial for aggregating features from neighboring frames without explicit spatial alignment. To aggregate long-term sharp features from detected sharp frames, we utilize a global Transformer with multi-scale matching capability. Moreover, our method can easily be extended to event-driven video deblurring by incorporating an event fusion module into the global Transformer. Extensive experiments on benchmark datasets demonstrate that our proposed method outperforms state-of-the-art video deblurring methods as well as event-driven video deblurring methods in terms of quantitative metrics and visual quality. The source code and trained models are available at https://github.com/shangwei5/STGTN.

Title: Gpachov at CheckThat! 2023: A Diverse Multi-Approach Ensemble for Subjectivity Detection in News Articles. (arXiv:2309.06844v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06844
• Code URL: null
• Copy Paste: [[2309.06844]] Gpachov at CheckThat! 2023: A Diverse Multi-Approach Ensemble for Subjectivity Detection in News Articles(http://arxiv.org/abs/2309.06844)
• Summary:

  The wide-spread use of social networks has given rise to subjective, misleading, and even false information on the Internet. Thus, subjectivity detection can play an important role in ensuring the objectiveness and the quality of a piece of information. This paper presents the solution built by the Gpachov team for the CLEF-2023 CheckThat! lab Task~2 on subjectivity detection. Three different research directions are explored. The first one is based on fine-tuning a sentence embeddings encoder model and dimensionality reduction. The second one explores a sample-efficient few-shot learning model. The third one evaluates fine-tuning a multilingual transformer on an altered dataset, using data from multiple languages. Finally, the three approaches are combined in a simple majority voting ensemble, resulting in 0.77 macro F1 on the test set and achieving 2nd place on the English subtask.

Title: Native Language Identification with Big Bird Embeddings. (arXiv:2309.06923v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06923
• Code URL: https://github.com/sergeykramp/mthesis-bigbird-embeddings
• Copy Paste: [[2309.06923]] Native Language Identification with Big Bird Embeddings(http://arxiv.org/abs/2309.06923)
• Summary:

  Native Language Identification (NLI) intends to classify an author's native language based on their writing in another language. Historically, the task has heavily relied on time-consuming linguistic feature engineering, and transformer-based NLI models have thus far failed to offer effective, practical alternatives. The current work investigates if input size is a limiting factor, and shows that classifiers trained using Big Bird embeddings outperform linguistic feature engineering models by a large margin on the Reddit-L2 dataset. Additionally, we provide further insight into input length dependencies, show consistent out-of-sample performance, and qualitatively analyze the embedding space. Given the effectiveness and computational efficiency of this method, we believe it offers a promising avenue for future NLI work.

Title: Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers. (arXiv:2309.06526v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06526
• Code URL: https://github.com/ibm/dp-tabtransformer
• Copy Paste: [[2309.06526]] Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers(http://arxiv.org/abs/2309.06526)
• Summary:

  For machine learning with tabular data, Table Transformer (TabTransformer) is a state-of-the-art neural network model, while Differential Privacy (DP) is an essential component to ensure data privacy. In this paper, we explore the benefits of combining these two aspects together in the scenario of transfer learning -- differentially private pre-training and fine-tuning of TabTransformers with a variety of parameter-efficient fine-tuning (PEFT) methods, including Adapter, LoRA, and Prompt Tuning. Our extensive experiments on the ACSIncome dataset show that these PEFT methods outperform traditional approaches in terms of the accuracy of the downstream task and the number of trainable parameters, thus achieving an improved trade-off among parameter efficiency, privacy, and accuracy. Our code is available at github.com/IBM/DP-TabTransformer.

generative

Title: Integrating GAN and Texture Synthesis for Enhanced Road Damage Detection. (arXiv:2309.06747v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06747
• Code URL: null
• Copy Paste: [[2309.06747]] Integrating GAN and Texture Synthesis for Enhanced Road Damage Detection(http://arxiv.org/abs/2309.06747)
• Summary:

  In the domain of traffic safety and road maintenance, precise detection of road damage is crucial for ensuring safe driving and prolonging road durability. However, current methods often fall short due to limited data. Prior attempts have used Generative Adversarial Networks to generate damage with diverse shapes and manually integrate it into appropriate positions. However, the problem has not been well explored and is faced with two challenges. First, they only enrich the location and shape of damage while neglect the diversity of severity levels, and the realism still needs further improvement. Second, they require a significant amount of manual effort. To address these challenges, we propose an innovative approach. In addition to using GAN to generate damage with various shapes, we further employ texture synthesis techniques to extract road textures. These two elements are then mixed with different weights, allowing us to control the severity of the synthesized damage, which are then embedded back into the original images via Poisson blending. Our method ensures both richness of damage severity and a better alignment with the background. To save labor costs, we leverage structural similarity for automated sample selection during embedding. Each augmented data of an original image contains versions with varying severity levels. We implement a straightforward screening strategy to mitigate distribution drift. Experiments are conducted on a public road damage dataset. The proposed method not only eliminates the need for manual labor but also achieves remarkable enhancements, improving the mAP by 4.1% and the F1-score by 4.5%.

Title: Instance Adaptive Prototypical Contrastive Embedding for Generalized Zero Shot Learning. (arXiv:2309.06987v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06987
• Code URL: null
• Copy Paste: [[2309.06987]] Instance Adaptive Prototypical Contrastive Embedding for Generalized Zero Shot Learning(http://arxiv.org/abs/2309.06987)
• Summary:

  Generalized zero-shot learning(GZSL) aims to classify samples from seen and unseen labels, assuming unseen labels are not accessible during training. Recent advancements in GZSL have been expedited by incorporating contrastive-learning-based (instance-based) embedding in generative networks and leveraging the semantic relationship between data points. However, existing embedding architectures suffer from two limitations: (1) limited discriminability of synthetic features' embedding without considering fine-grained cluster structures; (2) inflexible optimization due to restricted scaling mechanisms on existing contrastive embedding networks, leading to overlapped representations in the embedding space. To enhance the quality of representations in the embedding space, as mentioned in (1), we propose a margin-based prototypical contrastive learning embedding network that reaps the benefits of prototype-data (cluster quality enhancement) and implicit data-data (fine-grained representations) interaction while providing substantial cluster supervision to the embedding network and the generator. To tackle (2), we propose an instance adaptive contrastive loss that leads to generalized representations for unseen labels with increased inter-class margin. Through comprehensive experimental evaluation, we show that our method can outperform the current state-of-the-art on three benchmark datasets. Our approach also consistently achieves the best unseen performance in the GZSL setting.

Title: Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity. (arXiv:2309.06541v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06541
• Code URL: null
• Copy Paste: [[2309.06541]] Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity(http://arxiv.org/abs/2309.06541)
• Summary:

  Amidst the sharp rise in the evaluation of large language models (LLMs) on various tasks, we find that semantic textual similarity (STS) has been under-explored. In this study, we show that STS can be cast as a text generation problem while maintaining strong performance on multiple STS benchmarks. Additionally, we show generative LLMs significantly outperform existing encoder-based STS models when characterizing the semantic similarity between two texts with complex semantic relationships dependent on world knowledge. We validate this claim by evaluating both generative LLMs and existing encoder-based STS models on three newly collected STS challenge sets which require world knowledge in the domains of Health, Politics, and Sports. All newly collected data is sourced from social media content posted after May 2023 to ensure the performance of closed-source models like ChatGPT cannot be credited to memorization. Our results show that, on average, generative LLMs outperform the best encoder-only baselines by an average of 22.3% on STS tasks requiring world knowledge. Our results suggest generative language models with STS-specific prompting strategies achieve state-of-the-art performance in complex, domain-specific STS tasks.

Title: Do Generative Large Language Models need billions of parameters?. (arXiv:2309.06589v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06589
• Code URL: null
• Copy Paste: [[2309.06589]] Do Generative Large Language Models need billions of parameters?(http://arxiv.org/abs/2309.06589)
• Summary:

  This paper presents novel systems and methodologies for the development of efficient large language models (LLMs). It explores the trade-offs between model size, performance, and computational resources, with the aim of maximizing the efficiency of these AI systems. The research explores novel methods that allow different parts of the model to share parameters, reducing the total number of unique parameters required. This approach ensures that the model remains compact without sacrificing its ability to learn and represent complex language structures. This study provides valuable insights and tools for creating more efficient and effective LLMs, contributing to a more sustainable and accessible future for AI language modeling.

Title: Continual Learning with Dirichlet Generative-based Rehearsal. (arXiv:2309.06917v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06917
• Code URL: null
• Copy Paste: [[2309.06917]] Continual Learning with Dirichlet Generative-based Rehearsal(http://arxiv.org/abs/2309.06917)
• Summary:

  Recent advancements in data-driven task-oriented dialogue systems (ToDs) struggle with incremental learning due to computational constraints and time-consuming issues. Continual Learning (CL) attempts to solve this by avoiding intensive pre-training, but it faces the problem of catastrophic forgetting (CF). While generative-based rehearsal CL methods have made significant strides, generating pseudo samples that accurately reflect the underlying task-specific distribution is still a challenge. In this paper, we present Dirichlet Continual Learning (DCL), a novel generative-based rehearsal strategy for CL. Unlike the traditionally used Gaussian latent variable in the Conditional Variational Autoencoder (CVAE), DCL leverages the flexibility and versatility of the Dirichlet distribution to model the latent prior variable. This enables it to efficiently capture sentence-level features of previous tasks and effectively guide the generation of pseudo samples. In addition, we introduce Jensen-Shannon Knowledge Distillation (JSKD), a robust logit-based knowledge distillation method that enhances knowledge transfer during pseudo sample generation. Our experiments confirm the efficacy of our approach in both intent detection and slot-filling tasks, outperforming state-of-the-art methods.

large language model

Title: Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics. (arXiv:2309.07120v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.07120
• Code URL: https://github.com/ucsc-vlaa/sight-beyond-text
• Copy Paste: [[2309.07120]] Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics(http://arxiv.org/abs/2309.07120)
• Summary:

  Multi-modal large language models (MLLMs) are trained based on large language models (LLM), with an enhanced capability to comprehend multi-modal inputs and generate textual responses. While they excel in multi-modal tasks, the pure NLP abilities of MLLMs are often underestimated and left untested. In this study, we get out of the box and unveil an intriguing characteristic of MLLMs -- our preliminary results suggest that visual instruction tuning, a prevailing strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly helps models attain both improved truthfulness and ethical alignment in the pure NLP context. For example, a visual-instruction-tuned LLaMA2 7B model surpasses the performance of the LLaMA2-chat 7B model, fine-tuned with over one million human annotations, on TruthfulQA-mc and Ethics benchmarks. Further analysis reveals that the improved alignment can be attributed to the superior instruction quality inherent to visual-text data. In releasing our code at github.com/UCSC-VLAA/Sight-Beyond-Text, we aspire to foster further exploration into the intrinsic value of visual-text synergies and, in a broader scope, multi-modal interactions in alignment research.

Title: Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model. (arXiv:2309.06453v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06453
• Code URL: null
• Copy Paste: [[2309.06453]] Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model(http://arxiv.org/abs/2309.06453)
• Summary:

  Sentence Representation Learning (SRL) is a fundamental task in Natural Language Processing (NLP), with Contrastive learning of Sentence Embeddings (CSE) as the mainstream technique due to its superior performance. An intriguing phenomenon in CSE is the significant performance gap between supervised and unsupervised methods, even when their sentence encoder and loss function are the same. Previous works attribute this performance gap to differences in two representation properties (alignment and uniformity). However, alignment and uniformity only measure the results, which means they cannot answer "What happens during the training process that leads to the performance gap?" and "How can the performance gap be narrowed?". In this paper, we conduct empirical experiments to answer these "What" and "How" questions. We first answer the "What" question by thoroughly comparing the behavior of supervised and unsupervised CSE during their respective training processes. From the comparison, We observe a significant difference in fitting difficulty. Thus, we introduce a metric, called Fitting Difficulty Increment (FDI), to measure the fitting difficulty gap between the evaluation dataset and the held-out training dataset, and use the metric to answer the "What" question. Then, based on the insights gained from the "What" question, we tackle the "How" question by increasing the fitting difficulty of the training dataset. We achieve this by leveraging the In-Context Learning (ICL) capability of the Large Language Model (LLM) to generate data that simulates complex patterns. By utilizing the hierarchical patterns in the LLM-generated data, we effectively narrow the gap between supervised and unsupervised CSE.

Title: Leveraging Large Language Models for Automated Dialogue Analysis. (arXiv:2309.06490v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06490
• Code URL: https://github.com/emorynlp/gpt-abceval
• Copy Paste: [[2309.06490]] Leveraging Large Language Models for Automated Dialogue Analysis(http://arxiv.org/abs/2309.06490)
• Summary:

  Developing high-performing dialogue systems benefits from the automatic identification of undesirable behaviors in system responses. However, detecting such behaviors remains challenging, as it draws on a breadth of general knowledge and understanding of conversational practices. Although recent research has focused on building specialized classifiers for detecting specific dialogue behaviors, the behavior coverage is still incomplete and there is a lack of testing on real-world human-bot interactions. This paper investigates the ability of a state-of-the-art large language model (LLM), ChatGPT-3.5, to perform dialogue behavior detection for nine categories in real human-bot dialogues. We aim to assess whether ChatGPT can match specialized models and approximate human performance, thereby reducing the cost of behavior detection tasks. Our findings reveal that neither specialized models nor ChatGPT have yet achieved satisfactory results for this task, falling short of human performance. Nevertheless, ChatGPT shows promising potential and often outperforms specialized detection models. We conclude with an in-depth examination of the prevalent shortcomings of ChatGPT, offering guidance for future research to enhance LLM capabilities.

Title: AGIBench: A Multi-granularity, Multimodal, Human-referenced, Auto-scoring Benchmark for Large Language Models. (arXiv:2309.06495v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06495
• Code URL: null
• Copy Paste: [[2309.06495]] AGIBench: A Multi-granularity, Multimodal, Human-referenced, Auto-scoring Benchmark for Large Language Models(http://arxiv.org/abs/2309.06495)
• Summary:

  Large language models (LLMs) like ChatGPT have revealed amazing intelligence. How to evaluate the question-solving abilities of LLMs and their degrees of intelligence is a hot-spot but challenging issue. First, the question-solving abilities are interlaced with different ability branches like understanding and massive knowledge categories like mathematics. Second, the inputs of questions are multimodal that may involve text and images. Third, the response format of LLMs is diverse and thus poses great challenges for result extraction and evaluation. In this paper, we propose AGIBench -- a multi-granularity, multimodal, human-referenced, and auto-scoring benchmarking methodology for LLMs. Instead of a collection of blended questions, AGIBench focuses on three typical ability branches and adopts a four-tuple <ability branch, knowledge, difficulty, modal> to label the attributes of each question. First, it supports multi-granularity benchmarking, e.g., per-question, per-ability branch, per-knowledge, per-modal, per-dataset, and per-difficulty level granularities. Second, it contains multimodal input, including text and images. Third, it classifies all the questions into five degrees of difficulty according to the average accuracy rate of abundant educated humans (human-referenced). Fourth, it adopts zero-shot learning to avoid introducing additional unpredictability and provides an auto-scoring method to extract and judge the result. Finally, it defines multi-dimensional metrics, including accuracy under the average, worst, best, and majority voting cases, and repeatability. AGIBench is publically available from \url{https://www.benchcouncil.org/agibench}.

Title: Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets. (arXiv:2309.06503v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06503
• Code URL: null
• Copy Paste: [[2309.06503]] Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets(http://arxiv.org/abs/2309.06503)
• Summary:

  The COVID-19 pandemic has presented significant challenges to the healthcare industry and society as a whole. With the rapid development of COVID-19 vaccines, social media platforms have become a popular medium for discussions on vaccine-related topics. Identifying vaccine-related tweets and analyzing them can provide valuable insights for public health research-ers and policymakers. However, manual annotation of a large number of tweets is time-consuming and expensive. In this study, we evaluate the usage of Large Language Models, in this case GPT-4 (March 23 version), and weak supervision, to identify COVID-19 vaccine-related tweets, with the purpose of comparing performance against human annotators. We leveraged a manu-ally curated gold-standard dataset and used GPT-4 to provide labels without any additional fine-tuning or instructing, in a single-shot mode (no additional prompting).

Title: Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences. (arXiv:2309.06578v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06578
• Code URL: null
• Copy Paste: [[2309.06578]] Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences(http://arxiv.org/abs/2309.06578)
• Summary:

  Hypothesis formulation and testing are central to empirical research. A strong hypothesis is a best guess based on existing evidence and informed by a comprehensive view of relevant literature. However, with exponential increase in the number of scientific articles published annually, manual aggregation and synthesis of evidence related to a given hypothesis is a challenge. Our work explores the ability of current large language models (LLMs) to discern evidence in support or refute of specific hypotheses based on the text of scientific abstracts. We share a novel dataset for the task of scientific hypothesis evidencing using community-driven annotations of studies in the social sciences. We compare the performance of LLMs to several state-of-the-art benchmarks and highlight opportunities for future research in this area. The dataset is available at https://github.com/Sai90000/ScientificHypothesisEvidencing.git

Title: Simultaneous Machine Translation with Large Language Models. (arXiv:2309.06706v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06706
• Code URL: null
• Copy Paste: [[2309.06706]] Simultaneous Machine Translation with Large Language Models(http://arxiv.org/abs/2309.06706)
• Summary:

  Large language models (LLM) have demonstrated their abilities to solve various natural language processing tasks through dialogue-based interactions. For instance, research indicates that LLMs can achieve competitive performance in offline machine translation tasks for high-resource languages. However, applying LLMs to simultaneous machine translation (SimulMT) poses many challenges, including issues related to the training-inference mismatch arising from different decoding patterns. In this paper, we explore the feasibility of utilizing LLMs for SimulMT. Building upon conventional approaches, we introduce a simple yet effective mixture policy that enables LLMs to engage in SimulMT without requiring additional training. Furthermore, after Supervised Fine-Tuning (SFT) on a mixture of full and prefix sentences, the model exhibits significant performance improvements. Our experiments, conducted with Llama2-7B-chat on nine language pairs from the MUST-C dataset, demonstrate that LLM can achieve translation quality and latency comparable to dedicated SimulMT models.

Title: Scaled Prompt-Tuning for Few-Shot Natural Language Generation. (arXiv:2309.06759v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06759
• Code URL: null
• Copy Paste: [[2309.06759]] Scaled Prompt-Tuning for Few-Shot Natural Language Generation(http://arxiv.org/abs/2309.06759)
• Summary:

  The increasingly Large Language Models (LLMs) demonstrate stronger language understanding and generation capabilities, while the memory demand and computation cost of fine-tuning LLMs on downstream tasks are non-negligible. Besides, fine-tuning generally requires a certain amount of data from individual tasks whilst data collection cost is another issue to consider in real-world applications. In this work, we focus on Parameter-Efficient Fine-Tuning (PEFT) methods for few-shot Natural Language Generation (NLG), which freeze most parameters in LLMs and tune a small subset of parameters in few-shot cases so that memory footprint, training cost, and labeling cost are reduced while maintaining or even improving the performance. We propose a Scaled Prompt-Tuning (SPT) method which surpasses conventional PT with better performance and generalization ability but without an obvious increase in training cost. Further study on intermediate SPT suggests the superior transferability of SPT in few-shot scenarios, providing a recipe for data-deficient and computation-limited circumstances. Moreover, a comprehensive comparison of existing PEFT methods reveals that certain approaches exhibiting decent performance with modest training cost such as Prefix-Tuning in prior study could struggle in few-shot NLG tasks, especially on challenging datasets.

Title: Cognitive Mirage: A Review of Hallucinations in Large Language Models. (arXiv:2309.06794v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06794
• Code URL: https://github.com/hongbinye/cognitive-mirage-hallucinations-in-llms
• Copy Paste: [[2309.06794]] Cognitive Mirage: A Review of Hallucinations in Large Language Models(http://arxiv.org/abs/2309.06794)
• Summary:

  As large language models continue to develop in the field of AI, text generation systems are susceptible to a worrisome phenomenon known as hallucination. In this study, we summarize recent compelling insights into hallucinations in LLMs. We present a novel taxonomy of hallucinations from various text generation tasks, thus provide theoretical insights, detection methods and improvement approaches. Based on this, future research directions are proposed. Our contribution are threefold: (1) We provide a detailed and complete taxonomy for hallucinations appearing in text generation tasks; (2) We provide theoretical analyses of hallucinations in LLMs and provide existing detection and improvement methods; (3) We propose several research directions that can be developed in the future. As hallucinations garner significant attention from the community, we will maintain updates on relevant research progress.

Title: Auto-Regressive Next-Token Predictors are Universal Learners. (arXiv:2309.06979v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06979
• Code URL: null
• Copy Paste: [[2309.06979]] Auto-Regressive Next-Token Predictors are Universal Learners(http://arxiv.org/abs/2309.06979)
• Summary:

  Large language models display remarkable capabilities in logical and mathematical reasoning, allowing them to solve complex tasks. Interestingly, these abilities emerge in networks trained on the simple task of next-token prediction. In this work, we present a theoretical framework for studying auto-regressive next-token predictors. We demonstrate that even simple models such as linear next-token predictors, trained on Chain-of-Thought (CoT) data, can approximate any function efficiently computed by a Turing machine. We introduce a new complexity measure -- length complexity -- which measures the number of intermediate tokens in a CoT sequence required to approximate some target function, and analyze the interplay between length complexity and other notions of complexity. Finally, we show experimentally that simple next-token predictors, such as linear networks and shallow Multi-Layer Perceptrons (MLPs), display non-trivial performance on text generation and arithmetic tasks. Our results demonstrate that the power of language models can be attributed, to a great extent, to the auto-regressive next-token training scheme, and not necessarily to a particular choice of architecture.

Title: SafetyBench: Evaluating the Safety of Large Language Models with Multiple Choice Questions. (arXiv:2309.07045v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.07045
• Code URL: null
• Copy Paste: [[2309.07045]] SafetyBench: Evaluating the Safety of Large Language Models with Multiple Choice Questions(http://arxiv.org/abs/2309.07045)
• Summary:

  With the rapid development of Large Language Models (LLMs), increasing attention has been paid to their safety concerns. Consequently, evaluating the safety of LLMs has become an essential task for facilitating the broad applications of LLMs. Nevertheless, the absence of comprehensive safety evaluation benchmarks poses a significant impediment to effectively assess and enhance the safety of LLMs. In this work, we present SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data, facilitating the evaluation in both languages. Our extensive tests over 25 popular Chinese and English LLMs in both zero-shot and few-shot settings reveal a substantial performance advantage for GPT-4 over its counterparts, and there is still significant room for improving the safety of current LLMs. We believe SafetyBench will enable fast and comprehensive evaluation of LLMs' safety, and foster the development of safer LLMs. Data and evaluation guidelines are available at https://github.com/thu-coai/SafetyBench. Submission entrance and leaderboard are available at https://llmbench.ai/safety.

segmentation

Title: Action Segmentation Using 2D Skeleton Heatmaps. (arXiv:2309.06462v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06462
• Code URL: null
• Copy Paste: [[2309.06462]] Action Segmentation Using 2D Skeleton Heatmaps(http://arxiv.org/abs/2309.06462)
• Summary:

  This paper presents a 2D skeleton-based action segmentation method with applications in fine-grained human activity recognition. In contrast with state-of-the-art methods which directly take sequences of 3D skeleton coordinates as inputs and apply Graph Convolutional Networks (GCNs) for spatiotemporal feature learning, our main idea is to use sequences of 2D skeleton heatmaps as inputs and employ Temporal Convolutional Networks (TCNs) to extract spatiotemporal features. Despite lacking 3D information, our approach yields comparable/superior performances and better robustness against missing keypoints than previous methods on action segmentation datasets. Moreover, we improve the performances further by using both 2D skeleton heatmaps and RGB videos as inputs. To our best knowledge, this is the first work to utilize 2D skeleton heatmap inputs and the first work to explore 2D skeleton+RGB fusion for action segmentation.

Title: Multi-dimensional Fusion and Consistency for Semi-supervised Medical Image Segmentation. (arXiv:2309.06618v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06618
• Code URL: null
• Copy Paste: [[2309.06618]] Multi-dimensional Fusion and Consistency for Semi-supervised Medical Image Segmentation(http://arxiv.org/abs/2309.06618)
• Summary:

  In this paper, we introduce a novel semi-supervised learning framework tailored for medical image segmentation. Central to our approach is the innovative Multi-scale Text-aware ViT-CNN Fusion scheme. This scheme adeptly combines the strengths of both ViTs and CNNs, capitalizing on the unique advantages of both architectures as well as the complementary information in vision-language modalities. Further enriching our framework, we propose the Multi-Axis Consistency framework for generating robust pseudo labels, thereby enhancing the semi-supervised learning process. Our extensive experiments on several widely-used datasets unequivocally demonstrate the efficacy of our approach.

Title: Leveraging Foundation models for Unsupervised Audio-Visual Segmentation. (arXiv:2309.06728v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06728
• Code URL: null
• Copy Paste: [[2309.06728]] Leveraging Foundation models for Unsupervised Audio-Visual Segmentation(http://arxiv.org/abs/2309.06728)
• Summary:

  Audio-Visual Segmentation (AVS) aims to precisely outline audible objects in a visual scene at the pixel level. Existing AVS methods require fine-grained annotations of audio-mask pairs in supervised learning fashion. This limits their scalability since it is time consuming and tedious to acquire such cross-modality pixel level labels. To overcome this obstacle, in this work we introduce unsupervised audio-visual segmentation with no need for task-specific data annotations and model training. For tackling this newly proposed problem, we formulate a novel Cross-Modality Semantic Filtering (CMSF) approach to accurately associate the underlying audio-mask pairs by leveraging the off-the-shelf multi-modal foundation models (e.g., detection [1], open-world segmentation [2] and multi-modal alignment [3]). Guiding the proposal generation by either audio or visual cues, we design two training-free variants: AT-GDINO-SAM and OWOD-BIND. Extensive experiments on the AVS-Bench dataset show that our unsupervised approach can perform well in comparison to prior art supervised counterparts across complex scenarios with multiple auditory objects. Particularly, in situations where existing supervised AVS methods struggle with overlapping foreground objects, our models still excel in accurately segmenting overlapped auditory objects. Our code will be publicly released.

Title: Bayesian uncertainty-weighted loss for improved generalisability on polyp segmentation task. (arXiv:2309.06807v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06807
• Code URL: null
• Copy Paste: [[2309.06807]] Bayesian uncertainty-weighted loss for improved generalisability on polyp segmentation task(http://arxiv.org/abs/2309.06807)
• Summary:

  While several previous studies have devised methods for segmentation of polyps, most of these methods are not rigorously assessed on multi-center datasets. Variability due to appearance of polyps from one center to another, difference in endoscopic instrument grades, and acquisition quality result in methods with good performance on in-distribution test data, and poor performance on out-of-distribution or underrepresented samples. Unfair models have serious implications and pose a critical challenge to clinical applications. We adapt an implicit bias mitigation method which leverages Bayesian epistemic uncertainties during training to encourage the model to focus on underrepresented sample regions. We demonstrate the potential of this approach to improve generalisability without sacrificing state-of-the-art performance on a challenging multi-center polyp segmentation dataset (PolypGen) with different centers and image modalities.

Title: SAMUS: Adapting Segment Anything Model for Clinically-Friendly and Generalizable Ultrasound Image Segmentation. (arXiv:2309.06824v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06824
• Code URL: https://github.com/xianlin7/samus
• Copy Paste: [[2309.06824]] SAMUS: Adapting Segment Anything Model for Clinically-Friendly and Generalizable Ultrasound Image Segmentation(http://arxiv.org/abs/2309.06824)
• Summary:

  Segment anything model (SAM), an eminent universal image segmentation model, has recently gathered considerable attention within the domain of medical image segmentation. Despite the remarkable performance of SAM on natural images, it grapples with significant performance degradation and limited generalization when confronted with medical images, particularly with those involving objects of low contrast, faint boundaries, intricate shapes, and diminutive sizes. In this paper, we propose SAMUS, a universal model tailored for ultrasound image segmentation. In contrast to previous SAM-based universal models, SAMUS pursues not only better generalization but also lower deployment cost, rendering it more suitable for clinical applications. Specifically, based on SAM, a parallel CNN branch is introduced to inject local features into the ViT encoder through cross-branch attention for better medical image segmentation. Then, a position adapter and a feature adapter are developed to adapt SAM from natural to medical domains and from requiring large-size inputs (1024x1024) to small-size inputs (256x256) for more clinical-friendly deployment. A comprehensive ultrasound dataset, comprising about 30k images and 69k masks and covering six object categories, is collected for verification. Extensive comparison experiments demonstrate SAMUS's superiority against the state-of-the-art task-specific models and universal foundation models under both task-specific evaluation and generalization evaluation. Moreover, SAMUS is deployable on entry-level GPUs, as it has been liberated from the constraints of long sequence encoding. The code, data, and models will be released at https://github.com/xianlin7/SAMUS.