2024-11-22

Title: COOD: Concept-based Zero-shot OOD Detection

• Authors: Zhendong Liu, Yi Nian, Henry Peng Zou, Li Li, Xiyang Hu, Yue Zhao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13578
• Pdf URL: https://arxiv.org/pdf/2411.13578
• Copy Paste: [[2411.13578]] COOD: Concept-based Zero-shot OOD Detection(https://arxiv.org/abs/2411.13578)
• Keywords: robust, large language model
• Abstract: How can models effectively detect out-of-distribution (OOD) samples in complex, multi-label settings without extensive retraining? Existing OOD detection methods struggle to capture the intricate semantic relationships and label co-occurrences inherent in multi-label settings, often requiring large amounts of training data and failing to generalize to unseen label combinations. While large language models have revolutionized zero-shot OOD detection, they primarily focus on single-label scenarios, leaving a critical gap in handling real-world tasks where samples can be associated with multiple interdependent labels. To address these challenges, we introduce COOD, a novel zero-shot multi-label OOD detection framework. COOD leverages pre-trained vision-language models, enhancing them with a concept-based label expansion strategy and a new scoring function. By enriching the semantic space with both positive and negative concepts for each label, our approach models complex label dependencies, precisely differentiating OOD samples without the need for additional training. Extensive experiments demonstrate that our method significantly outperforms existing approaches, achieving approximately 95% average AUROC on both VOC and COCO datasets, while maintaining robust performance across varying numbers of labels and different types of OOD samples.

Title: A Multi-Server Information-Sharing Environment for Cross-Party Collaboration on A Private Cloud

• Authors: Jianping Zhang, Qiang Liu, Zhenzhong Hu, Jiarui Lin, Fangqiang Yu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13580
• Pdf URL: https://arxiv.org/pdf/2411.13580
• Copy Paste: [[2411.13580]] A Multi-Server Information-Sharing Environment for Cross-Party Collaboration on A Private Cloud(https://arxiv.org/abs/2411.13580)
• Keywords: privacy, extraction
• Abstract: Interoperability remains the key problem in multi-discipline collaboration based on building information modeling (BIM). Although various methods have been proposed to solve the technical issues of interoperability, such as data sharing and data consistency; organizational issues, including data ownership and data privacy, remain unresolved to date. These organizational issues prevent different stakeholders from sharing their data due to concerns regarding losing control of the data. This study proposes a multi-server information-sharing approach on a private cloud after analyzing the requirements for cross-party collaboration to address the aforementioned issues and prepare for massive data handling in the near future. This approach adopts a global controller to track the location, ownership and privacy of the data, which are stored in different servers that are controlled by different parties. Furthermore, data consistency conventions, parallel sub-model extraction, and sub-model integration with model verification are investigated in depth to support information sharing in a distributed environment and to maintain data consistency. Thus, with this approach, the ownership and privacy of the data can be controlled by its owner while still enabling certain required data to be shared with other parties. Application of the multi-server approach for information interoperability and cross-party collaboration is illustrated using a real construction project of an airport terminal. Validation shows that the proposed approach is feasible for maintaining the ownership and privacy of the data while supporting cross-party data sharing and collaboration at the same time, thus avoiding possible legal problems regarding data copyrights or other legal issues.

Title: Browser Extension for Fake URL Detection

• Authors: Latesh G. Malik, Rohini Shambharkar, Shivam Morey, Shubhlak Kanpate, Vedika Raut
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13581
• Pdf URL: https://arxiv.org/pdf/2411.13581
• Copy Paste: [[2411.13581]] Browser Extension for Fake URL Detection(https://arxiv.org/abs/2411.13581)
• Keywords: security, attack
• Abstract: In recent years, Cyber attacks have increased in number, and with them, the intensity of the attacks and their potential to damage the user have also increased significantly. In an ever-advancing world, users find it difficult to keep up with the latest developments in technology, which can leave them vulnerable to attacks. To avoid such situations we need tools to deter such attacks, for this machine learning models are among the best options. This paper presents a Browser Extension that uses machine learning models to enhance online security by integrating three crucial functionalities: Malicious URL detection, Spam Email detection and Network logs analysis. The proposed solution uses LGBM classifier for classification of Phishing websites, the model has been trained on a dataset with 87 features, this model achieved an accuracy of 96.5% with a precision of 96.8% and F1 score of 96.49%. The Model for Spam email detection uses Multinomial NB algorithm which has been trained on a dataset with over 5500 messages, this model achieved an accuracy of 97.09% with a precision of 100%. The results demonstrate the effectiveness of using machine learning models for cyber security.

Title: AddrLLM: Address Rewriting via Large Language Model on Nationwide Logistics Data

• Authors: Qinchen Yang, Zhiqing Hong, Dongjiang Cao, Haotian Wang, Zejun Xie, Tian He, Yunhuai Liu, Yu Yang, Desheng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13584
• Pdf URL: https://arxiv.org/pdf/2411.13584
• Copy Paste: [[2411.13584]] AddrLLM: Address Rewriting via Large Language Model on Nationwide Logistics Data(https://arxiv.org/abs/2411.13584)
• Keywords: large language model
• Abstract: Textual description of a physical location, commonly known as an address, plays an important role in location-based services(LBS) such as on-demand delivery and navigation. However, the prevalence of abnormal addresses, those containing inaccuracies that fail to pinpoint a location, have led to significant costs. Address rewriting has emerged as a solution to rectify these abnormal addresses. Despite the critical need, existing address rewriting methods are limited, typically tailored to correct specific error types, or frequently require retraining to process new address data effectively. In this study, we introduce AddrLLM, an innovative framework for address rewriting that is built upon a retrieval augmented large language model. AddrLLM overcomes aforementioned limitations through a meticulously designed Supervised Fine-Tuning module, an Address-centric Retrieval Augmented Generation module and a Bias-free Objective Alignment module. To the best of our knowledge, this study pioneers the application of LLM-based address rewriting approach to solve the issue of abnormal addresses. Through comprehensive offline testing with real-world data on a national scale and subsequent online deployment, AddrLLM has demonstrated superior performance in integration with existing logistics system. It has significantly decreased the rate of parcel re-routing by approximately 43\%, underscoring its exceptional efficacy in real-world applications.

Title: Artificial Intelligence in Cybersecurity: Building Resilient Cyber Diplomacy Frameworks

• Authors: Michael Stoltz
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13585
• Pdf URL: https://arxiv.org/pdf/2411.13585
• Copy Paste: [[2411.13585]] Artificial Intelligence in Cybersecurity: Building Resilient Cyber Diplomacy Frameworks(https://arxiv.org/abs/2411.13585)
• Keywords: secure, security
• Abstract: This paper explores how automation and artificial intelligence (AI) are transforming U.S. cyber diplomacy. Leveraging these technologies helps the U.S. manage the complexity and urgency of cyber diplomacy, improving decision-making, efficiency, and security. As global inter connectivity grows, cyber diplomacy, managing national interests in the digital space has become vital. The ability of AI and automation to quickly process vast data volumes enables timely responses to cyber threats and opportunities. This paper underscores the strategic integration of these tools to maintain U.S. competitive advantage and secure national interests. Automation enhances diplomatic communication and data processing, freeing diplomats to focus on strategic decisions. AI supports predictive analytics and real time decision making, offering critical insights and proactive measures during high stakes engagements. Case studies show AIs effectiveness in monitoring cyber activities and managing international cyber policy. Challenges such as ethical concerns, security vulnerabilities, and reliance on technology are also addressed, emphasizing human oversight and strong governance frameworks. Ensuring proper ethical guidelines and cybersecurity measures allows the U.S. to harness the benefits of automation and AI while mitigating risks. By adopting these technologies, U.S. cyber diplomacy can become more proactive and effective, navigating the evolving digital landscape with greater agility.

Title: Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study

• Authors: Xibo Sun, Jiarui Fang, Aoyu Li, Jinzhe Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13588
• Pdf URL: https://arxiv.org/pdf/2411.13588
• Copy Paste: [[2411.13588]] Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study(https://arxiv.org/abs/2411.13588)
• Keywords: diffusion, transformer
• Abstract: The increased model capacity of Diffusion Transformers (DiTs) and the demand for generating higher resolutions of images and videos have led to a significant rise in inference latency, impacting real-time performance adversely. While prior research has highlighted the presence of high similarity in activation values between adjacent diffusion steps (referred to as redundancy) and proposed various caching mechanisms to mitigate computational overhead, the exploration of redundancy in existing literature remains limited, with findings often not generalizable across different DiT models. This study aims to address this gap by conducting a comprehensive investigation into redundancy across a broad spectrum of mainstream DiT models. Our experimental analysis reveals substantial variations in the distribution of redundancy across diffusion steps among different DiT models. Interestingly, within a single model, the redundancy distribution remains stable regardless of variations in input prompts, step counts, or scheduling strategies. Given the lack of a consistent pattern across diverse models, caching strategies designed for a specific group of models may not easily transfer to others. To overcome this challenge, we introduce a tool for analyzing the redundancy of individual models, enabling subsequent research to develop tailored caching strategies for specific model architectures. The project is publicly available at this https URL.

Title: Towards Accessible Learning: Deep Learning-Based Potential Dysgraphia Detection and OCR for Potentially Dysgraphic Handwriting

• Authors: Vydeki D, Divyansh Bhandari, Pranav Pratap Patil, Aarush Anand Kulkarni
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13595
• Pdf URL: https://arxiv.org/pdf/2411.13595
• Copy Paste: [[2411.13595]] Towards Accessible Learning: Deep Learning-Based Potential Dysgraphia Detection and OCR for Potentially Dysgraphic Handwriting(https://arxiv.org/abs/2411.13595)
• Keywords: robust
• Abstract: Dysgraphia is a learning disorder that affects handwriting abilities, making it challenging for children to write legibly and consistently. Early detection and monitoring are crucial for providing timely support and interventions. This study applies deep learning techniques to address the dual tasks of dysgraphia detection and optical character recognition (OCR) on handwriting samples from children with potential dysgraphic symptoms. Using a dataset of handwritten samples from Malaysian schoolchildren, we developed a custom Convolutional Neural Network (CNN) model, alongside VGG16 and ResNet50, to classify handwriting as dysgraphic or non-dysgraphic. The custom CNN model outperformed the pre-trained models, achieving a test accuracy of 91.8% with high precision, recall, and AUC, demonstrating its robustness in identifying dysgraphic handwriting features. Additionally, an OCR pipeline was created to segment and recognize individual characters in dysgraphic handwriting, achieving a character recognition accuracy of approximately 43.5%. This research highlights the potential of deep learning in supporting dysgraphia assessment, laying a foundation for tools that could assist educators and clinicians in identifying dysgraphia and tracking handwriting progress over time. The findings contribute to advancements in assistive technologies for learning disabilities, offering hope for more accessible and accurate diagnostic tools in educational and clinical settings.

Title: Preserving Expert-Level Privacy in Offline Reinforcement Learning

• Authors: Navodita Sharma, Vishnu Vinod, Abhradeep Thakurta, Alekh Agarwal, Borja Balle, Christoph Dann, Aravindan Raghuveer
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13598
• Pdf URL: https://arxiv.org/pdf/2411.13598
• Copy Paste: [[2411.13598]] Preserving Expert-Level Privacy in Offline Reinforcement Learning(https://arxiv.org/abs/2411.13598)
• Keywords: privacy, protect
• Abstract: The offline reinforcement learning (RL) problem aims to learn an optimal policy from historical data collected by one or more behavioural policies (experts) by interacting with an environment. However, the individual experts may be privacy-sensitive in that the learnt policy may retain information about their precise choices. In some domains like personalized retrieval, advertising and healthcare, the expert choices are considered sensitive data. To provably protect the privacy of such experts, we propose a novel consensus-based expert-level differentially private offline RL training approach compatible with any existing offline RL algorithm. We prove rigorous differential privacy guarantees, while maintaining strong empirical performance. Unlike existing work in differentially private RL, we supplement the theory with proof-of-concept experiments on classic RL environments featuring large continuous state spaces, demonstrating substantial improvements over a natural baseline across multiple tasks.

Title: RadPhi-3: Small Language Models for Radiology

• Authors: Mercy Ranjit, Shaury Srivastav, Tanuja Ganu
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13604
• Pdf URL: https://arxiv.org/pdf/2411.13604
• Copy Paste: [[2411.13604]] RadPhi-3: Small Language Models for Radiology(https://arxiv.org/abs/2411.13604)
• Keywords: extraction
• Abstract: LLM based copilot assistants are useful in everyday tasks. There is a proliferation in the exploration of AI assistant use cases to support radiology workflows in a reliable manner. In this work, we present RadPhi-3, a Small Language Model instruction tuned from Phi-3-mini-4k-instruct with 3.8B parameters to assist with various tasks in radiology workflows. While impression summary generation has been the primary task which has been explored in prior works w.r.t radiology reports of Chest X-rays, we also explore other useful tasks like change summary generation comparing the current radiology report and its prior report, section extraction from radiology reports, tagging the reports with various pathologies and tubes, lines or devices present in them etc. In-addition, instruction tuning RadPhi-3 involved learning from a credible knowledge source used by radiologists, this http URL. RadPhi-3 can be used both to give reliable answers for radiology related queries as well as perform useful tasks related to radiology reports. RadPhi-3 achieves SOTA results on the RaLEs radiology report generation benchmark.

Title: Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization

• Authors: Hao Ju, Zhedong Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13610
• Pdf URL: https://arxiv.org/pdf/2411.13610
• Copy Paste: [[2411.13610]] Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization(https://arxiv.org/abs/2411.13610)
• Keywords: robust, diffusion
• Abstract: Existing approaches to drone visual geo-localization predominantly adopt the image-based setting, where a single drone-view snapshot is matched with images from other platforms. Such task formulation, however, underutilizes the inherent video output of the drone and is sensitive to occlusions and environmental constraints. To address these limitations, we formulate a new video-based drone geo-localization task and propose the Video2BEV paradigm. This paradigm transforms the video into a Bird's Eye View (BEV), simplifying the subsequent matching process. In particular, we employ Gaussian Splatting to reconstruct a 3D scene and obtain the BEV projection. Different from the existing transform methods, \eg, polar transform, our BEVs preserve more fine-grained details without significant distortion. To further improve model scalability toward diverse BEVs and satellite figures, our Video2BEV paradigm also incorporates a diffusion-based module for generating hard negative samples, which facilitates discriminative feature learning. To validate our approach, we introduce UniV, a new video-based geo-localization dataset that extends the image-based University-1652 dataset. UniV features flight paths at $30^\circ$ and $45^\circ$ elevation angles with increased frame rates of up to 10 frames per second (FPS). Extensive experiments on the UniV dataset show that our Video2BEV paradigm achieves competitive recall rates and outperforms conventional video-based methods. Compared to other methods, our proposed approach exhibits robustness at lower elevations with more occlusions.

Title: Efficient Streaming Voice Steganalysis in Challenging Detection Scenarios

• Authors: Pengcheng Zhou, Zhengyang Fang, Zhongliang Yang, Zhili Zhou, Linna Zhou
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13612
• Pdf URL: https://arxiv.org/pdf/2411.13612
• Copy Paste: [[2411.13612]] Efficient Streaming Voice Steganalysis in Challenging Detection Scenarios(https://arxiv.org/abs/2411.13612)
• Keywords: security
• Abstract: In recent years, there has been an increasing number of information hiding techniques based on network streaming media, focusing on how to covertly and efficiently embed secret information into real-time transmitted network media signals to achieve concealed communication. The misuse of these techniques can lead to significant security risks, such as the spread of malicious code, commands, and viruses. Current steganalysis methods for network voice streams face two major challenges: efficient detection under low embedding rates and short duration conditions. These challenges arise because, with low embedding rates (e.g., as low as 10%) and short transmission durations (e.g., only 0.1 second), detection models struggle to acquire sufficiently rich sample features, making effective steganalysis difficult. To address these challenges, this paper introduces a Dual-View VoIP Steganalysis Framework (DVSF). The framework first randomly obfuscates parts of the native steganographic descriptors in VoIP stream segments, making the steganographic features of hard-to-detect samples more pronounced and easier to learn. It then captures fine-grained local features related to steganography, building on the global features of VoIP. Specially constructed VoIP segment triplets further adjust the feature distances within the model. Ultimately, this method effectively address the detection difficulty in VoIP. Extensive experiments demonstrate that our method significantly improves the accuracy of streaming voice steganalysis in these challenging detection scenarios, surpassing existing state-of-the-art methods and offering superior near-real-time performance.

Title: Non-Linear Outlier Synthesis for Out-of-Distribution Detection

• Authors: Lars Doorenbos, Raphael Sznitman, Pablo Márquez-Neila
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13619
• Pdf URL: https://arxiv.org/pdf/2411.13619
• Copy Paste: [[2411.13619]] Non-Linear Outlier Synthesis for Out-of-Distribution Detection(https://arxiv.org/abs/2411.13619)
• Keywords: robust, diffusion
• Abstract: The reliability of supervised classifiers is severely hampered by their limitations in dealing with unexpected inputs, leading to great interest in out-of-distribution (OOD) detection. Recently, OOD detectors trained on synthetic outliers, especially those generated by large diffusion models, have shown promising results in defining robust OOD decision boundaries. Building on this progress, we present NCIS, which enhances the quality of synthetic outliers by operating directly in the diffusion's model embedding space rather than combining disjoint models as in previous work and by modeling class-conditional manifolds with a conditional volume-preserving network for more expressive characterization of the training distribution. We demonstrate that these improvements yield new state-of-the-art OOD detection results on standard ImageNet100 and CIFAR100 benchmarks and provide insights into the importance of data pre-processing and other key design choices. We make our code available at \url{this https URL}.

Title: Robust SG-NeRF: Robust Scene Graph Aided Neural Surface Reconstruction

• Authors: Yi Gu, Dongjun Ye, Zhaorui Wang, Jiaxu Wang, Jiahang Cao, Renjing Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13620
• Pdf URL: https://arxiv.org/pdf/2411.13620
• Copy Paste: [[2411.13620]] Robust SG-NeRF: Robust Scene Graph Aided Neural Surface Reconstruction(https://arxiv.org/abs/2411.13620)
• Keywords: robust
• Abstract: Neural surface reconstruction relies heavily on accurate camera poses as input. Despite utilizing advanced pose estimators like COLMAP or ARKit, camera poses can still be noisy. Existing pose-NeRF joint optimization methods handle poses with small noise (inliers) effectively but struggle with large noise (outliers), such as mirrored poses. In this work, we focus on mitigating the impact of outlier poses. Our method integrates an inlier-outlier confidence estimation scheme, leveraging scene graph information gathered during the data preparation phase. Unlike previous works directly using rendering metrics as the reference, we employ a detached color network that omits the viewing direction as input to minimize the impact caused by shape-radiance ambiguities. This enhanced confidence updating strategy effectively differentiates between inlier and outlier poses, allowing us to sample more rays from inlier poses to construct more reliable radiance fields. Additionally, we introduce a re-projection loss based on the current Signed Distance Function (SDF) and pose estimations, strengthening the constraints between matching image pairs. For outlier poses, we adopt a Monte Carlo re-localization method to find better solutions. We also devise a scene graph updating strategy to provide more accurate information throughout the training process. We validate our approach on the SG-NeRF and DTU datasets. Experimental results on various datasets demonstrate that our methods can consistently improve the reconstruction qualities and pose accuracies.

Title: Principles of Visual Tokens for Efficient Video Understanding

• Authors: Xinyue Hao, Gen Li, Shreyank N Gowda, Robert B Fisher, Jonathan Huang, Anurag Arnab, Laura Sevilla-Lara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13626
• Pdf URL: https://arxiv.org/pdf/2411.13626
• Copy Paste: [[2411.13626]] Principles of Visual Tokens for Efficient Video Understanding(https://arxiv.org/abs/2411.13626)
• Keywords: transformer
• Abstract: Video understanding has made huge strides in recent years, relying largely on the power of the transformer architecture. As this architecture is notoriously expensive and video is highly redundant, research into improving efficiency has become particularly relevant. This has led to many creative solutions, including token merging and token selection. While most methods succeed in reducing the cost of the model and maintaining accuracy, an interesting pattern arises: most methods do not outperform the random sampling baseline. In this paper we take a closer look at this phenomenon and make several observations. First, we develop an oracle for the value of tokens which exposes a clear Pareto distribution where most tokens have remarkably low value, and just a few carry most of the perceptual information. Second, we analyze why this oracle is extremely hard to learn, as it does not consistently coincide with visual cues. Third, we observe that easy videos need fewer tokens to maintain accuracy. We build on these and further insights to propose a lightweight video model we call LITE that can select a small number of tokens effectively, outperforming state-of-the-art and existing baselines across datasets (Kinetics400 and Something-Something-V2) in the challenging trade-off of computation (GFLOPs) vs accuracy.

Title: CryptoFormalEval: Integrating LLMs and Formal Verification for Automated Cryptographic Protocol Vulnerability Detection

• Authors: Cristian Curaba, Denis D'Ambrosi, Alessandro Minisini, Natalia Pérez-Campanero Antolín
• Subjects: cs.CR, cs.AI, cs.SC
• Abstract URL: https://arxiv.org/abs/2411.13627
• Pdf URL: https://arxiv.org/pdf/2411.13627
• Copy Paste: [[2411.13627]] CryptoFormalEval: Integrating LLMs and Formal Verification for Automated Cryptographic Protocol Vulnerability Detection(https://arxiv.org/abs/2411.13627)
• Keywords: security, attack, large language model
• Abstract: Cryptographic protocols play a fundamental role in securing modern digital infrastructure, but they are often deployed without prior formal verification. This could lead to the adoption of distributed systems vulnerable to attack vectors. Formal verification methods, on the other hand, require complex and time-consuming techniques that lack automatization. In this paper, we introduce a benchmark to assess the ability of Large Language Models (LLMs) to autonomously identify vulnerabilities in new cryptographic protocols through interaction with Tamarin: a theorem prover for protocol verification. We created a manually validated dataset of novel, flawed, communication protocols and designed a method to automatically verify the vulnerabilities found by the AI agents. Our results about the performances of the current frontier models on the benchmark provides insights about the possibility of cybersecurity applications by integrating LLMs with symbolic reasoning systems.

Title: MambaDETR: Query-based Temporal Modeling using State Space Model for Multi-View 3D Object Detection

• Authors: Tong Ning, Ke Lu, Xirui Jiang, Jian Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13628
• Pdf URL: https://arxiv.org/pdf/2411.13628
• Copy Paste: [[2411.13628]] MambaDETR: Query-based Temporal Modeling using State Space Model for Multi-View 3D Object Detection(https://arxiv.org/abs/2411.13628)
• Keywords: transformer
• Abstract: Utilizing temporal information to improve the performance of 3D detection has made great progress recently in the field of autonomous driving. Traditional transformer-based temporal fusion methods suffer from quadratic computational cost and information decay as the length of the frame sequence increases. In this paper, we propose a novel method called MambaDETR, whose main idea is to implement temporal fusion in the efficient state space. Moreover, we design a Motion Elimination module to remove the relatively static objects for temporal fusion. On the standard nuScenes benchmark, our proposed MambaDETR achieves remarkable result in the 3D object detection task, exhibiting state-of-the-art performance among existing temporal fusion methods.

Title: ID-Patch: Robust ID Association for Group Photo Personalization

• Authors: Yimeng Zhang, Tiancheng Zhi, Jing Liu, Shen Sang, Liming Jiang, Qing Yan, Sijia Liu, Linjie Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13632
• Pdf URL: https://arxiv.org/pdf/2411.13632
• Copy Paste: [[2411.13632]] ID-Patch: Robust ID Association for Group Photo Personalization(https://arxiv.org/abs/2411.13632)
• Keywords: robust, segmentation
• Abstract: The ability to synthesize personalized group photos and specify the positions of each identity offers immense creative potential. While such imagery can be visually appealing, it presents significant challenges for existing technologies. A persistent issue is identity (ID) leakage, where injected facial features interfere with one another, resulting in low face resemblance, incorrect positioning, and visual artifacts. Existing methods suffer from limitations such as the reliance on segmentation models, increased runtime, or a high probability of ID leakage. To address these challenges, we propose ID-Patch, a novel method that provides robust association between identities and 2D positions. Our approach generates an ID patch and ID embeddings from the same facial features: the ID patch is positioned on the conditional image for precise spatial control, while the ID embeddings integrate with text embeddings to ensure high resemblance. Experimental results demonstrate that ID-Patch surpasses baseline methods across metrics, such as face ID resemblance, ID-position association accuracy, and generation efficiency. Project Page is: this https URL

Title: FabuLight-ASD: Unveiling Speech Activity via Body Language

• Authors: Hugo Carneiro, Stefan Wermter
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE, cs.SD
• Abstract URL: https://arxiv.org/abs/2411.13674
• Pdf URL: https://arxiv.org/pdf/2411.13674
• Copy Paste: [[2411.13674]] FabuLight-ASD: Unveiling Speech Activity via Body Language(https://arxiv.org/abs/2411.13674)
• Keywords: robust
• Abstract: Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD's effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model's efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data. FabuLight-ASD's code and model weights are available at this https URL.

Title: Hymba: A Hybrid-head Architecture for Small Language Models

• Authors: Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, Pavlo Molchanov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13676
• Pdf URL: https://arxiv.org/pdf/2411.13676
• Copy Paste: [[2411.13676]] Hymba: A Hybrid-head Architecture for Small Language Models(https://arxiv.org/abs/2411.13676)
• Keywords: transformer
• Abstract: We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) for enhanced efficiency. Attention heads provide high-resolution recall, while SSM heads enable efficient context summarization. Additionally, we introduce learnable meta tokens that are prepended to prompts, storing critical information and alleviating the "forced-to-attend" burden associated with attention mechanisms. This model is further optimized by incorporating cross-layer key-value (KV) sharing and partial sliding window attention, resulting in a compact cache size. During development, we conducted a controlled study comparing various architectures under identical settings and observed significant advantages of our proposed architecture. Notably, Hymba achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model surpasses all sub-2B public models in performance and even outperforms Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size reduction, and 3.49x throughput.

Title: Differentially Private Learning Beyond the Classical Dimensionality Regime

• Authors: Cynthia Dwork, Pranay Tankala, Linjun Zhang
• Subjects: cs.LG, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2411.13682
• Pdf URL: https://arxiv.org/pdf/2411.13682
• Copy Paste: [[2411.13682]] Differentially Private Learning Beyond the Classical Dimensionality Regime(https://arxiv.org/abs/2411.13682)
• Keywords: privacy, robust
• Abstract: We initiate the study of differentially private learning in the proportional dimensionality regime, in which the number of data samples $n$ and problem dimension $d$ approach infinity at rates proportional to one another, meaning that $d / n \to \delta$ as $n \to \infty$ for an arbitrary, given constant $\delta \in (0, \infty)$. This setting is significantly more challenging than that of all prior theoretical work in high-dimensional differentially private learning, which, despite the name, has assumed that $\delta = 0$ or is sufficiently small for problems of sample complexity $O(d)$, a regime typically considered "low-dimensional" or "classical" by modern standards in high-dimensional statistics. We provide sharp theoretical estimates of the error of several well-studied differentially private algorithms for robust linear regression and logistic regression, including output perturbation, objective perturbation, and noisy stochastic gradient descent, in the proportional dimensionality regime. The $1 + o(1)$ factor precision of our error estimates enables a far more nuanced understanding of the price of privacy of these algorithms than that afforded by existing, coarser analyses, which are essentially vacuous in the regime we consider. We incorporate several probabilistic tools that have not previously been used to analyze differentially private learning algorithms, such as a modern Gaussian comparison inequality and recent universality laws with origins in statistical physics.

Title: Hierarchical Text Classification (HTC) vs. eXtreme Multilabel Classification (XML): Two Sides of the Same Medal

• Authors: Nerijus Bertalis, Paul Granse, Ferhat Gül, Florian Hauss, Leon Menkel, David Schüler, Tom Speier, Lukas Galke, Ansgar Scherp
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13687
• Pdf URL: https://arxiv.org/pdf/2411.13687
• Copy Paste: [[2411.13687]] Hierarchical Text Classification (HTC) vs. eXtreme Multilabel Classification (XML): Two Sides of the Same Medal(https://arxiv.org/abs/2411.13687)
• Keywords: transformer
• Abstract: Assigning a subset of labels from a fixed pool of labels to a given input text is a text classification problem with many real-world applications, such as in recommender systems. Two separate research streams address this issue. Hierarchical Text Classification (HTC) focuses on datasets with smaller label pools of hundreds of entries, accompanied by a semantic label hierarchy. In contrast, eXtreme Multi-Label Text Classification (XML) considers very large label pools with up to millions of entries, in which the labels are not arranged in any particular manner. However, in XML, a common approach is to construct an artificial hierarchy without any semantic information before or during the training process. Here, we investigate how state-of-the-art models from one domain perform when trained and tested on datasets from the other domain. The HBGL and HGLCR models from the HTC domain are trained and tested on the datasets Wiki10-31K, AmazonCat-13K, and Amazon-670K from the XML domain. On the other side, the XML models CascadeXML and XR-Transformer are trained and tested on the datasets Web of Science, The New York Times Annotated Corpus, and RCV1-V2 from the HTC domain. HTC models, on the other hand, are not equipped to handle the size of XML datasets and achieve poor transfer results. The code and numerous files that are needed to reproduce our results can be obtained from this https URL

Title: Investigating Graph Neural Networks and Classical Feature-Extraction Techniques in Activity-Cliff and Molecular Property Prediction

• Authors: Markus Dablander
• Subjects: cs.LG, q-bio.BM, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.13688
• Pdf URL: https://arxiv.org/pdf/2411.13688
• Copy Paste: [[2411.13688]] Investigating Graph Neural Networks and Classical Feature-Extraction Techniques in Activity-Cliff and Molecular Property Prediction(https://arxiv.org/abs/2411.13688)
• Keywords: robust, extraction
• Abstract: Molecular featurisation refers to the transformation of molecular data into numerical feature vectors. It is one of the key research areas in molecular machine learning and computational drug discovery. Recently, message-passing graph neural networks (GNNs) have emerged as a novel method to learn differentiable features directly from molecular graphs. While such techniques hold great promise, further investigations are needed to clarify if and when they indeed manage to definitively outcompete classical molecular featurisations such as extended-connectivity fingerprints (ECFPs) and physicochemical-descriptor vectors (PDVs). We systematically explore and further develop classical and graph-based molecular featurisation methods for two important tasks: molecular property prediction, in particular, quantitative structure-activity relationship (QSAR) prediction, and the largely unexplored challenge of activity-cliff (AC) prediction. We first give a technical description and critical analysis of PDVs, ECFPs and message-passing GNNs, with a focus on graph isomorphism networks (GINs). We then conduct a rigorous computational study to compare the performance of PDVs, ECFPs and GINs for QSAR and AC-prediction. Following this, we mathematically describe and computationally evaluate a novel twin neural network model for AC-prediction. We further introduce an operation called substructure pooling for the vectorisation of structural fingerprints as a natural counterpart to graph pooling in GNN architectures. We go on to propose Sort & Slice, a simple substructure-pooling technique for ECFPs that robustly outperforms hash-based folding at molecular property prediction. Finally, we outline two ideas for future research: (i) a graph-based self-supervised learning strategy to make classical molecular featurisations trainable, and (ii) trainable substructure-pooling via differentiable self-attention.

Title: Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU

• Authors: Haojia Sun, Yaqi Wang, Shuting Zhang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13691
• Pdf URL: https://arxiv.org/pdf/2411.13691
• Copy Paste: [[2411.13691]] Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU(https://arxiv.org/abs/2411.13691)
• Keywords: large language model
• Abstract: We designed a Retrieval-Augmented Generation (RAG) system to provide large language models with relevant documents for answering domain-specific questions about Pittsburgh and Carnegie Mellon University (CMU). We extracted over 1,800 subpages using a greedy scraping strategy and employed a hybrid annotation process, combining manual and Mistral-generated question-answer pairs, achieving an inter-annotator agreement (IAA) score of 0.7625. Our RAG framework integrates BM25 and FAISS retrievers, enhanced with a reranker for improved document retrieval accuracy. Experimental results show that the RAG system significantly outperforms a non-RAG baseline, particularly in time-sensitive and complex queries, with an F1 score improvement from 5.45% to 42.21% and recall of 56.18%. This study demonstrates the potential of RAG systems in enhancing answer precision and relevance, while identifying areas for further optimization in document retrieval and model training.

Title: PairSonic: Helping Groups Securely Exchange Contact Information

• Authors: Florentin Putz, Steffen Haesler, Thomas Völkl, Maximilian Gehring, Nils Rollshausen, Matthias Hollick
• Subjects: cs.CR, cs.HC, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.13693
• Pdf URL: https://arxiv.org/pdf/2411.13693
• Copy Paste: [[2411.13693]] PairSonic: Helping Groups Securely Exchange Contact Information(https://arxiv.org/abs/2411.13693)
• Keywords: secure
• Abstract: Securely exchanging contact information is essential for establishing trustworthy communication channels that facilitate effective online collaboration. However, current methods are neither user-friendly nor scalable for large groups of users. In response, we introduce PairSonic, a novel group pairing protocol that extends trust from physical encounters to online communication. PairSonic simplifies the pairing process by automating the tedious verification tasks of previous methods through an acoustic out-of-band channel using smartphones' built-in hardware. Our protocol not only facilitates connecting users for computer-supported collaboration, but also provides a more user-friendly and scalable solution to the authentication ceremonies currently used in end-to-end encrypted messengers like Signal or WhatsApp. PairSonic is available as open-source software: this https URL

Title: Decompose and Leverage Preferences from Expert Models for Improving Trustworthiness of MLLMs

• Authors: Rui Cao, Yuming Jiang, Michael Schlichtkrull, Andreas Vlachos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13697
• Pdf URL: https://arxiv.org/pdf/2411.13697
• Copy Paste: [[2411.13697]] Decompose and Leverage Preferences from Expert Models for Improving Trustworthiness of MLLMs(https://arxiv.org/abs/2411.13697)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) can enhance trustworthiness by aligning with human preferences. As human preference labeling is laborious, recent works employ evaluation models for assessing MLLMs' responses, using the model-based assessments to automate preference dataset construction. This approach, however, faces challenges with MLLMs' lengthy and compositional responses, which often require diverse reasoning skills that a single evaluation model may not fully possess. Additionally, most existing methods rely on closed-source models as evaluators. To address limitations, we propose DecompGen, a decomposable framework that uses an ensemble of open-sourced expert models. DecompGen breaks down each response into atomic verification tasks, assigning each task to an appropriate expert model to generate fine-grained assessments. The DecompGen feedback is used to automatically construct our preference dataset, DGPref. MLLMs aligned with DGPref via preference learning show improvements in trustworthiness, demonstrating the effectiveness of DecompGen.

Title: Test Security in Remote Testing Age: Perspectives from Process Data Analytics and AI

• Authors: Jiangang Hao, Michael Fauss
• Subjects: cs.CR, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.13699
• Pdf URL: https://arxiv.org/pdf/2411.13699
• Copy Paste: [[2411.13699]] Test Security in Remote Testing Age: Perspectives from Process Data Analytics and AI(https://arxiv.org/abs/2411.13699)
• Keywords: security
• Abstract: The COVID-19 pandemic has accelerated the implementation and acceptance of remotely proctored high-stake assessments. While the flexible administration of the tests brings forth many values, it raises test security-related concerns. Meanwhile, artificial intelligence (AI) has witnessed tremendous advances in the last five years. Many AI tools (such as the very recent ChatGPT) can generate high-quality responses to test items. These new developments require test security research beyond the statistical analysis of scores and response time. Data analytics and AI methods based on clickstream process data can get us deeper insight into the test-taking process and hold great promise for securing remotely administered high-stakes tests. This chapter uses real-world examples to show that this is indeed the case.

Title: Developing Normative Gait Cycle Parameters for Clinical Analysis Using Human Pose Estimation

• Authors: Rahm Ranjan, David Ahmedt-Aristizabal, Mohammad Ali Armin, Juno Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13716
• Pdf URL: https://arxiv.org/pdf/2411.13716
• Copy Paste: [[2411.13716]] Developing Normative Gait Cycle Parameters for Clinical Analysis Using Human Pose Estimation(https://arxiv.org/abs/2411.13716)
• Keywords: explainability
• Abstract: Gait analysis using computer vision is an emerging field in AI, offering clinicians an objective, multi-feature approach to analyse complex movements. Despite its promise, current applications using RGB video data alone are limited in measuring clinically relevant spatial and temporal kinematics and establishing normative parameters essential for identifying movement abnormalities within a gait cycle. This paper presents a data-driven method using RGB video data and 2D human pose estimation for developing normative kinematic gait parameters. By analysing joint angles, an established kinematic measure in biomechanics and clinical practice, we aim to enhance gait analysis capabilities and improve explainability. Our cycle-wise kinematic analysis enables clinicians to simultaneously measure and compare multiple joint angles, assessing individuals against a normative population using just monocular RGB video. This approach expands clinical capacity, supports objective decision-making, and automates the identification of specific spatial and temporal deviations and abnormalities within the gait cycle.

Title: Exploring Large Language Models for Climate Forecasting

• Authors: Yang Wang, Hassan A. Karimi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13724
• Pdf URL: https://arxiv.org/pdf/2411.13724
• Copy Paste: [[2411.13724]] Exploring Large Language Models for Climate Forecasting(https://arxiv.org/abs/2411.13724)
• Keywords: large language model
• Abstract: With the increasing impacts of climate change, there is a growing demand for accessible tools that can provide reliable future climate information to support planning, finance, and other decision-making applications. Large language models (LLMs), such as GPT-4, present a promising approach to bridging the gap between complex climate data and the general public, offering a way for non-specialist users to obtain essential climate insights through natural language interaction. However, an essential challenge remains under-explored: evaluating the ability of LLMs to provide accurate and reliable future climate predictions, which is crucial for applications that rely on anticipating climate trends. In this study, we investigate the capability of GPT-4 in predicting rainfall at short-term (15-day) and long-term (12-month) scales. We designed a series of experiments to assess GPT's performance under different conditions, including scenarios with and without expert data inputs. Our results indicate that GPT, when operating independently, tends to generate conservative forecasts, often reverting to historical averages in the absence of clear trend signals. This study highlights both the potential and challenges of applying LLMs for future climate predictions, providing insights into their integration with climate-related applications and suggesting directions for enhancing their predictive capabilities in the field.

Title: Delta-Influence: Unlearning Poisons via Influence Functions

• Authors: Wenjie Li, Jiawei Li, Christian Schroeder de Witt, Ameya Prabhu, Amartya Sanyal
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13731
• Pdf URL: https://arxiv.org/pdf/2411.13731
• Copy Paste: [[2411.13731]] Delta-Influence: Unlearning Poisons via Influence Functions(https://arxiv.org/abs/2411.13731)
• Keywords: attack
• Abstract: Addressing data integrity challenges, such as unlearning the effects of data poisoning after model training, is necessary for the reliable deployment of machine learning models. State-of-the-art influence functions, such as EK-FAC, often fail to accurately attribute abnormal model behavior to the specific poisoned training data responsible for the data poisoning attack. In addition, traditional unlearning algorithms often struggle to effectively remove the influence of poisoned samples, particularly when only a few affected examples can be identified. To address these challenge, we introduce $\Delta$-Influence, a novel approach that leverages influence functions to trace abnormal model behavior back to the responsible poisoned training data using as little as just one poisoned test example. $\Delta$-Influence applies data transformations that sever the link between poisoned training data and compromised test points without significantly affecting clean data. This allows $\Delta$-Influence to detect large negative shifts in influence scores following data transformations, a phenomenon we term as influence collapse, thereby accurately identifying poisoned training data. Unlearning this subset, e.g. through retraining, effectively eliminates the data poisoning. We validate our method across three vision-based poisoning attacks and three datasets, benchmarking against four detection algorithms and five unlearning strategies. We show that $\Delta$-Influence consistently achieves the best unlearning across all settings, showing the promise of influence functions for corrective unlearning. Our code is publicly available at: \url{this https URL}

Title: Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

• Authors: Tetiana Bas
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13738
• Pdf URL: https://arxiv.org/pdf/2411.13738
• Copy Paste: [[2411.13738]] Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics(https://arxiv.org/abs/2411.13738)
• Keywords: large language model
• Abstract: This study investigates gender bias in large language models (LLMs) by comparing their gender perception to that of human respondents, U.S. Bureau of Labor Statistics data, and a 50% no-bias benchmark. We created a new evaluation set using occupational data and role-specific sentences. Unlike common benchmarks included in LLM training data, our set is newly developed, preventing data leakage and test set contamination. Five LLMs were tested to predict the gender for each role using single-word answers. We used Kullback-Leibler (KL) divergence to compare model outputs with human perceptions, statistical data, and the 50% neutrality benchmark. All LLMs showed significant deviation from gender neutrality and aligned more with statistical data, still reflecting inherent biases.

Title: Federated Continual Learning for Edge-AI: A Comprehensive Survey

• Authors: Zi Wang, Fei Wu, Feng Yu, Yurui Zhou, Jia Hu, Geyong Min
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.13740
• Pdf URL: https://arxiv.org/pdf/2411.13740
• Copy Paste: [[2411.13740]] Federated Continual Learning for Edge-AI: A Comprehensive Survey(https://arxiv.org/abs/2411.13740)
• Keywords: privacy, federate
• Abstract: Edge-AI, the convergence of edge computing and artificial intelligence (AI), has become a promising paradigm that enables the deployment of advanced AI models at the network edge, close to users. In Edge-AI, federated continual learning (FCL) has emerged as an imperative framework, which fuses knowledge from different clients while preserving data privacy and retaining knowledge from previous tasks as it learns new ones. By so doing, FCL aims to ensure stable and reliable performance of learning models in dynamic and distributed environments. In this survey, we thoroughly review the state-of-the-art research and present the first comprehensive survey of FCL for Edge-AI. We categorize FCL methods based on three task characteristics: federated class continual learning, federated domain continual learning, and federated task continual learning. For each category, an in-depth investigation and review of the representative methods are provided, covering background, challenges, problem formalisation, solutions, and limitations. Besides, existing real-world applications empowered by FCL are reviewed, indicating the current progress and potential of FCL in diverse application domains. Furthermore, we discuss and highlight several prospective research directions of FCL such as algorithm-hardware co-design for FCL and FCL with foundation models, which could provide insights into the future development and practical deployment of FCL in the era of Edge-AI.

Title: FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splatting

• Authors: Ola Shorinwa, Jiankai Sun, Mac Schwager
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13753
• Pdf URL: https://arxiv.org/pdf/2411.13753
• Copy Paste: [[2411.13753]] FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splatting(https://arxiv.org/abs/2411.13753)
• Keywords: segmentation
• Abstract: We present FAST-Splat for fast, ambiguity-free semantic Gaussian Splatting, which seeks to address the main limitations of existing semantic Gaussian Splatting methods, namely: slow training and rendering speeds; high memory usage; and ambiguous semantic object localization. In deriving FAST-Splat , we formulate open-vocabulary semantic Gaussian Splatting as the problem of extending closed-set semantic distillation to the open-set (open-vocabulary) setting, enabling FAST-Splat to provide precise semantic object localization results, even when prompted with ambiguous user-provided natural-language queries. Further, by exploiting the explicit form of the Gaussian Splatting scene representation to the fullest extent, FAST-Splat retains the remarkable training and rendering speeds of Gaussian Splatting. Specifically, while existing semantic Gaussian Splatting methods distill semantics into a separate neural field or utilize neural models for dimensionality reduction, FAST-Splat directly augments each Gaussian with specific semantic codes, preserving the training, rendering, and memory-usage advantages of Gaussian Splatting over neural field methods. These Gaussian-specific semantic codes, together with a hash-table, enable semantic similarity to be measured with open-vocabulary user prompts and further enable FAST-Splat to respond with unambiguous semantic object labels and 3D masks, unlike prior methods. In experiments, we demonstrate that FAST-Splat is 4x to 6x faster to train with a 13x faster data pre-processing step, achieves between 18x to 75x faster rendering speeds, and requires about 3x smaller GPU memory, compared to the best-competing semantic Gaussian Splatting methods. Further, FAST-Splat achieves relatively similar or better semantic segmentation performance compared to existing methods. After the review period, we will provide links to the project website and the codebase.

Title: Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios

• Authors: Shantanu Jaiswal, Debaditya Roy, Basura Fernando, Cheston Tan
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13754
• Pdf URL: https://arxiv.org/pdf/2411.13754
• Copy Paste: [[2411.13754]] Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios(https://arxiv.org/abs/2411.13754)
• Keywords: robust, interpretability, transformer
• Abstract: Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capabilities beyond the immediate recognition and localization of objects and events. Here, we introduce a fully neural Iterative and Parallel Reasoning Mechanism (IPRM) that combines two distinct forms of computation -- iterative and parallel -- to better address complex VQA scenarios. Specifically, IPRM's "iterative" computation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need to be computed, stored, and recalled dynamically (e.g. when computing the query "determine the color of pen to the left of the child in red t-shirt sitting at the white table"). Meanwhile, its "parallel" computation allows for the simultaneous exploration of different reasoning paths and benefits more robust and efficient execution of operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts"). We design IPRM as a lightweight and fully-differentiable neural module that can be conveniently applied to both transformer and non-transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such as compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi-hop reasoning generalization (CLEVR-Humans) and causal event linking (CLEVRER-Humans). Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its errors.

Title: AttentionBreaker: Adaptive Evolutionary Optimization for Unmasking Vulnerabilities in LLMs through Bit-Flip Attacks

• Authors: Sanjay Das, Swastik Bhattacharya, Souvik Kundu, Shamik Kundu, Anand Menon, Arnab Raha, Kanad Basu
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13757
• Pdf URL: https://arxiv.org/pdf/2411.13757
• Copy Paste: [[2411.13757]] AttentionBreaker: Adaptive Evolutionary Optimization for Unmasking Vulnerabilities in LLMs through Bit-Flip Attacks(https://arxiv.org/abs/2411.13757)
• Keywords: attack, robust, transformer, large language model
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing (NLP), excelling in tasks like text generation and summarization. However, their increasing adoption in mission-critical applications raises concerns about hardware-based threats, particularly bit-flip attacks (BFAs). BFAs, enabled by fault injection methods such as Rowhammer, target model parameters in memory, compromising both integrity and performance. Identifying critical parameters for BFAs in the vast parameter space of LLMs poses significant challenges. While prior research suggests transformer-based architectures are inherently more robust to BFAs compared to traditional deep neural networks, we challenge this assumption. For the first time, we demonstrate that as few as three bit-flips can cause catastrophic performance degradation in an LLM with billions of parameters. Current BFA techniques are inadequate for exploiting this vulnerability due to the difficulty of efficiently identifying critical parameters within the immense parameter space. To address this, we propose AttentionBreaker, a novel framework tailored for LLMs that enables efficient traversal of the parameter space to identify critical parameters. Additionally, we introduce GenBFA, an evolutionary optimization strategy designed to refine the search further, isolating the most critical bits for an efficient and effective attack. Empirical results reveal the profound vulnerability of LLMs to AttentionBreaker. For example, merely three bit-flips (4.129 x 10^-9% of total parameters) in the LLaMA3-8B-Instruct 8-bit quantized (W8) model result in a complete performance collapse: accuracy on MMLU tasks drops from 67.3% to 0%, and Wikitext perplexity skyrockets from 12.6 to 4.72 x 10^5. These findings underscore the effectiveness of AttentionBreaker in uncovering and exploiting critical vulnerabilities within LLM architectures.

Title: A Framework for Evaluating LLMs Under Task Indeterminacy

• Authors: Luke Guerdan, Hanna Wallach, Solon Barocas, Alexandra Chouldechova
• Subjects: cs.LG, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.13760
• Pdf URL: https://arxiv.org/pdf/2411.13760
• Copy Paste: [[2411.13760]] A Framework for Evaluating LLMs Under Task Indeterminacy(https://arxiv.org/abs/2411.13760)
• Keywords: large language model
• Abstract: Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus. However, some tasks can be ambiguous -- i.e., they provide insufficient information to identify a unique interpretation -- or vague -- i.e., they do not clearly indicate where to draw the line when making a determination. Both ambiguity and vagueness can cause task indeterminacy -- the condition where some items in the evaluation corpus have more than one correct response. In this paper, we develop a framework for evaluating LLMs under task indeterminacy. Our framework disentangles the relationships between task specification, human ratings, and LLM responses in the LLM evaluation pipeline. Using our framework, we conduct a synthetic experiment showing that evaluations that use the "gold label" assumption underestimate the true performance. We also provide a method for estimating an error-adjusted performance interval given partial knowledge about indeterminate items in the evaluation corpus. We conclude by outlining implications of our work for the research community.

Title: Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals

• Authors: Hussni Mohd Zakir, Eric Tatt Wei Ho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13774
• Pdf URL: https://arxiv.org/pdf/2411.13774
• Copy Paste: [[2411.13774]] Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals(https://arxiv.org/abs/2411.13774)
• Keywords: segmentation
• Abstract: The Segment-Anything Model (SAM) is a vision foundation model for segmentation with a prompt-driven framework. SAM generates class-agnostic masks based on user-specified instance-referring prompts. However, adapting SAM for automated segmentation -- where manual input is absent -- of specific object classes often requires additional model training. We present Segment Any Class (SAC), a novel, training-free approach that task-adapts SAM for Multi-class segmentation. SAC generates Class-Region Proposals (CRP) on query images which allows us to automatically generate class-aware prompts on probable locations of class instances. CRPs are derived from elementary intra-class and inter-class feature distinctions without any additional training. Our method is versatile, accommodating any N-way K-shot configurations for the multi-class few-shot semantic segmentation (FSS) task. Unlike gradient-learning adaptation of generalist models which risk the loss of generalization and potentially suffer from catastrophic forgetting, SAC solely utilizes automated prompting and achieves superior results over state-of-the-art methods on the COCO-20i benchmark, particularly excelling in high N-way class scenarios. SAC is an interesting demonstration of a prompt-only approach to adapting foundation models for novel tasks with small, limited datasets without any modifications to the foundation model itself. This method offers interesting benefits such as intrinsic immunity to concept or feature loss and rapid, online task adaptation of foundation models.

Title: A Survey on Adversarial Robustness of LiDAR-based Machine Learning Perception in Autonomous Vehicles

• Authors: Junae Kim, Amardeep Kaur
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13778
• Pdf URL: https://arxiv.org/pdf/2411.13778
• Copy Paste: [[2411.13778]] A Survey on Adversarial Robustness of LiDAR-based Machine Learning Perception in Autonomous Vehicles(https://arxiv.org/abs/2411.13778)
• Keywords: security, defense, attack, robust
• Abstract: In autonomous driving, the combination of AI and vehicular technology offers great potential. However, this amalgamation comes with vulnerabilities to adversarial attacks. This survey focuses on the intersection of Adversarial Machine Learning (AML) and autonomous systems, with a specific focus on LiDAR-based systems. We comprehensively explore the threat landscape, encompassing cyber-attacks on sensors and adversarial perturbations. Additionally, we investigate defensive strategies employed in countering these threats. This paper endeavors to present a concise overview of the challenges and advances in securing autonomous driving systems against adversarial threats, emphasizing the need for robust defenses to ensure safety and security.

Title: NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews

• Authors: Michael Lu, Hyundong Justin Cho, Weiyan Shi, Jonathan May, Alexander Spangher
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13779
• Pdf URL: https://arxiv.org/pdf/2411.13779
• Copy Paste: [[2411.13779]] NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews(https://arxiv.org/abs/2411.13779)
• Keywords: extraction, large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in generating coherent text but often struggle with grounding language and strategic dialogue. To address this gap, we focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN, and reveal that LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions. Realizing that a fundamental deficit exists in multi-turn planning and strategic thinking, we develop a realistic simulated environment, incorporating source personas and persuasive elements, in order to facilitate the development of agents with longer-horizon rewards. Our experiments show that while source LLMs mimic human behavior in information sharing, interviewer LLMs struggle with recognizing when questions are answered and engaging persuasively, leading to suboptimal information extraction across model size and capability. These findings underscore the need for enhancing LLMs' strategic dialogue capabilities.

Title: $d_X$-Privacy for Text and the Curse of Dimensionality

• Authors: Hassan Jameel Asghar, Robin Carpentier, Benjamin Zi Hao Zhao, Dali Kaafar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13784
• Pdf URL: https://arxiv.org/pdf/2411.13784
• Copy Paste: [[2411.13784]] $d_X$-Privacy for Text and the Curse of Dimensionality(https://arxiv.org/abs/2411.13784)
• Keywords: privacy
• Abstract: A widely used method to ensure privacy of unstructured text data is the multidimensional Laplace mechanism for $d_X$-privacy, which is a relaxation of differential privacy for metric spaces. We identify an intriguing peculiarity of this mechanism. When applied on a word-by-word basis, the mechanism either outputs the original word, or completely dissimilar words, and very rarely any semantically similar words. We investigate this observation in detail, and tie it to the fact that the distance of the nearest neighbor of a word in any word embedding model (which are high-dimensional) is much larger than the relative difference in distances to any of its two consecutive neighbors. We also show that the dot product of the multidimensional Laplace noise vector with any word embedding plays a crucial role in designating the nearest neighbor. We derive the distribution, moments and tail bounds of this dot product. We further propose a fix as a post-processing step, which satisfactorily removes the above-mentioned issue.

Title: GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter

• Authors: Aniruddha Bala, Rohan Jaiswal, Loay Rashid, Siddharth Roheda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13794
• Pdf URL: https://arxiv.org/pdf/2411.13794
• Copy Paste: [[2411.13794]] GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter(https://arxiv.org/abs/2411.13794)
• Keywords: diffusion
• Abstract: Training of large-scale text-to-image and image-to-image models requires a huge amount of annotated data. While text-to-image datasets are abundant, data available for instruction-based image-to-image tasks like object addition and removal is limited. This is because of the several challenges associated with the data generation process, such as, significant human effort, limited automation, suboptimal end-to-end models, data diversity constraints and high expenses. We propose an automated data generation pipeline aimed at alleviating such limitations, and introduce GalaxyEdit - a large-scale image editing dataset for add and remove operations. We fine-tune the SD v1.5 model on our dataset and find that our model can successfully handle a broader range of objects and complex editing instructions, outperforming state-of-the-art methods in FID scores by 11.2\% and 26.1\% for add and remove tasks respectively. Furthermore, in light of on-device usage scenarios, we expand our research to include task-specific lightweight adapters leveraging the ControlNet-xs architecture. While ControlNet-xs excels in canny and depth guided generation, we propose to improve the communication between the control network and U-Net for more intricate add and remove tasks. We achieve this by enhancing ControlNet-xs with non-linear interaction layers based on Volterra filters. Our approach outperforms ControlNet-xs in both add/remove and canny-guided image generation tasks, highlighting the effectiveness of the proposed enhancement.

Title: Explaining GPT-4's Schema of Depression Using Machine Behavior Analysis

• Authors: Adithya V Ganesan, Vasudha Varadarajan, Yash Kumar Lal, Veerle C. Eijsbroek, Katarina Kjell, Oscar N.E. Kjell, Tanuja Dhanasekaran, Elizabeth C. Stade, Johannes C. Eichstaedt, Ryan L. Boyd, H. Andrew Schwartz, Lucie Flek
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13800
• Pdf URL: https://arxiv.org/pdf/2411.13800
• Copy Paste: [[2411.13800]] Explaining GPT-4's Schema of Depression Using Machine Behavior Analysis(https://arxiv.org/abs/2411.13800)
• Keywords: large language model
• Abstract: Use of large language models such as ChatGPT (GPT-4) for mental health support has grown rapidly, emerging as a promising route to assess and help people with mood disorders, like depression. However, we have a limited understanding of GPT-4's schema of mental disorders, that is, how it internally associates and interprets symptoms. In this work, we leveraged contemporary measurement theory to decode how GPT-4 interrelates depressive symptoms to inform both clinical utility and theoretical understanding. We found GPT-4's assessment of depression: (a) had high overall convergent validity (r = .71 with self-report on 955 samples, and r = .81 with experts judgments on 209 samples); (b) had moderately high internal consistency (symptom inter-correlates r = .23 to .78 ) that largely aligned with literature and self-report; except that GPT-4 (c) underemphasized suicidality's -- and overemphasized psychomotor's -- relationship with other symptoms, and (d) had symptom inference patterns that suggest nuanced hypotheses (e.g. sleep and fatigue are influenced by most other symptoms while feelings of worthlessness/guilt is mostly influenced by depressed mood).

Title: SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Model

• Authors: Christopher Nguyen, William Nguyen, Atsushi Suzuki, Daisuke Oku, Hong An Phan, Sang Dinh, Zooey Nguyen, Anh Ha, Shruti Raghavan, Huy Vo, Thang Nguyen, Lan Nguyen, Yoshikuni Hirayama
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.13802
• Pdf URL: https://arxiv.org/pdf/2411.13802
• Copy Paste: [[2411.13802]] SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Model(https://arxiv.org/abs/2411.13802)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated the potential to address some issues within the semiconductor industry. However, they are often general-purpose models that lack the specialized knowledge needed to tackle the unique challenges of this sector, such as the intricate physics and chemistry of semiconductor devices and processes. SemiKong, the first industry-specific LLM for the semiconductor domain, provides a foundation that can be used to develop tailored proprietary models. With SemiKong 1.0, we aim to develop a foundational model capable of understanding etching problems at an expert level. Our key contributions include (a) curating a comprehensive corpus of semiconductor-related texts, (b) creating a foundational model with in-depth semiconductor knowledge, and (c) introducing a framework for integrating expert knowledge, thereby advancing the evaluation process of domain-specific AI models. Through fine-tuning a pre-trained LLM using our curated dataset, we have shown that SemiKong outperforms larger, general-purpose LLMs in various semiconductor manufacturing and design tasks. Our extensive experiments underscore the importance of developing domain-specific LLMs as a foundation for company- or tool-specific proprietary models, paving the way for further research and applications in the semiconductor domain. Code and dataset will be available at this https URL

Title: MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control

• Authors: Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13807
• Pdf URL: https://arxiv.org/pdf/2411.13807
• Copy Paste: [[2411.13807]] MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control(https://arxiv.org/abs/2411.13807)
• Keywords: diffusion
• Abstract: The rapid advancement of diffusion models has greatly improved video synthesis, especially in controllable video generation, which is essential for applications like autonomous driving. However, existing methods are limited by scalability and how control conditions are integrated, failing to meet the needs for high-resolution and long videos for autonomous driving applications. In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT architecture, and tackle these challenges. Our method enhances scalability through flow matching and employs a progressive training strategy to manage complex scenarios. By incorporating spatial-temporal conditional encoding, MagicDriveDiT achieves precise control over spatial-temporal latents. Comprehensive experiments show its superior performance in generating realistic street scene videos with higher resolution and more frames. MagicDriveDiT significantly improves video generation quality and spatial-temporal controls, expanding its potential applications across various tasks in autonomous driving.

Title: AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning

• Authors: Changhai Zhou, Shiyang Zhang, Yuhua Zhou, Zekai Liu, Shichao Weng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13814
• Pdf URL: https://arxiv.org/pdf/2411.13814
• Copy Paste: [[2411.13814]] AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning(https://arxiv.org/abs/2411.13814)
• Keywords: large language model
• Abstract: Fine-tuning large language models (LLMs) under resource constraints is a significant challenge in deep learning. Low-Rank Adaptation (LoRA), pruning, and quantization are all effective methods for improving resource efficiency. However, combining them directly often results in suboptimal performance, especially with uniform quantization across all model layers. This is due to the complex, uneven interlayer relationships introduced by pruning, necessitating more refined quantization strategies. To address this, we propose AutoMixQ, an end-to-end optimization framework that selects optimal quantization configurations for each LLM layer. AutoMixQ leverages lightweight performance models to guide the selection process, significantly reducing time and computational resources compared to exhaustive search methods. By incorporating Pareto optimality, AutoMixQ balances memory usage and performance, approaching the upper bounds of model capability under strict resource constraints. Our experiments on widely used benchmarks show that AutoMixQ reduces memory consumption while achieving superior performance. For example, at a 30\% pruning rate in LLaMA-7B, AutoMixQ achieved 66.21\% on BoolQ compared to 62.45\% for LoRA and 58.96\% for LoftQ, while reducing memory consumption by 35.5\% compared to LoRA and 27.5\% compared to LoftQ.

Title: Robust Steganography with Boundary-Preserving Overflow Alleviation and Adaptive Error Correction

• Authors: Yu Cheng, Zhenlin Luo, Zhaoxia Yin
• Subjects: cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2411.13819
• Pdf URL: https://arxiv.org/pdf/2411.13819
• Copy Paste: [[2411.13819]] Robust Steganography with Boundary-Preserving Overflow Alleviation and Adaptive Error Correction(https://arxiv.org/abs/2411.13819)
• Keywords: security, robust
• Abstract: With the rapid evolution of the Internet, the vast amount of data has created opportunities for fostering the development of steganographic techniques. However, traditional steganographic techniques encounter challenges due to distortions in online social networks, such as JPEG recompression. Presently, research into the lossy operations of spatial truncation in JPEG recompression remains limited. Existing methods aim to ensure the stability of the quantized coefficients by reducing the effects of spatial truncation. Nevertheless, these approaches may induce notable alterations to image pixels, potentially compromising anti-steganalysis performance. In this study, we analyzed the overflow characteristics of spatial blocks and observed that pixel values at the boundaries of spatial blocks are more prone to overflow. Building upon this observation, we proposed a preprocessing method that performs overflow removal operations based on the actual overflow conditions of spatial blocks. After preprocessing, our algorithm enhances coefficient stability while minimizing modifications to spatial block boundaries, favoring image quality preservation. Subsequently, we employed adaptive error correction coding to reduce coding redundancy, thereby augmenting robustness and mitigating its impact on anti-steganalysis performance. The experimental results indicate that the proposed method possesses a strong embedding capacity, maintaining a high level of robustness while enhancing security.

Title: InstCache: A Predictive Cache for LLM Serving

• Authors: Longwei Zou, Tingfeng Liu, Kai Chen, Jiangang Kong, Yangdong Deng
• Subjects: cs.CL, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.13820
• Pdf URL: https://arxiv.org/pdf/2411.13820
• Copy Paste: [[2411.13820]] InstCache: A Predictive Cache for LLM Serving(https://arxiv.org/abs/2411.13820)
• Keywords: large language model
• Abstract: Large language models are revolutionizing every aspect of human life. However, the unprecedented power comes at the cost of significant computing intensity, suggesting long latency and large energy footprint. Key-Value Cache and Semantic Cache have been proposed as a solution to the above problem, but both suffer from limited scalability due to significant memory cost for each token or instruction embeddings. Motivated by the observations that most instructions are short, repetitive and predictable by LLMs, we propose to predict user-instructions by an instruction-aligned LLM and store them in a predictive cache, so-called InstCache. We introduce an instruction pre-population algorithm based on the negative log likelihood of instructions, determining the cache size with regard to the hit rate. The proposed InstCache is efficiently implemented as a hash table with minimal lookup latency for deployment. Experimental results show that InstCache can achieve up to 51.34% hit rate on LMSys dataset, which corresponds to a 2x speedup, at a memory cost of only 4.5GB.

Title: Interactive and Expressive Code-Augmented Planning with Large Language Models

• Authors: Anthony Z. Liu, Xinhe Wang, Jacob Sansom, Yao Fu, Jongwook Choi, Sungryull Sohn, Jaekyeom Kim, Honglak Lee
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13826
• Pdf URL: https://arxiv.org/pdf/2411.13826
• Copy Paste: [[2411.13826]] Interactive and Expressive Code-Augmented Planning with Large Language Models(https://arxiv.org/abs/2411.13826)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making, but often struggle with complex, long-horizon planning tasks. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. These techniques include using variables (to track important information) and functions (to divide complex tasks into smaller re-usable sub-tasks). However, purely code-based approaches can be error-prone and insufficient for handling ambiguous or unstructured data. To address these challenges, we propose REPL-Plan, an LLM planning approach that is fully code-expressive (it can utilize all the benefits of code) while also being dynamic (it can flexibly adapt from errors and use the LLM for fuzzy situations). In REPL-Plan, an LLM solves tasks by interacting with a Read-Eval-Print Loop (REPL), which iteratively executes and evaluates code, similar to language shells or interactive code notebooks, allowing the model to flexibly correct errors and handle tasks dynamically. We demonstrate that REPL-Plan achieves strong results across various planning domains compared to previous methods.

Title: Designing a Secure Device-to-Device File Transfer Mechanism

• Authors: Chaitanya Rahalkar, Anushka Virgaonkar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.13827
• Pdf URL: https://arxiv.org/pdf/2411.13827
• Copy Paste: [[2411.13827]] Designing a Secure Device-to-Device File Transfer Mechanism(https://arxiv.org/abs/2411.13827)
• Keywords: secure
• Abstract: Secure, reliable, and fast transfer of files across the Internet is a problem attempted to be solved through many application-layer protocols. In this paper, we aim to design a secure, reliable, opendesign, and performant file transfer protocol that is inspired by the WebRTC protocol stack. Traditionally, transferring files involves a publicly exposed (available on the public network) third-party server that serves the uploaded files to the receiver. Here, the third party server has to bear the storage and bandwidth cost to transfer the files between the two parties. We propose a protocol that uses a relay server to relay the files from the client to the server. A relay server has several advantages over a regular file-hosting server. Firstly, a relay server does not retain the uploaded files, it simply relays them. Secondly, a relay server has a full-duplex communication channel and therefore the receiver is not required to wait for the sender to upload the files completely. In this paper, we study available file transfer approaches and their known flaws. We propose our idea and compare our stack with the WebRTC stack. Finally, we perform empirical analysis and, benchmark our device-to-device transfer approach along with other available options including WebRTC.

Title: CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation

• Authors: Lin Sun, Jiale Cao, Jin Xie, Xiaoheng Jiang, Yanwei Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13836
• Pdf URL: https://arxiv.org/pdf/2411.13836
• Copy Paste: [[2411.13836]] CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2411.13836)
• Keywords: diffusion, segmentation
• Abstract: Contrastive Language-Image Pre-training (CLIP) exhibits strong zero-shot classification ability on various image-level tasks, leading to the research to adapt CLIP for pixel-level open-vocabulary semantic segmentation without additional training. The key is to improve spatial representation of image-level CLIP, such as replacing self-attention map at last layer with self-self attention map or vision foundation model based attention map. In this paper, we present a novel hierarchical framework, named CLIPer, that hierarchically improves spatial representation of CLIP. The proposed CLIPer includes an early-layer fusion module and a fine-grained compensation module. We observe that, the embeddings and attention maps at early layers can preserve spatial structural information. Inspired by this, we design the early-layer fusion module to generate segmentation map with better spatial coherence. Afterwards, we employ a fine-grained compensation module to compensate the local details using the self-attention maps of diffusion model. We conduct the experiments on seven segmentation datasets. Our proposed CLIPer achieves the state-of-the-art performance on these datasets. For instance, using ViT-L, CLIPer has the mIoU of 69.8% and 43.3% on VOC and COCO Object, outperforming ProxyCLIP by 9.2% and 4.1% respectively.

Title: Segment Anything in Light Fields for Real-Time Applications via Constrained Prompting

• Authors: Nikolai Goncharov, Donald G. Dansereau
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13840
• Pdf URL: https://arxiv.org/pdf/2411.13840
• Copy Paste: [[2411.13840]] Segment Anything in Light Fields for Real-Time Applications via Constrained Prompting(https://arxiv.org/abs/2411.13840)
• Keywords: segmentation
• Abstract: Segmented light field images can serve as a powerful representation in many of computer vision tasks exploiting geometry and appearance of objects, such as object pose tracking. In the light field domain, segmentation presents an additional objective of recognizing the same segment through all the views. Segment Anything Model 2 (SAM 2) allows producing semantically meaningful segments for monocular images and videos. However, using SAM 2 directly on light fields is highly ineffective due to unexploited constraints. In this work, we present a novel light field segmentation method that adapts SAM 2 to the light field domain without retraining or modifying the model. By utilizing the light field domain constraints, the method produces high quality and view-consistent light field masks, outperforming the SAM 2 video tracking baseline and working 7 times faster, with a real-time speed. We achieve this by exploiting the epipolar geometry cues to propagate the masks between the views, probing the SAM 2 latent space to estimate their occlusion, and further prompting SAM 2 for their refinement.

Title: Detecting Human Artifacts from Text-to-Image Models

• Authors: Kaihong Wang, Lingzhi Zhang, Jianming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13842
• Pdf URL: https://arxiv.org/pdf/2411.13842
• Copy Paste: [[2411.13842]] Detecting Human Artifacts from Text-to-Image Models(https://arxiv.org/abs/2411.13842)
• Keywords: diffusion, generative
• Abstract: Despite recent advancements, text-to-image generation models often produce images containing artifacts, especially in human figures. These artifacts appear as poorly generated human bodies, including distorted, missing, or extra body parts, leading to visual inconsistencies with typical human anatomy and greatly impairing overall fidelity. In this study, we address this challenge by curating Human Artifact Dataset (HAD), the first large-scale dataset specifically designed to identify and localize human artifacts. HAD comprises over 37,000 images generated by several popular text-to-image models, annotated for human artifact localization. Using this dataset, we train the Human Artifact Detection Models (HADM), which can identify diverse artifact types across multiple generative domains and demonstrate strong generalization, even on images from unseen generators. Additionally, to further improve generators' perception of human structural coherence, we use the predictions from our HADM as feedback for diffusion model finetuning. Our experiments confirm a reduction in human artifacts in the resulting model. Furthermore, we showcase a novel application of our HADM in an iterative inpainting framework to correct human artifacts in arbitrary images directly, demonstrating its utility in improving image quality. Our dataset and detection models are available at: \url{this https URL}.

Title: Multitask Learning for SAR Ship Detection with Gaussian-Mask Joint Segmentation

• Authors: Ming Zhao, Xin Zhang, André Kaup
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13847
• Pdf URL: https://arxiv.org/pdf/2411.13847
• Copy Paste: [[2411.13847]] Multitask Learning for SAR Ship Detection with Gaussian-Mask Joint Segmentation(https://arxiv.org/abs/2411.13847)
• Keywords: robust, segmentation
• Abstract: Detecting ships in synthetic aperture radar (SAR) images is challenging due to strong speckle noise, complex surroundings, and varying scales. This paper proposes MLDet, a multitask learning framework for SAR ship detection, consisting of object detection, speckle suppression, and target segmentation tasks. An angle classification loss with aspect ratio weighting is introduced to improve detection accuracy by addressing angular periodicity and object proportions. The speckle suppression task uses a dual-feature fusion attention mechanism to reduce noise and fuse shallow and denoising features, enhancing robustness. The target segmentation task, leveraging a rotated Gaussian-mask, aids the network in extracting target regions from cluttered backgrounds and improves detection efficiency with pixel-level predictions. The Gaussian-mask ensures ship centers have the highest probabilities, gradually decreasing outward under a Gaussian distribution. Additionally, a weighted rotated boxes fusion (WRBF) strategy combines multi-direction anchor predictions, filtering anchors beyond boundaries or with high overlap but low confidence. Extensive experiments on SSDD+ and HRSID datasets demonstrate the effectiveness and superiority of MLDet.

Title: Dealing with Synthetic Data Contamination in Online Continual Learning

• Authors: Maorong Wang, Nicolas Michel, Jiafeng Mao, Toshihiko Yamasaki
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13852
• Pdf URL: https://arxiv.org/pdf/2411.13852
• Copy Paste: [[2411.13852]] Dealing with Synthetic Data Contamination in Online Continual Learning(https://arxiv.org/abs/2411.13852)
• Keywords: diffusion
• Abstract: Image generation has shown remarkable results in generating high-fidelity realistic images, in particular with the advancement of diffusion-based models. However, the prevalence of AI-generated images may have side effects for the machine learning community that are not clearly identified. Meanwhile, the success of deep learning in computer vision is driven by the massive dataset collected on the Internet. The extensive quantity of synthetic data being added to the Internet would become an obstacle for future researchers to collect "clean" datasets without AI-generated content. Prior research has shown that using datasets contaminated by synthetic images may result in performance degradation when used for training. In this paper, we investigate the potential impact of contaminated datasets on Online Continual Learning (CL) research. We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Also, we propose Entropy Selection with Real-synthetic similarity Maximization (ESRM), a method to alleviate the performance deterioration caused by synthetic images when training online CL models. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. For reproducibility, the source code of our work is available at this https URL.

Title: Decoupled Sparse Priors Guided Diffusion Compression Model for Point Clouds

• Authors: Xiaoge Zhang, Zijie Wu, Mehwish Nasim, Mingtao Feng, Ajmal Mian
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.13860
• Pdf URL: https://arxiv.org/pdf/2411.13860
• Copy Paste: [[2411.13860]] Decoupled Sparse Priors Guided Diffusion Compression Model for Point Clouds(https://arxiv.org/abs/2411.13860)
• Keywords: diffusion
• Abstract: Lossy compression methods rely on an autoencoder to transform a point cloud into latent points for storage, leaving the inherent redundancy of latent representations unexplored. To reduce redundancy in latent points, we propose a sparse priors guided method that achieves high reconstruction quality, especially at high compression ratios. This is accomplished by a dual-density scheme separately processing the latent points (intended for reconstruction) and the decoupled sparse priors (intended for storage). Our approach features an efficient dual-density data flow that relaxes size constraints on latent points, and hybridizes a progressive conditional diffusion model to encapsulate essential details for reconstruction within the conditions, which are decoupled hierarchically to intra-point and inter-point priors. Specifically, our method encodes the original point cloud into latent points and decoupled sparse priors through separate encoders. Latent points serve as intermediates, while sparse priors act as adaptive conditions. We then employ a progressive attention-based conditional denoiser to generate latent points conditioned on the decoupled priors, allowing the denoiser to dynamically attend to geometric and semantic cues from the priors at each encoding and decoding layer. Additionally, we integrate the local distribution into the arithmetic encoder and decoder to enhance local context modeling of the sparse points. The original point cloud is reconstructed through a point decoder. Compared to state-of-the-art, our method obtains superior rate-distortion trade-off, evidenced by extensive evaluations on the ShapeNet dataset and standard test datasets from MPEG group including 8iVFB, and Owlii.

Title: Robust Detection of Watermarks for Large Language Models Under Human Edits

• Authors: Xiang Li, Feng Ruan, Huiyuan Wang, Qi Long, Weijie J. Su
• Subjects: cs.LG, cs.CL, math.ST, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.13868
• Pdf URL: https://arxiv.org/pdf/2411.13868
• Copy Paste: [[2411.13868]] Robust Detection of Watermarks for Large Language Models Under Human Edits(https://arxiv.org/abs/2411.13868)
• Keywords: robust, watermark, large language model
• Abstract: Watermarking has offered an effective approach to distinguishing text generated by large language models (LLMs) from human-written text. However, the pervasive presence of human edits on LLM-generated text dilutes watermark signals, thereby significantly degrading detection performance of existing methods. In this paper, by modeling human edits through mixture model detection, we introduce a new method in the form of a truncated goodness-of-fit test for detecting watermarked text under human edits, which we refer to as Tr-GoF. We prove that the Tr-GoF test achieves optimality in robust detection of the Gumbel-max watermark in a certain asymptotic regime of substantial text modifications and vanishing watermark signals. Importantly, Tr-GoF achieves this optimality \textit{adaptively} as it does not require precise knowledge of human edit levels or probabilistic specifications of the LLMs, in contrast to the optimal but impractical (Neyman--Pearson) likelihood ratio test. Moreover, we establish that the Tr-GoF test attains the highest detection efficiency rate in a certain regime of moderate text modifications. In stark contrast, we show that sum-based detection rules, as employed by existing methods, fail to achieve optimal robustness in both regimes because the additive nature of their statistics is less resilient to edit-induced noise. Finally, we demonstrate the competitive and sometimes superior empirical performance of the Tr-GoF test on both synthetic data and open-source LLMs in the OPT and LLaMA families.

Title: Sli2Vol+: Segmenting 3D Medical Images Based on an Object Estimation Guided Correspondence Flow Network

• Authors: Delin An, Pengfei Gu, Milan Sonka, Chaoli Wang, Danny Z. Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13873
• Pdf URL: https://arxiv.org/pdf/2411.13873
• Copy Paste: [[2411.13873]] Sli2Vol+: Segmenting 3D Medical Images Based on an Object Estimation Guided Correspondence Flow Network(https://arxiv.org/abs/2411.13873)
• Keywords: segmentation
• Abstract: Deep learning (DL) methods have shown remarkable successes in medical image segmentation, often using large amounts of annotated data for model training. However, acquiring a large number of diverse labeled 3D medical image datasets is highly difficult and expensive. Recently, mask propagation DL methods were developed to reduce the annotation burden on 3D medical images. For example, Sli2Vol~\cite{yeung2021sli2vol} proposed a self-supervised framework (SSF) to learn correspondences by matching neighboring slices via slice reconstruction in the training stage; the learned correspondences were then used to propagate a labeled slice to other slices in the test stage. But, these methods are still prone to error accumulation due to the inter-slice propagation of reconstruction errors. Also, they do not handle discontinuities well, which can occur between consecutive slices in 3D images, as they emphasize exploiting object continuity. To address these challenges, in this work, we propose a new SSF, called \proposed, {for segmenting any anatomical structures in 3D medical images using only a single annotated slice per training and testing volume.} Specifically, in the training stage, we first propagate an annotated 2D slice of a training volume to the other slices, generating pseudo-labels (PLs). Then, we develop a novel Object Estimation Guided Correspondence Flow Network to learn reliable correspondences between consecutive slices and corresponding PLs in a self-supervised manner. In the test stage, such correspondences are utilized to propagate a single annotated slice to the other slices of a test volume. We demonstrate the effectiveness of our method on various medical image segmentation tasks with different datasets, showing better generalizability across different organs, modalities, and modals. Code is available at \url{this https URL}

Title: Next-Generation Phishing: How LLM Agents Empower Cyber Attackers

• Authors: Khalifa Afane, Wenqi Wei, Ying Mao, Junaid Farooq, Juntao Chen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13874
• Pdf URL: https://arxiv.org/pdf/2411.13874
• Copy Paste: [[2411.13874]] Next-Generation Phishing: How LLM Agents Empower Cyber Attackers(https://arxiv.org/abs/2411.13874)
• Keywords: security, defense, attack, robust, large language model
• Abstract: The escalating threat of phishing emails has become increasingly sophisticated with the rise of Large Language Models (LLMs). As attackers exploit LLMs to craft more convincing and evasive phishing emails, it is crucial to assess the resilience of current phishing defenses. In this study we conduct a comprehensive evaluation of traditional phishing detectors, such as Gmail Spam Filter, Apache SpamAssassin, and Proofpoint, as well as machine learning models like SVM, Logistic Regression, and Naive Bayes, in identifying both traditional and LLM-rephrased phishing emails. We also explore the emerging role of LLMs as phishing detection tools, a method already adopted by companies like NTT Security Holdings and JPMorgan Chase. Our results reveal notable declines in detection accuracy for rephrased emails across all detectors, highlighting critical weaknesses in current phishing defenses. As the threat landscape evolves, our findings underscore the need for stronger security controls and regulatory oversight on LLM-generated content to prevent its misuse in creating advanced phishing attacks. This study contributes to the development of more effective Cyber Threat Intelligence (CTI) by leveraging LLMs to generate diverse phishing variants that can be used for data augmentation, harnessing the power of LLMs to enhance phishing detection, and paving the way for more robust and adaptable threat detection systems.

Title: CLFace: A Scalable and Resource-Efficient Continual Learning Framework for Lifelong Face Recognition

• Authors: Md Mahedi Hasan, Shoaib Meraj Sami, Nasser Nasrabadi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13886
• Pdf URL: https://arxiv.org/pdf/2411.13886
• Copy Paste: [[2411.13886]] CLFace: A Scalable and Resource-Efficient Continual Learning Framework for Lifelong Face Recognition(https://arxiv.org/abs/2411.13886)
• Keywords: privacy
• Abstract: An important aspect of deploying face recognition (FR) algorithms in real-world applications is their ability to learn new face identities from a continuous data stream. However, the online training of existing deep neural network-based FR algorithms, which are pre-trained offline on large-scale stationary datasets, encounter two major challenges: (I) catastrophic forgetting of previously learned identities, and (II) the need to store past data for complete retraining from scratch, leading to significant storage constraints and privacy concerns. In this paper, we introduce CLFace, a continual learning framework designed to preserve and incrementally extend the learned knowledge. CLFace eliminates the classification layer, resulting in a resource-efficient FR model that remains fixed throughout lifelong learning and provides label-free supervision to a student model, making it suitable for open-set face recognition during incremental steps. We introduce an objective function that employs feature-level distillation to reduce drift between feature maps of the student and teacher models across multiple stages. Additionally, it incorporates a geometry-preserving distillation scheme to maintain the orientation of the teacher model's feature embedding. Furthermore, a contrastive knowledge distillation is incorporated to continually enhance the discriminative power of the feature representation by matching similarities between new identities. Experiments on several benchmark FR datasets demonstrate that CLFace outperforms baseline approaches and state-of-the-art methods on unseen identities using both in-domain and out-of-domain datasets.

Title: Schemato -- An LLM for Netlist-to-Schematic Conversion

• Authors: Ryoga Matsuo, Stefan Uhlich, Arun Venkitaraman, Andrea Bonetti, Chia-Yu Hsieh, Ali Momeni, Lukas Mauch, Augusto Capone, Eisaku Ohbuchi, Lorenzo Servadei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13899
• Pdf URL: https://arxiv.org/pdf/2411.13899
• Copy Paste: [[2411.13899]] Schemato -- An LLM for Netlist-to-Schematic Conversion(https://arxiv.org/abs/2411.13899)
• Keywords: interpretability, large language model
• Abstract: Machine learning models are advancing circuit design, particularly in analog circuits. They typically generate netlists that lack human interpretability. This is a problem as human designers heavily rely on the interpretability of circuit diagrams or schematics to intuitively understand, troubleshoot, and develop designs. Hence, to integrate domain knowledge effectively, it is crucial to translate ML-generated netlists into interpretable schematics quickly and accurately. We propose Schemato, a large language model (LLM) for netlist-to-schematic conversion. In particular, we consider our approach in the two settings of converting netlists to .asc files for LTSpice and LATEX files for CircuiTikz schematics. Experiments on our circuit dataset show that Schemato achieves up to 93% compilation success rate for the netlist-to-LaTeX conversion task, surpassing the 26% rate scored by the state-of-the-art LLMs. Furthermore, our experiments show that Schemato generates schematics with a mean structural similarity index measure that is 3xhigher than the best performing LLMs, therefore closer to the reference human design.

Title: Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation

• Authors: Gayatri Deshmukh, Somsubhra De, Chirag Sehgal, Jishu Sen Gupta, Sparsh Mittal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13901
• Pdf URL: https://arxiv.org/pdf/2411.13901
• Copy Paste: [[2411.13901]] Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation(https://arxiv.org/abs/2411.13901)
• Keywords: generative
• Abstract: Specialized datasets that capture the fashion industry's rich language and styling elements can boost progress in AI-driven fashion design. We present FLORA (Fashion Language Outfit Representation for Apparel Generation), the first comprehensive dataset containing 4,330 curated pairs of fashion outfits and corresponding textual descriptions. Each description utilizes industry-specific terminology and jargon commonly used by professional fashion designers, providing precise and detailed insights into the outfits. Hence, the dataset captures the delicate features and subtle stylistic elements necessary to create high-fidelity fashion designs. We demonstrate that fine-tuning generative models on the FLORA dataset significantly enhances their capability to generate accurate and stylistically rich images from textual descriptions of fashion sketches. FLORA will catalyze the creation of advanced AI models capable of comprehending and producing subtle, stylistically rich fashion designs. It will also help fashion designers and end-users to bring their ideas to life. As a second orthogonal contribution, we introduce KAN Adapters, which leverage Kolmogorov-Arnold Networks (KAN) as adaptive modules. They serve as replacements for traditional MLP-based LoRA adapters. With learnable spline-based activations, KAN Adapters excel in modeling complex, non-linear relationships, achieving superior fidelity, faster convergence and semantic alignment. Extensive experiments and ablation studies on our proposed FLORA dataset validate the superiority of KAN Adapters over LoRA adapters. To foster further research and collaboration, we will open-source both the FLORA and our implementation code.

Title: PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation

• Authors: Zhijie Bao, Qingyun Liu, Ying Guo, Zhengqiang Ye, Jun Shen, Shirong Xie, Jiajie Peng, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13902
• Pdf URL: https://arxiv.org/pdf/2411.13902
• Copy Paste: [[2411.13902]] PIORS: Personalized Intelligent Outpatient Reception based on Large Language Model with Multi-Agents Medical Scenario Simulation(https://arxiv.org/abs/2411.13902)
• Keywords: large language model
• Abstract: In China, receptionist nurses face overwhelming workloads in outpatient settings, limiting their time and attention for each patient and ultimately reducing service quality. In this paper, we present the Personalized Intelligent Outpatient Reception System (PIORS). This system integrates an LLM-based reception nurse and a collaboration between LLM and hospital information system (HIS) into real outpatient reception setting, aiming to deliver personalized, high-quality, and efficient reception services. Additionally, to enhance the performance of LLMs in real-world healthcare scenarios, we propose a medical conversational data generation framework named Service Flow aware Medical Scenario Simulation (SFMSS), aiming to adapt the LLM to the real-world environments and PIORS settings. We evaluate the effectiveness of PIORS and SFMSS through automatic and human assessments involving 15 users and 15 clinical experts. The results demonstrate that PIORS-Nurse outperforms all baselines, including the current state-of-the-art model GPT-4o, and aligns with human preferences and clinical needs. Further details and demo can be found at this https URL

Title: Split Federated Learning Over Heterogeneous Edge Devices: Algorithm and Optimization

• Authors: Yunrui Sun, Gang Hu, Yinglei Teng, Dunbo Cai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13907
• Pdf URL: https://arxiv.org/pdf/2411.13907
• Copy Paste: [[2411.13907]] Split Federated Learning Over Heterogeneous Edge Devices: Algorithm and Optimization(https://arxiv.org/abs/2411.13907)
• Keywords: privacy, federate
• Abstract: Split Learning (SL) is a promising collaborative machine learning approach, enabling resource-constrained devices to train models without sharing raw data, while reducing computational load and preserving privacy simultaneously. However, current SL algorithms face limitations in training efficiency and suffer from prolonged latency, particularly in sequential settings, where the slowest device can bottleneck the entire process due to heterogeneous resources and frequent data exchanges between clients and servers. To address these challenges, we propose the Heterogeneous Split Federated Learning (HSFL) framework, which allows resource-constrained clients to train their personalized client-side models in parallel, utilizing different cut layers. Aiming to mitigate the impact of heterogeneous environments and accelerate the training process, we formulate a latency minimization problem that optimizes computational and transmission resources jointly. Additionally, we design a resource allocation algorithm that combines the Sample Average Approximation (SAA), Genetic Algorithm (GA), Lagrangian relaxation and Branch and Bound (B\&B) methods to efficiently solve this problem. Simulation results demonstrate that HSFL outperforms other frameworks in terms of both convergence rate and model accuracy on heterogeneous devices with non-iid data, while the optimization algorithm is better than other baseline methods in reducing latency.

Title: Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

• Authors: Honglin Li, Yuting Gao, Chenglu Zhu, Jingdong Chen, Ming Yang, Lin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13909
• Pdf URL: https://arxiv.org/pdf/2411.13909
• Copy Paste: [[2411.13909]] Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts(https://arxiv.org/abs/2411.13909)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) are closing the gap to human visual perception capability rapidly, while, still lag behind on attending to subtle images details or locating small objects precisely, etc. Common schemes to tackle these issues include deploying multiple vision encoders or operating on original high-resolution images. Few studies have concentrated on taking the textual instruction into improving visual representation, resulting in losing focus in some vision-centric tasks, a phenomenon we herein termed as Amblyopia. In this work, we introduce Panther, a MLLM that closely adheres to user instruction and locates targets of interests precisely, with the finesse of a black panther. Specifically, Panther comprises three integral components: Panther-VE, Panther-Bridge, and Panther-Decoder. Panther-VE integrates user instruction information at the early stages of the vision encoder, thereby extracting the most relevant and useful visual representations. The Panther-Bridge module, equipped with powerful filtering capabilities, significantly reduces redundant visual information, leading to a substantial savings in training costs. The Panther-Decoder is versatile and can be employed with any decoder-only architecture of LLMs without discrimination. Experimental results, particularly on vision-centric benchmarks, have demonstrated the effectiveness of Panther.

Title: Quantization without Tears

• Authors: Minghao Fu, Hao Yu, Jie Shao, Junjie Zhou, Ke Zhu, Jianxin Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13918
• Pdf URL: https://arxiv.org/pdf/2411.13918
• Copy Paste: [[2411.13918]] Quantization without Tears(https://arxiv.org/abs/2411.13918)
• Keywords: robust
• Abstract: Deep neural networks, while achieving remarkable success across diverse tasks, demand significant resources, including computation, GPU memory, bandwidth, storage, and energy. Network quantization, as a standard compression and acceleration technique, reduces storage costs and enables potential inference acceleration by discretizing network weights and activations into a finite set of integer values. However, current quantization methods are often complex and sensitive, requiring extensive task-specific hyperparameters, where even a single misconfiguration can impair model performance, limiting generality across different models and tasks. In this paper, we propose Quantization without Tears (QwT), a method that simultaneously achieves quantization speed, accuracy, simplicity, and generality. The key insight of QwT is to incorporate a lightweight additional structure into the quantized network to mitigate information loss during quantization. This structure consists solely of a small set of linear layers, keeping the method simple and efficient. More importantly, it provides a closed-form solution, allowing us to improve accuracy effortlessly under 2 minutes. Extensive experiments across various vision, language, and multimodal tasks demonstrate that QwT is both highly effective and versatile. In fact, our approach offers a robust solution for network quantization that combines simplicity, accuracy, and adaptability, which provides new insights for the design of novel quantization paradigms.

Title: NBMLSS: probabilistic forecasting of electricity prices via Neural Basis Models for Location Scale and Shape

• Authors: Alessandro Brusaferri, Danial Ramin, Andrea Ballarino
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13921
• Pdf URL: https://arxiv.org/pdf/2411.13921
• Copy Paste: [[2411.13921]] NBMLSS: probabilistic forecasting of electricity prices via Neural Basis Models for Location Scale and Shape(https://arxiv.org/abs/2411.13921)
• Keywords: interpretability
• Abstract: Forecasters using flexible neural networks (NN) in multi-horizon distributional regression setups often struggle to gain detailed insights into the underlying mechanisms that lead to the predicted feature-conditioned distribution parameters. In this work, we deploy a Neural Basis Model for Location, Scale and Shape, that blends the principled interpretability of GAMLSS with a computationally scalable shared basis decomposition, combined by linear projections supporting dedicated stepwise and parameter-wise feature shape functions aggregations. Experiments have been conducted on multiple market regions, achieving probabilistic forecasting performance comparable to that of distributional neural networks, while providing more insights into the model behavior through the learned nonlinear feature level maps to the distribution parameters across the prediction steps.

Title: Multimodal 3D Reasoning Segmentation with Complex Scenes

• Authors: Xueying Jiang, Lewei Lu, Ling Shao, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13927
• Pdf URL: https://arxiv.org/pdf/2411.13927
• Copy Paste: [[2411.13927]] Multimodal 3D Reasoning Segmentation with Complex Scenes(https://arxiv.org/abs/2411.13927)
• Keywords: segmentation
• Abstract: The recent development in multimodal learning has greatly advanced the research in 3D scene understanding in various real-world tasks such as embodied AI. However, most existing work shares two typical constraints: 1) they are short of reasoning ability for interaction and interpretation of human intension and 2) they focus on scenarios with single-category objects only which leads to over-simplified textual descriptions due to the negligence of multi-object scenarios and spatial relations among objects. We bridge the research gaps by proposing a 3D reasoning segmentation task for multiple objects in scenes. The task allows producing 3D segmentation masks and detailed textual explanations as enriched by 3D spatial relations among objects. To this end, we create ReasonSeg3D, a large-scale and high-quality benchmark that integrates 3D spatial relations with generated question-answer pairs and 3D segmentation masks. In addition, we design MORE3D, a simple yet effective method that enables multi-object 3D reasoning segmentation with user questions and textual outputs. Extensive experiments show that MORE3D excels in reasoning and segmenting complex multi-object 3D scenes, and the created ReasonSeg3D offers a valuable platform for future exploration of 3D reasoning segmentation. The dataset and code will be released.

Title: Transforming Engineering Diagrams: A Novel Approach for P&ID Digitization using Transformers

• Authors: Jan Marius Stürmer, Marius Graumann, Tobias Koch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13929
• Pdf URL: https://arxiv.org/pdf/2411.13929
• Copy Paste: [[2411.13929]] Transforming Engineering Diagrams: A Novel Approach for P&ID Digitization using Transformers(https://arxiv.org/abs/2411.13929)
• Keywords: extraction, transformer
• Abstract: The digitization of complex technical systems, such as Piping and Instrumentation Diagrams (P&IDs), is crucial for efficient maintenance and operation of complex systems in hydraulic and process engineering. Previous approaches often rely on separate modules that analyze diagram elements individually, neglecting the diagram's overall structure. We address this limitation by proposing a novel approach that utilizes the Relationformer, a state-of-the-art deep learning architecture, to extract graphs from P&IDs. Our method leverages the ability of the Relationformer to simultaneously detect objects and their relationships in images, making it suitable for the task of graph extraction from engineering diagrams. We apply our proposed approach to both real-world and synthetically created P&ID datasets, and evaluate its effectiveness by comparing it with a modular digitization approach based on recent literature. We present PID2Graph, the first publicly accessible P&ID dataset featuring comprehensive labels for the graph structure, including symbols, nodes and their connections that is used for evaluation. To understand the effect of patching and stitching of both of the approaches, we compare values before and after merging the patches. For the real-world data, the Relationformer achieves convincing results, outperforming the modular digitization approach for edge detection by more than 25%. Our work provides a comprehensive framework for assessing the performance of P&ID digitization methods and opens up new avenues for research in this area using transformer architectures. The P&ID dataset used for evaluation will be published and publicly available upon acceptance of the paper.

Title: Learning to Cooperate with Humans using Generative Agents

• Authors: Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, Natasha Jaques
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2411.13934
• Pdf URL: https://arxiv.org/pdf/2411.13934
• Copy Paste: [[2411.13934]] Learning to Cooperate with Humans using Generative Agents(https://arxiv.org/abs/2411.13934)
• Keywords: generative
• Abstract: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method -- \textbf{G}enerative \textbf{A}gent \textbf{M}odeling for \textbf{M}ulti-agent \textbf{A}daptation (GAMMA) -- on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.

Title: Separable Mixture of Low-Rank Adaptation for Continual Visual Instruction Tuning

• Authors: Ziqi Wang, Chang Che, Qi Wang, Yangyang Li, Zenglin Shi, Meng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13949
• Pdf URL: https://arxiv.org/pdf/2411.13949
• Copy Paste: [[2411.13949]] Separable Mixture of Low-Rank Adaptation for Continual Visual Instruction Tuning(https://arxiv.org/abs/2411.13949)
• Keywords: robust, large language model
• Abstract: Visual instruction tuning (VIT) enables multimodal large language models (MLLMs) to effectively handle a wide range of vision tasks by framing them as language-based instructions. Building on this, continual visual instruction tuning (CVIT) extends the capability of MLLMs to incrementally learn new tasks, accommodating evolving functionalities. While prior work has advanced CVIT through the development of new benchmarks and approaches to mitigate catastrophic forgetting, these efforts largely follow traditional continual learning paradigms, neglecting the unique challenges specific to CVIT. We identify a dual form of catastrophic forgetting in CVIT, where MLLMs not only forget previously learned visual understanding but also experience a decline in instruction following abilities as they acquire new tasks. To address this, we introduce the Separable Mixture of Low-Rank Adaptation (SMoLoRA) framework, which employs separable routing through two distinct modules - one for visual understanding and another for instruction following. This dual-routing design enables specialized adaptation in both domains, preventing forgetting while improving performance. Furthermore, we propose a novel CVIT benchmark that goes beyond existing benchmarks by additionally evaluating a model's ability to generalize to unseen tasks and handle diverse instructions across various tasks. Extensive experiments demonstrate that SMoLoRA outperforms existing methods in mitigating dual forgetting, improving generalization to unseen tasks, and ensuring robustness in following diverse instructions.

Title: A Dataset for Evaluating Online Anomaly Detection Approaches for Discrete Multivariate Time Series

• Authors: Lucas Correia, Jan-Christoph Goos, Thomas Bäck, Anna V. Kononova
• Subjects: cs.LG, cs.AI, cs.CE, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.13951
• Pdf URL: https://arxiv.org/pdf/2411.13951
• Copy Paste: [[2411.13951]] A Dataset for Evaluating Online Anomaly Detection Approaches for Discrete Multivariate Time Series(https://arxiv.org/abs/2411.13951)
• Keywords: robust
• Abstract: Benchmarking anomaly detection approaches for multivariate time series is challenging due to the lack of high-quality datasets. Current publicly available datasets are too small, not diverse and feature trivial anomalies, which hinders measurable progress in this research area. We propose a solution: a diverse, extensive, and non-trivial dataset generated via state-of-the-art simulation tools that reflects realistic behaviour of an automotive powertrain, including its multivariate, dynamic and variable-state properties. To cater for both unsupervised and semi-supervised anomaly detection settings, as well as time series generation and forecasting, we make different versions of the dataset available, where training and test subsets are offered in contaminated and clean versions, depending on the task. We also provide baseline results from a small selection of approaches based on deterministic and variational autoencoders, as well as a non-parametric approach. As expected, the baseline experimentation shows that the approaches trained on the semi-supervised version of the dataset outperform their unsupervised counterparts, highlighting a need for approaches more robust to contaminated training data.

Title: Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion

• Authors: Jinhong He, Shivakumara Palaiahnakote, Aoxiang Ning, Minglong Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13961
• Pdf URL: https://arxiv.org/pdf/2411.13961
• Copy Paste: [[2411.13961]] Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion(https://arxiv.org/abs/2411.13961)
• Keywords: robust, diffusion
• Abstract: Due to the singularity of real-world paired datasets and the complexity of low-light environments, this leads to supervised methods lacking a degree of scene generalisation. Meanwhile, limited by poor lighting and content guidance, existing zero-shot methods cannot handle unknown severe degradation well. To address this problem, we will propose a new zero-shot low-light enhancement method to compensate for the lack of light and structural information in the diffusion sampling process by effectively combining the wavelet and Fourier frequency domains to construct rich a priori information. The key to the inspiration comes from the similarity between the wavelet and Fourier frequency domains: both light and structure information are closely related to specific frequency domain regions, respectively. Therefore, by transferring the diffusion process to the wavelet low-frequency domain and combining the wavelet and Fourier frequency domains by continuously decomposing them in the inverse process, the constructed rich illumination prior is utilised to guide the image generation enhancement process. Sufficient experiments show that the framework is robust and effective in various scenarios. The code will be available at: \href{this https URL}{this https URL}.

Title: Transforming Static Images Using Generative Models for Video Salient Object Detection

• Authors: Suhwan Cho, Minhyeok Lee, Jungho Lee, Sangyoun Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13975
• Pdf URL: https://arxiv.org/pdf/2411.13975
• Copy Paste: [[2411.13975]] Transforming Static Images Using Generative Models for Video Salient Object Detection(https://arxiv.org/abs/2411.13975)
• Keywords: diffusion, generative
• Abstract: In many video processing tasks, leveraging large-scale image datasets is a common strategy, as image data is more abundant and facilitates comprehensive knowledge transfer. A typical approach for simulating video from static images involves applying spatial transformations, such as affine transformations and spline warping, to create sequences that mimic temporal progression. However, in tasks like video salient object detection, where both appearance and motion cues are critical, these basic image-to-video techniques fail to produce realistic optical flows that capture the independent motion properties of each object. In this study, we show that image-to-video diffusion models can generate realistic transformations of static images while understanding the contextual relationships between image components. This ability allows the model to generate plausible optical flows, preserving semantic integrity while reflecting the independent motion of scene elements. By augmenting individual images in this way, we create large-scale image-flow pairs that significantly enhance model training. Our approach achieves state-of-the-art performance across all public benchmark datasets, outperforming existing approaches.

Title: On the Fairness, Diversity and Reliability of Text-to-Image Generative Models

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13981
• Pdf URL: https://arxiv.org/pdf/2411.13981
• Copy Paste: [[2411.13981]] On the Fairness, Diversity and Reliability of Text-to-Image Generative Models(https://arxiv.org/abs/2411.13981)
• Keywords: fair, generative
• Abstract: The widespread availability of multimodal generative models has sparked critical discussions on their fairness, reliability, and potential for misuse. While text-to-image models can produce high-fidelity, user-guided images, they also exhibit unpredictable behavior and vulnerabilities, which can be exploited to manipulate class or concept representations. To address this, we propose an evaluation framework designed to assess model reliability through their responses to globally- and locally-applied `semantic' perturbations in the embedding space, pinpointing inputs that trigger unreliable behavior. Our approach offers deeper insights into two essential aspects: (i) generative diversity, evaluating the breadth of visual representations for learned concepts, and (ii) generative fairness, examining how removing concepts from input prompts affects semantic guidance. Beyond these evaluations, our method lays the groundwork for detecting unreliable, bias-injected models and retrieval of bias provenance. We will release our code. Keywords: Fairness, Reliability, AI Ethics, Bias, Text-to-Image Models

Title: Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13982
• Pdf URL: https://arxiv.org/pdf/2411.13982
• Copy Paste: [[2411.13982]] Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction(https://arxiv.org/abs/2411.13982)
• Keywords: generative
• Abstract: Training multimodal generative models on large, uncurated datasets can result in users being exposed to harmful, unsafe and controversial or culturally-inappropriate outputs. While model editing has been proposed to remove or filter undesirable concepts in embedding and latent spaces, it can inadvertently damage learned manifolds, distorting concepts in close semantic proximity. We identify limitations in current model editing techniques, showing that even benign, proximal concepts may become misaligned. To address the need for safe content generation, we propose a modular, dynamic solution that leverages safety-context embeddings and a dual reconstruction process using tunable weighted summation in the latent space to generate safer images. Our method preserves global context without compromising the structural integrity of the learned manifolds. We achieve state-of-the-art results on safe image generation benchmarks, while offering controllable variation of model safety. We identify trade-offs between safety and censorship, which presents a necessary perspective in the development of ethical AI models. We will release our code. Keywords: Text-to-Image Models, Generative AI, Safety, Reliability, Model Editing

Title: Generative Intervention Models for Causal Perturbation Modeling

• Authors: Nora Schneider, Lars Lorch, Niki Kilbertus, Bernhard Schölkopf, Andreas Krause
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.14003
• Pdf URL: https://arxiv.org/pdf/2411.14003
• Copy Paste: [[2411.14003]] Generative Intervention Models for Causal Perturbation Modeling(https://arxiv.org/abs/2411.14003)
• Keywords: robust, generative
• Abstract: We consider the problem of predicting perturbation effects via causal models. In many applications, it is a priori unknown which mechanisms of a system are modified by an external perturbation, even though the features of the perturbation are available. For example, in genomics, some properties of a drug may be known, but not their causal effects on the regulatory pathways of cells. We propose a generative intervention model (GIM) that learns to map these perturbation features to distributions over atomic interventions in a jointly-estimated causal model. Contrary to prior approaches, this enables us to predict the distribution shifts of unseen perturbation features while gaining insights about their mechanistic effects in the underlying data-generating process. On synthetic data and scRNA-seq drug perturbation data, GIMs achieve robust out-of-distribution predictions on par with unstructured approaches, while effectively inferring the underlying perturbation mechanisms, often better than other causal inference methods.

Title: RISecure-PUF: Multipurpose PUF-Driven Security Extensions with Lookaside Buffer in RISC-V

• Authors: Chenghao Chen, Xiaolin Zhang, Kailun Qin, Tengfei Wang, Yipeng Shi, Tianyi Huang, Chi Zhang, Dawu Gu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14025
• Pdf URL: https://arxiv.org/pdf/2411.14025
• Copy Paste: [[2411.14025]] RISecure-PUF: Multipurpose PUF-Driven Security Extensions with Lookaside Buffer in RISC-V(https://arxiv.org/abs/2411.14025)
• Keywords: secure, security, attack
• Abstract: RISC-V's limited security features hinder its use in confidential computing and heterogeneous platforms. This paper introduces RISecure-PUF, a security extension utilizing existing Physical Unclonable Functions for key generation and secure protocol purposes. A one-way hash function is integrated to ensure provable security against modeling attacks, while a lookaside buffer accelerates batch sampling and minimizes reliance on error correction codes. Implemented on the Genesys 2 FPGA, RISecure-PUF improves at least $2.72\times$ in batch scenarios with negligible hardware overhead and a maximum performance reduction of $10.7\%$, enabled by reusing the hash function module in integrated environments such as cryptographic engines.

Title: Relation-aware based Siamese Denoising Autoencoder for Malware Few-shot Classification

• Authors: Jinting Zhu, Julian Jang-Jaccard, Ian Welch, Harith AI-Sahaf, Seyit Camtepe, Aeryn Dunmore, Cybersecurity Lab
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14029
• Pdf URL: https://arxiv.org/pdf/2411.14029
• Copy Paste: [[2411.14029]] Relation-aware based Siamese Denoising Autoencoder for Malware Few-shot Classification(https://arxiv.org/abs/2411.14029)
• Keywords: security, attack, robust
• Abstract: When malware employs an unseen zero-day exploit, traditional security measures such as vulnerability scanners and antivirus software can fail to detect them. This is because these tools rely on known patches and signatures, which do not exist for new zero-day attacks. Furthermore, existing machine learning methods, which are trained on specific and occasionally outdated malware samples, may struggle to adapt to features in new malware. To address this issue, there is a need for a more robust machine learning model that can identify relationships between malware samples without being trained on a particular malware feature set. This is particularly crucial in the field of cybersecurity, where the number of malware samples is limited and obfuscation techniques are widely used. Current approaches using stacked autoencoders aim to remove the noise introduced by obfuscation techniques through reconstruction of the input. However, this approach ignores the semantic relationships between features across different malware samples. To overcome this limitation, we propose a novel Siamese Neural Network (SNN) that uses relation-aware embeddings to calculate more accurate similarity probabilities based on semantic details of different malware samples. In addition, by using entropy images as inputs, our model can extract better structural information and subtle differences in malware signatures, even in the presence of obfuscation techniques. Evaluations on two large malware sample sets using the N-shot and N-way methods show that our proposed model is highly effective in predicting previously unseen malware, even in the presence of obfuscation techniques.

Title: REFOL: Resource-Efficient Federated Online Learning for Traffic Flow Forecasting

• Authors: Qingxiang Liu, Sheng Sun, Yuxuan Liang, Xiaolong Xu, Min Liu, Muhammad Bilal, Yuwei Wang, Xujing Li, Yu Zheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14046
• Pdf URL: https://arxiv.org/pdf/2411.14046
• Copy Paste: [[2411.14046]] REFOL: Resource-Efficient Federated Online Learning for Traffic Flow Forecasting(https://arxiv.org/abs/2411.14046)
• Keywords: privacy, federate
• Abstract: Multiple federated learning (FL) methods are proposed for traffic flow forecasting (TFF) to avoid heavy-transmission and privacy-leaking concerns resulting from the disclosure of raw data in centralized methods. However, these FL methods adopt offline learning which may yield subpar performance, when concept drift occurs, i.e., distributions of historical and future data vary. Online learning can detect concept drift during model training, thus more applicable to TFF. Nevertheless, the existing federated online learning method for TFF fails to efficiently solve the concept drift problem and causes tremendous computing and communication overhead. Therefore, we propose a novel method named Resource-Efficient Federated Online Learning (REFOL) for TFF, which guarantees prediction performance in a communication-lightweight and computation-efficient way. Specifically, we design a data-driven client participation mechanism to detect the occurrence of concept drift and determine clients' participation necessity. Subsequently, we propose an adaptive online optimization strategy, which guarantees prediction performance and meanwhile avoids meaningless model updates. Then, a graph convolution-based model aggregation mechanism is designed, aiming to assess participants' contribution based on spatial correlation without importing extra communication and computing consumption on clients. Finally, we conduct extensive experiments on real-world datasets to demonstrate the superiority of REFOL in terms of prediction improvement and resource economization.

Title: Stereo Anything: Unifying Stereo Matching with Large-Scale Mixed Data

• Authors: Xianda Guo, Chenming Zhang, Youmin Zhang, Dujun Nie, Ruilin Wang, Wenzhao Zheng, Matteo Poggi, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14053
• Pdf URL: https://arxiv.org/pdf/2411.14053
• Copy Paste: [[2411.14053]] Stereo Anything: Unifying Stereo Matching with Large-Scale Mixed Data(https://arxiv.org/abs/2411.14053)
• Keywords: robust
• Abstract: Stereo matching has been a pivotal component in 3D vision, aiming to find corresponding points between pairs of stereo images to recover depth information. In this work, we introduce StereoAnything, a highly practical solution for robust stereo matching. Rather than focusing on a specialized model, our goal is to develop a versatile foundational model capable of handling stereo images across diverse environments. To this end, we scale up the dataset by collecting labeled stereo images and generating synthetic stereo pairs from unlabeled monocular images. To further enrich the model's ability to generalize across different conditions, we introduce a novel synthetic dataset that complements existing data by adding variability in baselines, camera angles, and scene types. We extensively evaluate the zero-shot capabilities of our model on five public datasets, showcasing its impressive ability to generalize to new, unseen data. Code will be available at \url{this https URL}.

Title: FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs

• Authors: Shinbok Lee, Gaeun Seo, Daniel Lee, Byeongil Ko, Sunghee Jung, Myeongcheol Shin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14054
• Pdf URL: https://arxiv.org/pdf/2411.14054
• Copy Paste: [[2411.14054]] FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs(https://arxiv.org/abs/2411.14054)
• Keywords: generative
• Abstract: This study investigates language models' generative capabilities in tool-use dialogs. We categorize the models' outputs in tool-use dialogs into four distinct types: Tool Call, Answer Completion, Slot Question, and Relevance Detection, which serve as aspects for evaluation. We introduce FunctionChat-Bench, comprising 700 evaluation items and automated assessment programs. Using this benchmark, we evaluate several language models that support function calling. Our findings indicate that while language models may exhibit high accuracy in single-turn Tool Call scenarios, this does not necessarily translate to superior generative performance in multi-turn environments. We argue that the capabilities required for function calling extend beyond generating tool call messages; they must also effectively generate conversational messages that engage the user.

Title: DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization

• Authors: Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Min Zhang, Zhaopeng Tu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14055
• Pdf URL: https://arxiv.org/pdf/2411.14055
• Copy Paste: [[2411.14055]] DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization(https://arxiv.org/abs/2411.14055)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance. To address this, we propose DRPruning, which incorporates distributionally robust optimization to restore balanced performance across domains, along with further improvements to enhance robustness. Experiments in monolingual and multilingual settings show that our method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. We further provide analysis demonstrating the robustness of our method towards various domains and distribution shifts. Furthermore, our method automatically determines optimal reference losses and data ratios, suggesting potential for broader applications. Our code is available at this https URL.

Title: MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective

• Authors: Hailang Huang, Yong Wang, Zixuan Huang, Huaqiu Li, Tongwen Huang, Xiangxiang Chu, Richong Zhang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14062
• Pdf URL: https://arxiv.org/pdf/2411.14062
• Copy Paste: [[2411.14062]] MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective(https://arxiv.org/abs/2411.14062)
• Keywords: generative
• Abstract: Large Multimodal Models (LMMs) have demonstrated remarkable capabilities. While existing benchmarks for evaluating LMMs mainly focus on image comprehension, few works evaluate them from the image generation perspective. To address this issue, we propose a straightforward automated evaluation pipeline. Specifically, this pipeline requires LMMs to generate an image-prompt from a given input image. Subsequently, it employs text-to-image generative models to create a new image based on these generated prompts. Finally, we evaluate the performance of LMMs by comparing the original image with the generated one. Furthermore, we introduce MMGenBench-Test, a comprehensive benchmark developed to evaluate LMMs across 13 distinct image patterns, and MMGenBench-Domain, targeting the performance evaluation of LMMs within the generative image domain. A thorough evaluation involving over 50 popular LMMs demonstrates the effectiveness and reliability in both the pipeline and benchmark. Our observations indicate that numerous LMMs excelling in existing benchmarks fail to adequately complete the basic tasks, related to image understanding and description. This finding highlights the substantial potential for performance improvement in current LMMs and suggests avenues for future model optimization. Concurrently, our pipeline facilitates the efficient assessment of LMMs performance across diverse domains by using solely image inputs.

Title: Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model

• Authors: Ege Kesim, Selahattin Serdar Helli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14064
• Pdf URL: https://arxiv.org/pdf/2411.14064
• Copy Paste: [[2411.14064]] Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model(https://arxiv.org/abs/2411.14064)
• Keywords: generative
• Abstract: Parameter efficient finetuning (PEFT) methods are widely used in LLMs and generative models in computer vision. Especially one can use multiple of these during inference to change the behavior of the base model. In this paper we investigated whether multiple LoRA adapters trained on computer vision tasks can be merged together and used during inference without loss in performance. By achieving this, multitask models can be created just by merging different LoRAs. Merging these will reduce inference time and it will not require any additional retraining. We have trained adapters on six different tasks and evaluated their performance when they are merged together. For comparison we used a model with a frozen backbone and finetuned its head. Our results show that even with simple merging techniques creating a multitask model by merging adapters is achievable by slightly loosing performance in some cases. In our experiments we merged up to three adapters together. Depending on the task and the similarity of the data adapters were trained on, merges can outperform head finetuning. We have observed that LoRAs trained with dissimilar datasets tend to perform better compared to model trained on similar datasets.

Title: Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

• Authors: Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, Dieuwke Hupkes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14103
• Pdf URL: https://arxiv.org/pdf/2411.14103
• Copy Paste: [[2411.14103]] Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models(https://arxiv.org/abs/2411.14103)
• Keywords: large language model
• Abstract: In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks. In this paper, we investigate if NLI tasks, that are rarely used for LLM evaluation, can still be informative for evaluating LLMs. Focusing on five different NLI benchmarks across six models of different scales, we investigate if they are able to discriminate models of different size and quality and how their accuracies develop during training. Furthermore, we investigate the extent to which the softmax distributions of models align with human distributions in cases where statements are ambiguous or vague. Overall, our results paint a positive picture for the NLI tasks: we find that they are able to discriminate well between models at various stages of training, yet are not (all) saturated. Furthermore, we find that while the similarity of model distributions with human label distributions increases with scale, it is still much higher than the similarity between two populations of humans, making it a potentially interesting statistic to consider.

Title: RAG-Thief: Scalable Extraction of Private Data from Retrieval-Augmented Generation Applications with Agent-based Attacks

• Authors: Changyue Jiang, Xudong Pan, Geng Hong, Chenfu Bao, Min Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14110
• Pdf URL: https://arxiv.org/pdf/2411.14110
• Copy Paste: [[2411.14110]] RAG-Thief: Scalable Extraction of Private Data from Retrieval-Augmented Generation Applications with Agent-based Attacks(https://arxiv.org/abs/2411.14110)
• Keywords: privacy, attack, extraction, generative, large language model
• Abstract: While large language models (LLMs) have achieved notable success in generative tasks, they still face limitations, such as lacking up-to-date knowledge and producing hallucinations. Retrieval-Augmented Generation (RAG) enhances LLM performance by integrating external knowledge bases, providing additional context which significantly improves accuracy and knowledge coverage. However, building these external knowledge bases often requires substantial resources and may involve sensitive information. In this paper, we propose an agent-based automated privacy attack called RAG-Thief, which can extract a scalable amount of private data from the private database used in RAG applications. We conduct a systematic study on the privacy risks associated with RAG applications, revealing that the vulnerability of LLMs makes the private knowledge bases suffer significant privacy risks. Unlike previous manual attacks which rely on traditional prompt injection techniques, RAG-Thief starts with an initial adversarial query and learns from model responses, progressively generating new queries to extract as many chunks from the knowledge base as possible. Experimental results show that our RAG-Thief can extract over 70% information from the private knowledge bases within customized RAG applications deployed on local machines and real-world platforms, including OpenAI's GPTs and ByteDance's Coze. Our findings highlight the privacy vulnerabilities in current RAG applications and underscore the pressing need for stronger safeguards.

Title: Point Cloud Resampling with Learnable Heat Diffusion

• Authors: Wenqiang Xu, Wenrui Dai, Duoduo Xue, Ziyang Zheng, Chenglin Li, Junni Zou, Hongkai Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14120
• Pdf URL: https://arxiv.org/pdf/2411.14120
• Copy Paste: [[2411.14120]] Point Cloud Resampling with Learnable Heat Diffusion(https://arxiv.org/abs/2411.14120)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models have shown empirical successes in point cloud resampling, generating a denser and more uniform distribution of points from sparse or noisy 3D point clouds by progressively refining noise into structure. However, existing diffusion models employ manually predefined schemes, which often fail to recover the underlying point cloud structure due to the rigid and disruptive nature of the geometric degradation. To address this issue, we propose a novel learnable heat diffusion framework for point cloud resampling, which directly parameterizes the marginal distribution for the forward process by learning the adaptive heat diffusion schedules and local filtering scales of the time-varying heat kernel, and consequently, generates an adaptive conditional prior for the reverse process. Unlike previous diffusion models with a fixed prior, the adaptive conditional prior selectively preserves geometric features of the point cloud by minimizing a refined variational lower bound, guiding the points to evolve towards the underlying surface during the reverse process. Extensive experimental results demonstrate that the proposed point cloud resampling achieves state-of-the-art performance in representative reconstruction tasks including point cloud denoising and upsampling.

Title: Learning from "Silly" Questions Improves Large Language Models, But Only Slightly

• Authors: Tingyuan Zhu, Shudong Liu, Yidong Wang, Derek F. Wong, Han Yu, Takahiro Shinozaki, Jindong Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14121
• Pdf URL: https://arxiv.org/pdf/2411.14121
• Copy Paste: [[2411.14121]] Learning from "Silly" Questions Improves Large Language Models, But Only Slightly(https://arxiv.org/abs/2411.14121)
• Keywords: large language model
• Abstract: Constructing high-quality Supervised Fine-Tuning (SFT) datasets is critical for the training of large language models (LLMs). Recent studies have shown that using data from a specific source, Ruozhiba, a Chinese website where users ask "silly" questions to better understand certain topics, can lead to better fine-tuning performance. This paper aims to explore some hidden factors: the potential interpretations of its success and a large-scale evaluation of the performance. First, we leverage GPT-4 to analyze the successful cases of Ruozhiba questions from the perspective of education, psychology, and cognitive science, deriving a set of explanatory rules. Then, we construct fine-tuning datasets by applying these rules to the MMLU training set. Surprisingly, our results indicate that rules can significantly improve model performance in certain tasks, while potentially diminishing performance on others. For example, SFT data generated following the "Counterintuitive Thinking" rule can achieve approximately a 5% improvement on the "Global Facts" task, whereas the "Blurring the Conceptual Boundaries" rule leads to a performance drop of 6.14% on the "Econometrics" task. In addition, for specific tasks, different rules tend to have a consistent impact on model performance. This suggests that the differences between the extracted rules are not as significant, and the effectiveness of the rules is relatively consistent across tasks. Our research highlights the importance of considering task diversity and rule applicability when constructing SFT datasets to achieve more comprehensive performance improvements.

Title: RestorerID: Towards Tuning-Free Face Restoration with ID Preservation

• Authors: Jiacheng Ying, Mushui Liu, Zhe Wu, Runming Zhang, Zhu Yu, Siming Fu, Si-Yuan Cao, Chao Wu, Yunlong Yu, Hui-Liang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14125
• Pdf URL: https://arxiv.org/pdf/2411.14125
• Copy Paste: [[2411.14125]] RestorerID: Towards Tuning-Free Face Restoration with ID Preservation(https://arxiv.org/abs/2411.14125)
• Keywords: diffusion
• Abstract: Blind face restoration has made great progress in producing high-quality and lifelike images. Yet it remains challenging to preserve the ID information especially when the degradation is heavy. Current reference-guided face restoration approaches either require face alignment or personalized test-tuning, which are unfaithful or time-consuming. In this paper, we propose a tuning-free method named RestorerID that incorporates ID preservation during face restoration. RestorerID is a diffusion model-based method that restores low-quality images with varying levels of degradation by using a single reference image. To achieve this, we propose a unified framework to combine the ID injection with the base blind face restoration model. In addition, we design a novel Face ID Rebalancing Adapter (FIR-Adapter) to tackle the problems of content unconsistency and contours misalignment that are caused by information conflicts between the low-quality input and reference image. Furthermore, by employing an Adaptive ID-Scale Adjusting strategy, RestorerID can produce superior restored images across various levels of degradation. Experimental results on the Celeb-Ref dataset and real-world scenarios demonstrate that RestorerID effectively delivers high-quality face restoration with ID preservation, achieving a superior performance compared to the test-tuning approaches and other reference-guided ones. The code of RestorerID is available at \url{this https URL}.

Title: GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs

• Authors: Advik Raj Basani, Xiao Zhang
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14133
• Pdf URL: https://arxiv.org/pdf/2411.14133
• Copy Paste: [[2411.14133]] GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs(https://arxiv.org/abs/2411.14133)
• Keywords: attack, generative, large language model
• Abstract: Large Language Models (LLMs) have shown impressive proficiency across a range of natural language processing tasks yet remain vulnerable to adversarial prompts, known as jailbreak attacks, carefully designed to elicit harmful responses from LLMs. Traditional methods rely on manual heuristics, which suffer from limited generalizability. While being automatic, optimization-based attacks often produce unnatural jailbreak prompts that are easy to detect by safety filters or require high computational overhead due to discrete token optimization. Witnessing the limitations of existing jailbreak methods, we introduce Generative Adversarial Suffix Prompter (GASP), a novel framework that combines human-readable prompt generation with Latent Bayesian Optimization (LBO) to improve adversarial suffix creation in a fully black-box setting. GASP leverages LBO to craft adversarial suffixes by efficiently exploring continuous embedding spaces, gradually optimizing the model to improve attack efficacy while balancing prompt coherence through a targeted iterative refinement procedure. Our experiments show that GASP can generate natural jailbreak prompts, significantly improving attack success rates, reducing training times, and accelerating inference speed, thus making it an efficient and scalable solution for red-teaming LLMs.

Title: Point Cloud Denoising With Fine-Granularity Dynamic Graph Convolutional Networks

• Authors: Wenqiang Xu, Wenrui Dai, Duoduo Xue, Ziyang Zheng, Chenglin Li, Junni Zou, Hongkai Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14158
• Pdf URL: https://arxiv.org/pdf/2411.14158
• Copy Paste: [[2411.14158]] Point Cloud Denoising With Fine-Granularity Dynamic Graph Convolutional Networks(https://arxiv.org/abs/2411.14158)
• Keywords: robust
• Abstract: Due to limitations in acquisition equipment, noise perturbations often corrupt 3-D point clouds, hindering down-stream tasks such as surface reconstruction, rendering, and further processing. Existing 3-D point cloud denoising methods typically fail to reliably fit the underlying continuous surface, resulting in a degradation of reconstruction performance. This paper introduces fine-granularity dynamic graph convolutional networks called GD-GCN, a novel approach to denoising in 3-D point clouds. The GD-GCN employs micro-step temporal graph convolution (MST-GConv) to perform feature learning in a gradual manner. Compared with the conventional GCN, which commonly uses discrete integer-step graph convolution, this modification introduces a more adaptable and nuanced approach to feature learning within graph convolution networks. It more accurately depicts the process of fitting the point cloud with noise to the underlying surface by and the learning process for MST-GConv acts like a changing system and is managed through a type of neural network known as neural Partial Differential Equations (PDEs). This means it can adapt and improve over time. GD-GCN approximates the Riemannian metric, calculating distances between points along a low-dimensional manifold. This capability allows it to understand the local geometric structure and effectively capture diverse relationships between points from different geometric regions through geometric graph construction based on Riemannian distances. Additionally, GD-GCN incorporates robust graph spectral filters based on the Bernstein polynomial approximation, which modulate eigenvalues for complex and arbitrary spectral responses, providing theoretical guarantees for BIBO stability. Symmetric channel mixing matrices further enhance filter flexibility by enabling channel-level scaling and shifting in the spectral domain.

Title: FoPru: Focal Pruning for Efficient Large Vision-Language Models

• Authors: Lei Jiang, Weizhe Huang, Tongxuan Liu, Yuting Zeng, Jing Li, Lechao Cheng, Xiaohua Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14164
• Pdf URL: https://arxiv.org/pdf/2411.14164
• Copy Paste: [[2411.14164]] FoPru: Focal Pruning for Efficient Large Vision-Language Models(https://arxiv.org/abs/2411.14164)
• Keywords: large language model
• Abstract: Large Vision-Language Models (LVLMs) represent a significant advancement toward achieving superior multimodal capabilities by enabling powerful Large Language Models (LLMs) to understand visual input. Typically, LVLMs utilize visual encoders, such as CLIP, to transform images into visual tokens, which are then aligned with textual tokens through projection layers before being input into the LLM for inference. Although existing LVLMs have achieved significant success, their inference efficiency is still limited by the substantial number of visual tokens and the potential redundancy among them. To mitigate this issue, we propose Focal Pruning (FoPru), a training-free method that prunes visual tokens based on the attention-based token significance derived from the vision encoder. Specifically, we introduce two alternative pruning strategies: 1) the rank strategy, which leverages all token significance scores to retain more critical tokens in a global view; 2) the row strategy, which focuses on preserving continuous key information in images from a local perspective. Finally, the selected tokens are reordered to maintain their original positional relationships. Extensive experiments across various LVLMs and multimodal datasets demonstrate that our method can prune a large number of redundant tokens while maintaining high accuracy, leading to significant improvements in inference efficiency.

Title: Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting

• Authors: Jingyi Xu, Xieyuanli Chen, Junyi Ma, Jiawei Huang, Jintao Xu, Yue Wang, Ling Pei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14169
• Pdf URL: https://arxiv.org/pdf/2411.14169
• Copy Paste: [[2411.14169]] Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting(https://arxiv.org/abs/2411.14169)
• Keywords: robust
• Abstract: The task of occupancy forecasting (OCF) involves utilizing past and present perception data to predict future occupancy states of autonomous vehicle surrounding environments, which is critical for downstream tasks such as obstacle avoidance and path planning. Existing 3D OCF approaches struggle to predict plausible spatial details for movable objects and suffer from slow inference speeds due to neglecting the bias and uneven distribution of changing occupancy states in both space and time. In this paper, we propose a novel spatiotemporal decoupling vision-based paradigm to explicitly tackle the bias and achieve both effective and efficient 3D OCF. To tackle spatial bias in empty areas, we introduce a novel spatial representation that decouples the conventional dense 3D format into 2D bird's-eye view (BEV) occupancy with corresponding height values, enabling 3D OCF derived only from 2D predictions thus enhancing efficiency. To reduce temporal bias on static voxels, we design temporal decoupling to improve end-to-end OCF by temporally associating instances via predicted flows. We develop an efficient multi-head network EfficientOCF to achieve 3D OCF with our devised spatiotemporally decoupled representation. A new metric, conditional IoU (C-IoU), is also introduced to provide a robust 3D OCF performance assessment, especially in datasets with missing or incomplete annotations. The experimental results demonstrate that EfficientOCF surpasses existing baseline methods on accuracy and efficiency, achieving state-of-the-art performance with a fast inference time of 82.33ms with a single GPU. Our code will be released as open source.

Title: CompetitorFormer: Competitor Transformer for 3D Instance Segmentation

• Authors: Duanchu Wang (1), Jing Liu (2), Haoran Gong (2), Yinghui Quan (1), Di Wang (2) ((1) School of Electronic Engineering, Xidian University (2) School of Software Engineering, Xian Jiaotong University)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14179
• Pdf URL: https://arxiv.org/pdf/2411.14179
• Copy Paste: [[2411.14179]] CompetitorFormer: Competitor Transformer for 3D Instance Segmentation(https://arxiv.org/abs/2411.14179)
• Keywords: transformer, segmentation
• Abstract: Transformer-based methods have become the dominant approach for 3D instance segmentation. These methods predict instance masks via instance queries, ranking them by classification confidence and IoU scores to select the top prediction as the final outcome. However, it has been observed that the current models employ a fixed and higher number of queries than the instances present within a scene. In such instances, multiple queries predict the same instance, yet only a single query is ultimately optimized. The close scores of queries in the lower-level decoders make it challenging for the dominant query to distinguish itself rapidly, which ultimately impairs the model's accuracy and convergence efficiency. This phenomenon is referred to as inter-query competition. To address this challenge, we put forth a series of plug-and-play competition-oriented designs, collectively designated as the CompetitorFormer, with the aim of reducing competition and facilitating a dominant query. Experiments showed that integrating our designs with state-of-the-art frameworks consistently resulted in significant performance improvements in 3D instance segmentation across a range of datasets.

Title: OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

• Authors: Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, Hannaneh Hajishirzi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14199
• Pdf URL: https://arxiv.org/pdf/2411.14199
• Copy Paste: [[2411.14199]] OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs(https://arxiv.org/abs/2411.14199)
• Keywords: large language model
• Abstract: Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.

Title: Novel View Extrapolation with Video Diffusion Priors

• Authors: Kunhao Liu, Ling Shao, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14208
• Pdf URL: https://arxiv.org/pdf/2411.14208
• Copy Paste: [[2411.14208]] Novel View Extrapolation with Video Diffusion Priors(https://arxiv.org/abs/2411.14208)
• Keywords: diffusion, generative
• Abstract: The field of novel view synthesis has made significant strides thanks to the development of radiance field methods. However, most radiance field techniques are far better at novel view interpolation than novel view extrapolation where the synthesis novel views are far beyond the observed training views. We design ViewExtrapolator, a novel view synthesis approach that leverages the generative priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation. By redesigning the SVD denoising process, ViewExtrapolator refines the artifact-prone views rendered by radiance fields, greatly enhancing the clarity and realism of the synthesized novel views. ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available. Additionally, ViewExtrapolator requires no fine-tuning of SVD, making it both data-efficient and computation-efficient. Extensive experiments demonstrate the superiority of ViewExtrapolator in novel view extrapolation. Project page: \url{this https URL}.

Title: Generative Outpainting To Enhance the Memorability of Short-Form Videos

• Authors: Alan Byju, Aman Sudhindra Ladwa, Lorin Sweeney, Alan F. Smeaton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14213
• Pdf URL: https://arxiv.org/pdf/2411.14213
• Copy Paste: [[2411.14213]] Generative Outpainting To Enhance the Memorability of Short-Form Videos(https://arxiv.org/abs/2411.14213)
• Keywords: generative
• Abstract: With the expanding use of the short-form video format in advertising, social media, entertainment, education and more, there is a need for such media to both captivate and be remembered. Video memorability indicates to us how likely a video is to be remembered by a viewer who has no emotional or personal connection with its content. This paper presents the results of using generative outpainting to expand the screen size of a short-form video with a view to improving its memorability. Advances in machine learning and deep learning are compared and leveraged to understand how extending the borders of video screensizes can affect their memorability to viewers. Using quantitative evaluation we determine the best-performing model for outpainting and the impact of outpainting based on image saliency on video memorability scores

Title: Evaluating the Robustness of Analogical Reasoning in Large Language Models

• Authors: Martha Lewis, Melanie Mitchell
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14215
• Pdf URL: https://arxiv.org/pdf/2411.14215
• Copy Paste: [[2411.14215]] Evaluating the Robustness of Analogical Reasoning in Large Language Models(https://arxiv.org/abs/2411.14215)
• Keywords: robust, large language model
• Abstract: LLMs have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, there is debate on the extent to which they are performing general abstract reasoning versus employing non-robust processes, e.g., that overly rely on similarity to pre-training data. Here we investigate the robustness of analogy-making abilities previously claimed for LLMs on three of four domains studied by Webb, Holyoak, and Lu (2023): letter-string analogies, digit matrices, and story analogies. For each domain we test humans and GPT models on robustness to variants of the original analogy problems that test the same abstract reasoning abilities but are likely dissimilar from tasks in the pre-training data. The performance of a system that uses robust abstract reasoning should not decline substantially on these variants. On simple letter-string analogies, we find that while the performance of humans remains high for two types of variants we tested, the GPT models' performance declines sharply. This pattern is less pronounced as the complexity of these problems is increased, as both humans and GPT models perform poorly on both the original and variant problems requiring more complex analogies. On digit-matrix problems, we find a similar pattern but only on one out of the two types of variants we tested. On story-based analogy problems, we find that, unlike humans, the performance of GPT models are susceptible to answer-order effects, and that GPT models also may be more sensitive than humans to paraphrasing. This work provides evidence that LLMs often lack the robustness of zero-shot human analogy-making, exhibiting brittleness on most of the variations we tested. More generally, this work points to the importance of carefully evaluating AI systems not only for accuracy but also robustness when testing their cognitive capabilities.

Title: FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression

• Authors: Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, Sheng Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14228
• Pdf URL: https://arxiv.org/pdf/2411.14228
• Copy Paste: [[2411.14228]] FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression(https://arxiv.org/abs/2411.14228)
• Keywords: large language model
• Abstract: Recent advances on Multi-modal Large Language Models have demonstrated that high-resolution image input is crucial for model capabilities, especially for fine-grained tasks. However, high-resolution images lead to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We argue that removing visual redundancy can simultaneously improve both efficiency and performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user this http URL these two modules, the proposed FocusLLaVA achieves improvements in both efficiency and performance. We validate the effectiveness of our approach on a wide range of evaluation datasets.

Title: AnywhereDoor: Multi-Target Backdoor Attacks on Object Detection

• Authors: Jialin Lu, Junjie Shan, Ziqi Zhao, Ka-Ho Chow
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14243
• Pdf URL: https://arxiv.org/pdf/2411.14243
• Copy Paste: [[2411.14243]] AnywhereDoor: Multi-Target Backdoor Attacks on Object Detection(https://arxiv.org/abs/2411.14243)
• Keywords: attack, robust
• Abstract: As object detection becomes integral to many safety-critical applications, understanding its vulnerabilities is essential. Backdoor attacks, in particular, pose a significant threat by implanting hidden backdoor in a victim model, which adversaries can later exploit to trigger malicious behaviors during inference. However, current backdoor techniques are limited to static scenarios where attackers must define a malicious objective before training, locking the attack into a predetermined action without inference-time adaptability. Given the expressive output space in object detection, including object existence detection, bounding box estimation, and object classification, the feasibility of implanting a backdoor that provides inference-time control with a high degree of freedom remains unexplored. This paper introduces AnywhereDoor, a flexible backdoor attack tailored for object detection. Once implanted, AnywhereDoor enables adversaries to specify different attack types (object vanishing, fabrication, or misclassification) and configurations (untargeted or targeted with specific classes) to dynamically control detection behavior. This flexibility is achieved through three key innovations: (i) objective disentanglement to support a broader range of attack combinations well beyond what existing methods allow; (ii) trigger mosaicking to ensure backdoor activations are robust, even against those object detectors that extract localized regions from the input image for recognition; and (iii) strategic batching to address object-level data imbalances that otherwise hinders a balanced manipulation. Extensive experiments demonstrate that AnywhereDoor provides attackers with a high degree of control, achieving an attack success rate improvement of nearly 80% compared to adaptations of existing methods for such flexible control.

Title: Natural Language Reinforcement Learning

• Authors: Xidong Feng, Ziyu Wan, Haotian Fu, Bo Liu, Mengyue Yang, Girish A. Koushik, Zhiyuan Hu, Ying Wen, Jun Wang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14251
• Pdf URL: https://arxiv.org/pdf/2411.14251
• Copy Paste: [[2411.14251]] Natural Language Reinforcement Learning(https://arxiv.org/abs/2411.14251)
• Keywords: interpretability, large language model
• Abstract: Reinforcement Learning (RL) mathematically formulates decision-making with Markov Decision Process (MDP). With MDPs, researchers have achieved remarkable breakthroughs across various domains, including games, robotics, and language models. This paper seeks a new possibility, Natural Language Reinforcement Learning (NLRL), by extending traditional MDP to natural language-based representation space. Specifically, NLRL innovatively redefines RL principles, including task objectives, policy, value function, Bellman equation, and policy iteration, into their language counterparts. With recent advancements in large language models (LLMs), NLRL can be practically implemented to achieve RL-like policy and value improvement by either pure prompting or gradient-based training. Experiments over Maze, Breakthrough, and Tic-Tac-Toe games demonstrate the effectiveness, efficiency, and interpretability of the NLRL framework among diverse use cases. Our code will be released at this https URL.

Title: Intent-Aware Dialogue Generation and Multi-Task Contrastive Learning for Multi-Turn Intent Classification

• Authors: Junhua Liu, Yong Keat Tan, Bin Fu, Kwan Hui Lim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14252
• Pdf URL: https://arxiv.org/pdf/2411.14252
• Copy Paste: [[2411.14252]] Intent-Aware Dialogue Generation and Multi-Task Contrastive Learning for Multi-Turn Intent Classification(https://arxiv.org/abs/2411.14252)
• Keywords: large language model
• Abstract: Generating large-scale, domain-specific, multilingual multi-turn dialogue datasets remains a significant hurdle for training effective Multi-Turn Intent Classification models in chatbot systems. In this paper, we introduce Chain-of-Intent, a novel mechanism that combines Hidden Markov Models with Large Language Models (LLMs) to generate contextually aware, intent-driven conversations through self-play. By extracting domain-specific knowledge from e-commerce chat logs, we estimate conversation turns and intent transitions, which guide the generation of coherent dialogues. Leveraging LLMs to enhance emission probabilities, our approach produces natural and contextually consistent questions and answers. We also propose MINT-CL, a framework for multi-turn intent classification using multi-task contrastive learning, improving classification accuracy without the need for extensive annotated data. Evaluations show that our methods outperform baselines in dialogue quality and intent classification accuracy, especially in multilingual settings, while significantly reducing data generation efforts. Furthermore, we release MINT-E, a multilingual, intent-aware multi-turn e-commerce dialogue corpus to support future research in this area.

Title: BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI

• Authors: Natenaile Asmamaw Shiferaw, Simpenzwe Honore Leandre, Aman Sinha, Dillip Rout
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14254
• Pdf URL: https://arxiv.org/pdf/2411.14254
• Copy Paste: [[2411.14254]] BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI(https://arxiv.org/abs/2411.14254)
• Keywords: interpretability
• Abstract: Course Outcome (CO) and Program Outcome (PO)/Program-Specific Outcome (PSO) alignment is a crucial task for ensuring curriculum coherence and assessing educational effectiveness. The construction of a Course Articulation Matrix (CAM), which quantifies the relationship between COs and POs/PSOs, typically involves assigning numerical values (0, 1, 2, 3) to represent the degree of alignment. In this study, We experiment with four models from the BERT family: BERT Base, DistilBERT, ALBERT, and RoBERTa, and use multiclass classification to assess the alignment between CO and PO/PSO pairs. We first evaluate traditional machine learning classifiers, such as Decision Tree, Random Forest, and XGBoost, and then apply transfer learning to evaluate the performance of the pretrained BERT models. To enhance model interpretability, we apply Explainable AI technique, specifically Local Interpretable Model-agnostic Explanations (LIME), to provide transparency into the decision-making process. Our system achieves accuracy, precision, recall, and F1-score values of 98.66%, 98.67%, 98.66%, and 98.66%, respectively. This work demonstrates the potential of utilizing transfer learning with BERT-based models for the automated generation of CAMs, offering high performance and interpretability in educational outcome assessment.

Title: Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models

• Authors: Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, Neel Nanda
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14257
• Pdf URL: https://arxiv.org/pdf/2411.14257
• Copy Paste: [[2411.14257]] Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models(https://arxiv.org/abs/2411.14257)
• Keywords: interpretability, large language model
• Abstract: Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem. Using sparse autoencoders as an interpretability tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is one it can recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting it doesn't know about an athlete or a movie. This suggests that models can have self-knowledge: internal representations about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggesting that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions in the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token.

Title: Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective

• Authors: Ernests Lavrinovics, Russa Biswas, Johannes Bjerva, Katja Hose
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14258
• Pdf URL: https://arxiv.org/pdf/2411.14258
• Copy Paste: [[2411.14258]] Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective(https://arxiv.org/abs/2411.14258)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) based applications including automated text generation, question answering, chatbots, and others. However, they face a significant challenge: hallucinations, where models produce plausible-sounding but factually incorrect responses. This undermines trust and limits the applicability of LLMs in different domains. Knowledge Graphs (KGs), on the other hand, provide a structured collection of interconnected facts represented as entities (nodes) and their relationships (edges). In recent research, KGs have been leveraged to provide context that can fill gaps in an LLM understanding of certain topics offering a promising approach to mitigate hallucinations in LLMs, enhancing their reliability and accuracy while benefiting from their wide applicability. Nonetheless, it is still a very active area of research with various unresolved open problems. In this paper, we discuss these open challenges covering state-of-the-art datasets and benchmarks as well as methods for knowledge integration and evaluating hallucinations. In our discussion, we consider the current use of KGs in LLM systems and identify future directions within each of these challenges.

Title: Generating Realistic Adversarial Examples for Business Processes using Variational Autoencoders

• Authors: Alexander Stevens, Jari Peeperkorn, Johannes De Smedt, Jochen De Weerdt
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14263
• Pdf URL: https://arxiv.org/pdf/2411.14263
• Copy Paste: [[2411.14263]] Generating Realistic Adversarial Examples for Business Processes using Variational Autoencoders(https://arxiv.org/abs/2411.14263)
• Keywords: attack
• Abstract: In predictive process monitoring, predictive models are vulnerable to adversarial attacks, where input perturbations can lead to incorrect predictions. Unlike in computer vision, where these perturbations are designed to be imperceptible to the human eye, the generation of adversarial examples in predictive process monitoring poses unique challenges. Minor changes to the activity sequences can create improbable or even impossible scenarios to occur due to underlying constraints such as regulatory rules or process constraints. To address this, we focus on generating realistic adversarial examples tailored to the business process context, in contrast to the imperceptible, pixel-level changes commonly seen in computer vision adversarial attacks. This paper introduces two novel latent space attacks, which generate adversaries by adding noise to the latent space representation of the input data, rather than directly modifying the input attributes. These latent space methods are domain-agnostic and do not rely on process-specific knowledge, as we restrict the generation of adversarial examples to the learned class-specific data distributions by directly perturbing the latent space representation of the business process executions. We evaluate these two latent space methods with six other adversarial attacking methods on eleven real-life event logs and four predictive models. The first three attacking methods directly permute the activities of the historically observed business process executions. The fourth method constrains the adversarial examples to lie within the same data distribution as the original instances, by projecting the adversarial examples to the original data distribution.

Title: Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models

• Authors: Iacopo Ghinassi, Leonardo Catalano, Tommaso Colella
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14272
• Pdf URL: https://arxiv.org/pdf/2411.14272
• Copy Paste: [[2411.14272]] Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models(https://arxiv.org/abs/2411.14272)
• Keywords: generative, large language model
• Abstract: The use of Natural Language Processing (NLP) for helping decision-makers with Climate Change action has recently been highlighted as a use case aligning with a broader drive towards NLP technologies for social good. In this context, Aspect-Based Summarization (ABS) systems that extract and summarize relevant information are particularly useful as they provide stakeholders with a convenient way of finding relevant information in expert-curated reports. In this work, we release a new dataset for ABS of Climate Change reports and we employ different Large Language Models (LLMs) and so-called Small Language Models (SLMs) to tackle this problem in an unsupervised way. Considering the problem at hand, we also show how SLMs are not significantly worse for the problem while leading to reduced carbon footprint; we do so by applying for the first time an existing framework considering both energy efficiency and task performance to the evaluation of zero-shot generative models for ABS. Overall, our results show that modern language models, both big and small, can effectively tackle ABS for Climate Change reports but more research is needed when we frame the problem as a Retrieval Augmented Generation (RAG) problem and our work and dataset will help foster efforts in this direction.

Title: Adaptive Anomaly Detection for Identifying Attacks in Cyber-Physical Systems: A Systematic Literature Review

• Authors: Pablo Moriano, Steven C. Hespeler, Mingyan Li, Maria Mahbub
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14278
• Pdf URL: https://arxiv.org/pdf/2411.14278
• Copy Paste: [[2411.14278]] Adaptive Anomaly Detection for Identifying Attacks in Cyber-Physical Systems: A Systematic Literature Review(https://arxiv.org/abs/2411.14278)
• Keywords: attack
• Abstract: Modern cyberattacks in cyber-physical systems (CPS) rapidly evolve and cannot be deterred effectively with most current methods which focused on characterizing past threats. Adaptive anomaly detection (AAD) is among the most promising techniques to detect evolving cyberattacks focused on fast data processing and model adaptation. AAD has been researched in the literature extensively; however, to the best of our knowledge, our work is the first systematic literature review (SLR) on the current research within this field. We present a comprehensive SLR, gathering 397 relevant papers and systematically analyzing 65 of them (47 research and 18 survey papers) on AAD in CPS studies from 2013 to 2023 (November). We introduce a novel taxonomy considering attack types, CPS application, learning paradigm, data management, and algorithms. Our analysis indicates, among other findings, that reviewed works focused on a single aspect of adaptation (either data processing or model adaptation) but rarely in both at the same time. We aim to help researchers to advance the state of the art and help practitioners to become familiar with recent progress in this field. We identify the limitations of the state of the art and provide recommendations for future research directions.

Title: EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild

• Authors: Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Marc Habermann, Christian Theobalt, Yuexin Ma, Wenping Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14280
• Pdf URL: https://arxiv.org/pdf/2411.14280
• Copy Paste: [[2411.14280]] EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild(https://arxiv.org/abs/2411.14280)
• Keywords: robust, segmentation
• Abstract: Our work aims to reconstruct hand-object interactions from a single-view image, which is a fundamental but ill-posed task. Unlike methods that reconstruct from videos, multi-view images, or predefined 3D templates, single-view reconstruction faces significant challenges due to inherent ambiguities and occlusions. These challenges are further amplified by the diverse nature of hand poses and the vast variety of object shapes and sizes. Our key insight is that current foundational models for segmentation, inpainting, and 3D reconstruction robustly generalize to in-the-wild images, which could provide strong visual and geometric priors for reconstructing hand-object interactions. Specifically, given a single image, we first design a novel pipeline to estimate the underlying hand pose and object shape using off-the-shelf large models. Furthermore, with the initial reconstruction, we employ a prior-guided optimization scheme, which optimizes hand pose to comply with 3D physical constraints and the 2D input image content. We perform experiments across several datasets and show that our method consistently outperforms baselines and faithfully reconstructs a diverse set of hand-object interactions. Here is the link of our project page: this https URL

Title: StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart

• Authors: Jian Shi, Qian Wang, Zhenyu Li, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14295
• Pdf URL: https://arxiv.org/pdf/2411.14295
• Copy Paste: [[2411.14295]] StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart(https://arxiv.org/abs/2411.14295)
• Keywords: robust, diffusion
• Abstract: Generating high-quality stereo videos that mimic human binocular vision requires maintaining consistent depth perception and temporal coherence across frames. While diffusion models have advanced image and video synthesis, generating high-quality stereo videos remains challenging due to the difficulty of maintaining consistent temporal and spatial coherence between left and right views. We introduce \textit{StereoCrafter-Zero}, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without the need for paired training data. Key innovations include a noisy restart strategy to initialize stereo-aware latents and an iterative refinement process that progressively harmonizes the latent space, addressing issues like temporal flickering and view inconsistencies. Comprehensive evaluations, including quantitative metrics and user studies, demonstrate that \textit{StereoCrafter-Zero} produces high-quality stereo videos with improved depth consistency and temporal smoothness, even when depth estimations are imperfect. Our framework is robust and adaptable across various diffusion models, setting a new benchmark for zero-shot stereo video generation and enabling more immersive visual experiences. Our code can be found in~\url{this https URL}.

Title: Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training

• Authors: Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Chen Qi, Peng Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14318
• Pdf URL: https://arxiv.org/pdf/2411.14318
• Copy Paste: [[2411.14318]] Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training(https://arxiv.org/abs/2411.14318)
• Keywords: large language model
• Abstract: It is well-known that a diverse corpus is critical for training large language models, which are typically constructed from a mixture of various domains. In general, previous efforts resort to sampling training data from different domains with static proportions, as well as adjusting data proportions during training. However, few methods have addressed the complexities of domain-adaptive continual pre-training. To fill this gap, we propose Velocitune, a novel framework dynamically assesses learning velocity and adjusts data proportions accordingly, favoring slower-learning domains while shunning faster-learning ones, which is guided by a scaling law to indicate the desired learning goal for each domain with less associated cost. To evaluate the effectiveness of Velocitune, we conduct experiments in a reasoning-focused dataset with CodeLlama, as well as in a corpus specialised for system command generation with Llama3 and Mistral. Velocitune achieves performance gains in both math and code reasoning tasks and command-line generation benchmarks. Further analysis reveals that key factors driving Velocitune's effectiveness include target loss prediction and data ordering.

Title: UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages

• Authors: Bethel Melesse Tessema (1), Akhil Kedia (2), Tae-Sun Chung (1) ((1) Ajou University, (2) Independent Researcher)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14343
• Pdf URL: https://arxiv.org/pdf/2411.14343
• Copy Paste: [[2411.14343]] UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages(https://arxiv.org/abs/2411.14343)
• Keywords: large language model
• Abstract: Large language models (LLMs) under-perform on low-resource languages due to limited training data. We present a method to efficiently collect text data for low-resource languages from the entire Common Crawl corpus. Our approach, UnifiedCrawl, filters and extracts common crawl using minimal compute resources, yielding mono-lingual datasets much larger than previously available sources. We demonstrate that leveraging this data to fine-tuning multilingual LLMs via efficient adapter methods (QLoRA) significantly boosts performance on the low-resource language, while minimizing VRAM usage. Our experiments show large improvements in language modeling perplexity and an increase in few-shot prompting scores. Our work and released source code provide an affordable approach to improve LLMs for low-resource languages using consumer hardware. Our source code is available here at this https URL.

Title: Layer Pruning with Consensus: A Triple-Win Solution

• Authors: Leandro Giusti Mugnaini, Carolina Tavares Duarte, Anna H. Reali Costa, Artur Jordao
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14345
• Pdf URL: https://arxiv.org/pdf/2411.14345
• Copy Paste: [[2411.14345]] Layer Pruning with Consensus: A Triple-Win Solution(https://arxiv.org/abs/2411.14345)
• Keywords: attack, robust
• Abstract: Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics into a single expressive measure of low-importance layers, called the Consensus criterion. Our technique delivers a triple-win solution: low accuracy drop, high-performance improvement, and increased robustness to adversarial attacks. With up to 78.80% FLOPs reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.

Title: DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

• Authors: Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song, Yuhong Zhang, Hongjie Huang, Han Gao, Shilong Liu, Hao Zhang, Feng Li, Kent Yu, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14347
• Pdf URL: https://arxiv.org/pdf/2411.14347
• Copy Paste: [[2411.14347]] DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding(https://arxiv.org/abs/2411.14347)
• Keywords: transformer, segmentation
• Abstract: In this paper, we introduce DINO-X, which is a unified object-centric vision model developed by IDEA Research with the best open-world object detection performance to date. DINO-X employs the same Transformer-based encoder-decoder architecture as Grounding DINO 1.5 to pursue an object-level representation for open-world object understanding. To make long-tailed object detection easy, DINO-X extends its input options to support text prompt, visual prompt, and customized prompt. With such flexible prompt options, we develop a universal object prompt to support prompt-free open-world detection, making it possible to detect anything in an image without requiring users to provide any prompt. To enhance the model's core grounding capability, we have constructed a large-scale dataset with over 100 million high-quality grounding samples, referred to as Grounding-100M, for advancing the model's open-vocabulary detection performance. Pre-training on such a large-scale grounding dataset leads to a foundational object-level representation, which enables DINO-X to integrate multiple perception heads to simultaneously support multiple object perception and understanding tasks, including detection, segmentation, pose estimation, object captioning, object-based QA, etc. Experimental results demonstrate the superior performance of DINO-X. Specifically, the DINO-X Pro model achieves 56.0 AP, 59.8 AP, and 52.4 AP on the COCO, LVIS-minival, and LVIS-val zero-shot object detection benchmarks, respectively. Notably, it scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val benchmarks, both improving the previous SOTA performance by 5.8 AP. Such a result underscores its significantly improved capacity for recognizing long-tailed objects.

Title: Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation

• Authors: Yuanhao Cai, He Zhang, Kai Zhang, Yixun Liang, Mengwei Ren, Fujun Luan, Qing Liu, Soo Ye Kim, Jianming Zhang, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Alan Yuille
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.14384
• Pdf URL: https://arxiv.org/pdf/2411.14384
• Copy Paste: [[2411.14384]] Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation(https://arxiv.org/abs/2411.14384)
• Keywords: robust, diffusion
• Abstract: Existing feed-forward image-to-3D methods mainly rely on 2D multi-view diffusion models that cannot guarantee 3D consistency. These methods easily collapse when changing the prompt view direction and mainly handle object-centric prompt images. In this paper, we propose a novel single-stage 3D diffusion model, DiffusionGS, for object and scene generation from a single view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to enforce view consistency and allow the model to generate robustly given prompt views of any directions, beyond object-centric inputs. Plus, to improve the capability and generalization ability of DiffusionGS, we scale up 3D training data by developing a scene-object mixed training strategy. Experiments show that our method enjoys better generation quality (2.20 dB higher in PSNR and 23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA methods. The user study and text-to-3D applications also reveals the practical values of our method. Our Project page at this https URL shows the video and interactive generation results.

Title: Securing Legacy Communication Networks via Authenticated Cyclic Redundancy Integrity Check

• Authors: Alessandro Lotto, Alessandro Brighente, Mauro Conti
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14394
• Pdf URL: https://arxiv.org/pdf/2411.14394
• Copy Paste: [[2411.14394]] Securing Legacy Communication Networks via Authenticated Cyclic Redundancy Integrity Check(https://arxiv.org/abs/2411.14394)
• Keywords: security, protect, attack, robust
• Abstract: Integrating modern communication technologies into legacy systems, such as Industrial Control Systems and in-vehicle networks, invalidates the assumptions of isolated and trusted operating environments. Security incidents like the 2015 Ukraine power grid attack and the 2021 compromise of a U.S. water treatment facility demonstrate how increased interconnectivity, paired with insufficient security measures, expose these critical systems to cyber threats, posing risks to national and public safety. These attacks were favored by the lack of proper message authentication, highlighting its importance as a primary countermeasure to enhance system security. Solutions proposed in the literature remain largely unadopted in practice due to challenges such as preserving backward compatibility, additional hardware requirements, and limited computational resources on legacy devices. Moreover, many solutions are protocol-specific, necessitating complex and costly multiple implementations in heterogeneous systems. In this paper, we propose Authenticated Cyclic Redundancy Integrity Check (ACRIC), a novel security mechanism that overcomes these limitations by leveraging a cryptographic computation of the existing Cyclyic Redundancy Check (CRC) field to ensure message integrity protection and authentication. ACRIC preserves backward compatibility without requiring additional hardware and is protocol agnostic. This makes it applicable across various systems, suitable for diverse legacy network protocols including point-to-point and broadcast communications. Experimental results, supported by formal verification and real-world testing, demonstrate that ACRIC offers robust security with minimal transmission overhead (<< 1 ms). This proves ACRIC's practicality, cost-effectiveness, and suitability for real-world adoption.

Title: Lightweight Safety Guardrails Using Fine-tuned BERT Embeddings

• Authors: Aaron Zheng, Mansi Rana, Andreas Stolcke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14398
• Pdf URL: https://arxiv.org/pdf/2411.14398
• Copy Paste: [[2411.14398]] Lightweight Safety Guardrails Using Fine-tuned BERT Embeddings(https://arxiv.org/abs/2411.14398)
• Keywords: robust, large language model
• Abstract: With the recent proliferation of large language models (LLMs), enterprises have been able to rapidly develop proof-of-concepts and prototypes. As a result, there is a growing need to implement robust guardrails that monitor, quantize and control an LLM's behavior, ensuring that the use is reliable, safe, accurate and also aligned with the users' expectations. Previous approaches for filtering out inappropriate user prompts or system outputs, such as LlamaGuard and OpenAI's MOD API, have achieved significant success by fine-tuning existing LLMs. However, using fine-tuned LLMs as guardrails introduces increased latency and higher maintenance costs, which may not be practical or scalable for cost-efficient deployments. We take a different approach, focusing on fine-tuning a lightweight architecture: Sentence-BERT. This method reduces the model size from LlamaGuard's 7 billion parameters to approximately 67 million, while maintaining comparable performance on the AEGIS safety benchmark.

Title: Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding

• Authors: Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zenghui Ding, Xianjun Yang, Yining Sun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14401
• Pdf URL: https://arxiv.org/pdf/2411.14401
• Copy Paste: [[2411.14401]] Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding(https://arxiv.org/abs/2411.14401)
• Keywords: robust, large language model
• Abstract: Recent advancements in multimodal large language models (MLLMs) have opened new avenues for video understanding. However, achieving high fidelity in zero-shot video tasks remains challenging. Traditional video processing methods rely heavily on fine-tuning to capture nuanced spatial-temporal details, which incurs significant data and computation costs. In contrast, training-free approaches, though efficient, often lack robustness in preserving context-rich features across complex video content. To this end, we propose DYTO, a novel dynamic token merging framework for zero-shot video understanding that adaptively optimizes token efficiency while preserving crucial scene details. DYTO integrates a hierarchical frame selection and a bipartite token merging strategy to dynamically cluster key frames and selectively compress token sequences, striking a balance between computational efficiency with semantic richness. Extensive experiments across multiple benchmarks demonstrate the effectiveness of DYTO, achieving superior performance compared to both fine-tuned and training-free methods and setting a new state-of-the-art for zero-shot video understanding.

Title: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation

• Authors: Zhuoman Liu, Weicai Ye, Yan Luximon, Pengfei Wan, Di Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14423
• Pdf URL: https://arxiv.org/pdf/2411.14423
• Copy Paste: [[2411.14423]] Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation(https://arxiv.org/abs/2411.14423)
• Keywords: diffusion
• Abstract: Realistic simulation of dynamic scenes requires accurately capturing diverse material properties and modeling complex object interactions grounded in physical principles. However, existing methods are constrained to basic material types with limited predictable parameters, making them insufficient to represent the complexity of real-world materials. We introduce a novel approach that leverages multi-modal foundation models and video diffusion to achieve enhanced 4D dynamic scene simulation. Our method utilizes multi-modal models to identify material types and initialize material parameters through image queries, while simultaneously inferring 3D Gaussian splats for detailed scene representation. We further refine these material parameters using video diffusion with a differentiable Material Point Method (MPM) and optical flow guidance rather than render loss or Score Distillation Sampling (SDS) loss. This integrated framework enables accurate prediction and realistic simulation of dynamic interactions in real-world scenarios, advancing both accuracy and flexibility in physics-based simulations.

Title: Learning Fair Robustness via Domain Mixup

• Authors: Meiyu Zhong, Ravi Tandon
• Subjects: cs.LG, cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2411.14424
• Pdf URL: https://arxiv.org/pdf/2411.14424
• Copy Paste: [[2411.14424]] Learning Fair Robustness via Domain Mixup(https://arxiv.org/abs/2411.14424)
• Keywords: attack, robust, fair
• Abstract: Adversarial training is one of the predominant techniques for training classifiers that are robust to adversarial attacks. Recent work, however has found that adversarial training, which makes the overall classifier robust, it does not necessarily provide equal amount of robustness for all classes. In this paper, we propose the use of mixup for the problem of learning fair robust classifiers, which can provide similar robustness across all classes. Specifically, the idea is to mix inputs from the same classes and perform adversarial training on mixed up inputs. We present a theoretical analysis of this idea for the case of linear classifiers and show that mixup combined with adversarial training can provably reduce the class-wise robustness disparity. This method not only contributes to reducing the disparity in class-wise adversarial risk, but also the class-wise natural risk. Complementing our theoretical analysis, we also provide experimental results on both synthetic data and the real world dataset (CIFAR-10), which shows improvement in class wise disparities for both natural and adversarial risks.

Title: Revisiting the Integration of Convolution and Attention for Vision Backbone

• Authors: Lei Zhu, Xinjiang Wang, Wayne Zhang, Rynson W. H. Lau
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14429
• Pdf URL: https://arxiv.org/pdf/2411.14429
• Copy Paste: [[2411.14429]] Revisiting the Integration of Convolution and Attention for Vision Backbone(https://arxiv.org/abs/2411.14429)
• Keywords: extraction, interpretability, transformer, segmentation
• Abstract: Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \url{this https URL}.

Title: Stable Flow: Vital Layers for Training-Free Image Editing

• Authors: Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14430
• Pdf URL: https://arxiv.org/pdf/2411.14430
• Copy Paste: [[2411.14430]] Stable Flow: Vital Layers for Training-Free Image Editing(https://arxiv.org/abs/2411.14430)
• Keywords: diffusion, transformer
• Abstract: Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT), and employed flow-matching for improved training and sampling. However, they exhibit limited generation diversity. In this work, we leverage this limitation to perform consistent image edits via selective injection of attention features. The main challenge is that, unlike the UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it unclear in which layers to perform the injection. Therefore, we propose an automatic method to identify "vital layers" within DiT, crucial for image formation, and demonstrate how these layers facilitate a range of controlled stable edits, from non-rigid modifications to object addition, using the same mechanism. Next, to enable real-image editing, we introduce an improved image inversion method for flow models. Finally, we evaluate our approach through qualitative and quantitative comparisons, along with a user study, and demonstrate its effectiveness across multiple applications. The project page is available at this https URL

Title: Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

• Authors: Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14432
• Pdf URL: https://arxiv.org/pdf/2411.14432
• Copy Paste: [[2411.14432]] Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models(https://arxiv.org/abs/2411.14432)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) demonstrate enhanced capabilities and reliability by reasoning more, evolving from Chain-of-Thought prompting to product-level solutions like OpenAI o1. Despite various efforts to improve LLM reasoning, high-quality long-chain reasoning data and optimized training pipelines still remain inadequately explored in vision-language tasks. In this paper, we present Insight-V, an early effort to 1) scalably produce long and robust reasoning data for complex multi-modal tasks, and 2) an effective training pipeline to enhance the reasoning capabilities of multi-modal large language models (MLLMs). Specifically, to create long and structured reasoning data without human labor, we design a two-step pipeline with a progressive strategy to generate sufficiently long and diverse reasoning paths and a multi-granularity assessment method to ensure data quality. We observe that directly supervising MLLMs with such long and complex reasoning data will not yield ideal reasoning ability. To tackle this problem, we design a multi-agent system consisting of a reasoning agent dedicated to performing long-chain reasoning and a summary agent trained to judge and summarize reasoning results. We further incorporate an iterative DPO algorithm to enhance the reasoning agent's generation stability and quality. Based on the popular LLaVA-NeXT model and our stronger base MLLM, we demonstrate significant performance gains across challenging multi-modal benchmarks requiring visual reasoning. Benefiting from our multi-agent system, Insight-V can also easily maintain or improve performance on perception-focused multi-modal tasks.