Mitigating Fine-Tuning Risks in LLMs Via Safety-Aware Probing Optimization

The significant progress of large language models LLMs has led to remarkable achievements across numerous applications. However, their ability to generate harmful content has sparked substantial safety concerns. Despite the implementation of safety alignment techniques during the pre-training...

Advancing Security with Digital Twins: a Comprehensive Survey

The proliferation of electronic devices has greatly transformed every aspect of human life, such as communication, healthcare, transportation, and energy. Unfortunately, the global electronics supply chain is vulnerable to various attacks, including piracy of intellectual properties, tampering,...

CAIN: Hijacking LLM-Humans Conversations Via a Two-Stage Malicious System Prompt Generation and Refining Framework

Large language models LLMs have advanced many applications, but are also known to be vulnerable to adversarial attacks. In this work, we introduce a novel security threat: hijacking AI-human conversations by manipulating LLMs' system prompts to produce malicious answers only to specific targeted...

MTSA: Multi-Turn Safety Alignment for LLMs through Multi-Round Red-Teaming

Whitepaper called MTSA: Multi-Turn Safety Alignment For LLMs Through Multi-Round Red-Teaming...

Unlearning Isn'T Deletion: Investigating Reversibility of Machine Unlearning in LLMs

Unlearning in large language models LLMs is intended to remove the influence of specific data, yet current evaluations rely heavily on token-level metrics such as accuracy and perplexity. We show that these metrics can be misleading: models often appear to forget, but their original behavior can ...

CoTSRF: Utilize Chain of Thought As Stealthy and Robust Fingerprint of Large Language Models

Despite providing superior performance, open-source large language models LLMs are vulnerable to abusive usage. To address this issue, recent works propose LLM fingerprinting methods to identify the specific source LLMs behind suspect applications. However, these methods fail to provide stealthy...

DuFFin: a Dual-Level Fingerprinting Framework for LLMs IP Protection

Whitepaper called DuFFin: A Dual-Level Fingerprinting Framework For LLMs IP Protection...

Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: an Empirical Study on Popular Open-Source Projects

Command injection vulnerabilities are a significant security threat in dynamic languages like Python, particularly in widely used open-source projects where security issues can have extensive impact. With the proven effectiveness of Large Language ModelsLLMs in code-related tasks, such as testing...

One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems

Large Language Models LLMs enhanced with Retrieval-Augmented Generation RAG have shown improved performance in generating accurate responses. However, the dependence on external knowledge bases introduces potential security vulnerabilities, particularly when these knowledge bases are publicly...

MorphMark: Flexible Adaptive Watermarking for Large Language Models

Watermarking by altering token sampling probabilities based on red-green list is a promising method for tracing the origin of text generated by large language models LLMs. However, existing watermark methods often struggle with a fundamental dilemma: improving watermark effectiveness the...

Fragments to Facts: Partial-Information Fragment Inference from LLMs

Large language models LLMs can leak sensitive training data through memorization and membership inference attacks. Prior work has primarily focused on strong adversarial assumptions, including attacker access to entire samples or long, ordered prefixes, leaving open the question of how vulnerable...

Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models LLMs offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue...

On Membership Inference Attacks in Knowledge Distillation

Nowadays, Large Language Models LLMs are trained on huge datasets, some including sensitive information. This poses a serious privacy concern because privacy attacks such as Membership Inference Attacks MIAs may detect this sensitive information. While knowledge distillation compresses LLMs into...

ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

The integration of large language models LLMs into a wide range of applications has highlighted the critical role of well-crafted system prompts, which require extensive testing and domain expertise. These prompts enhance task performance but may also encode sensitive information and filtering...

Cape: Context-Aware Prompt Perturbation Mechanism with Differential Privacy

Large Language Models LLMs have gained significant popularity due to their remarkable capabilities in text understanding and generation. However, despite their widespread deployment in inference services such as ChatGPT, concerns about the potential leakage of sensitive user data have arisen...

S3C2 Summit 2024-09: Industry Secure Software Supply Chain Summit

While providing economic and software development value, software supply chains are only as strong as their weakest link. Over the past several years, there has been an exponential increase in cyberattacks, specifically targeting vulnerable links in critical software supply chains. These attacks...

On Technique Identification and Threat-Actor Attribution Using LLMs and Embedding Models

Attribution of cyber-attacks remains a complex but critical challenge for cyber defenders. Currently, manual extraction of behavioral indicators from dense forensic documentation causes significant attribution delays, especially following major incidents at the international scale. This research...

Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent

As Large Language Models LLMs are widely used, understanding them systematically is key to improving their safety and realizing their full potential. Although many models are aligned using techniques such as reinforcement learning from human feedback RLHF, they are still vulnerable to jailbreakin...

Red Teaming the Mind of the Machine: a Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs

Large Language Models LLMs are increasingly integrated into consumer and enterprise applications. Despite their capabilities, they remain susceptible to adversarial attacks such as prompt injection and jailbreaks that override alignment safeguards. This paper provides a systematic investigation o...

Federated Large Language Models: Feasibility, Robustness, Security and Future Directions

The integration of Large Language Models LLMs and Federated Learning FL presents a promising solution for joint training on distributed data while preserving privacy and addressing data silo issues. However, this emerging field, known as Federated Large Language Models FLLM, faces significant...