Anthropic recently announced that future Claude models would embed an invisible watermark in their output. This watermark is based on Google DeepMind’s SynthID-Text. Text watermarking has regulatory relevance due to Article 50(2) of the EU AI Act, which requires providers of AI systems generating synthetic text to mark their outputs in a machine-readable format to indicate whether they are artificially generated or manipulated. The watermarking is intended for provenance, but SynthID-Text alters the model's token generation process, potentially affecting safety behavior, including the model's ability to refuse harmful requests and the effectiveness of those refusals under prompt injection. This phenomenon is referred to as sampling drift.
The empirical question of whether this drift manifests in practice has been investigated, revealing that it does affect both model refusal behavior and agent tool calling. The effects are dependent on the model and key used and can be obscured by aggregate scores when changes counterbalance each other. Both net performance and paired disagreement between watermarked and unwatermarked runs are reported. The study discusses implications for AI safety and security and recommendations for developers.
Watermarking embeds a signal that identifies AI-generated output. Existing methods include post-processing and direct integration into LLM generation. The study utilizes SynthID’s non-distortionary configuration, which preserves the original token distribution while allowing for variability in individual generations. Anthropic's watermarking applies at the model level and affects models accessed through the Claude Platform API and cloud providers, making it relevant for agents built around these models.
The study employs a paired design for two experiments, evaluating tool calling and refusal behavior across various models and settings. Results indicate that watermarking reduces accuracy in tool selection across six of seven models, with significant decreases in four. The net change in accuracy does not reflect individual call success rates, as incorrect calls can be offset by correct ones. The churn rate, defined as the share of items with differing verdicts between watermarked and unwatermarked runs, averages 6.5% across 21 model-temperature combinations.
Error types are also significant, with malformed output preventing intended calls, while incorrect arguments can still execute successfully. The study finds that watermarking affects refusal behavior, particularly under prompt injection conditions, where disagreement increases and compliance rises. For example, churn rates for gemma-3-27b increase from 6.0% on bare harmful requests to 23.5% under prompt injection, indicating a shift in compliance behavior.
The study also compares watermark-induced disagreement with disagreement observed when changing temperature settings, finding that watermark-induced churn is significantly higher in many models. Additionally, the effect of watermarking varies based on the watermark key used, with different keys producing varying levels of attack success across models. Overall, the findings suggest that the impact of watermarking on AI agent behavior is complex and model-dependent, necessitating careful consideration in AI development.