Verify responses

Engy can attach a cryptographic proof to any response. You can then recompute the proof against the model weights to confirm that the advertised model generated the output. This guide is intended for developers who want to verify individual responses independently.

How verification worksCopy Anchor Link

When you request a proof, the GPU worker that serves your request records a fingerprint of the model's internal activations for each generated token. This fingerprint is a TOPLOC proof.

To verify the proof, you run the same model on your own hardware, recompute the activations for the same tokens, and compare them with the proof:

• If the activations match, the advertised model produced the output.
• If the worker used a different, smaller, or modified model, the activations do not match, and verification fails.

Engy passes the proof to you unchanged. Engy does not store the proof, so requesting one does not affect zero data retention.

Request a proofCopy Anchor Link

Set provider.require_proof to true in the request body:

curl https://api.engy.ai/v1/chat/completions \
  -H "Authorization: Bearer $ENGY_API_KEY" \
  -H "Content-Type: application/json" \
  -o response.json \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider": {"require_proof": true}
  }'

Engy routes the request only to workers that can produce proofs. If no such worker is available for the model, the request fails with HTTP status 503.

Where the proof appearsCopy Anchor Link

The proof is returned in a top-level proof field. Its location depends on the endpoint and on whether you stream the response:

If you did not request a proof, or the worker returned none, the proof field is absent.

Proof formatCopy Anchor Link

{
  "proof": {
    "model_root": "...",
    "output_token_ids": [...],
    "toploc": {
      "proofs": ["..."],
      "topk": 128,
      "decode_batching_size": 32,
      "skip_prefill": true,
      "n_tokens": 9,
      "prompt_ids": [...]
    }
  }
}

Verify a proofCopy Anchor Link

PrerequisitesCopy Anchor Link

Before you begin, make sure you have:

• A GPU with enough memory to run the model. For example, qwen3.8-27b runs on a single NVIDIA H200 GPU.
• The model checkpoint from Hugging Face. Each model is served from one fixed checkpoint, and you must verify against the same checkpoint. For qwen3.8-27b, use Qwen/Qwen3.8-27B. For other models, see Models and pricing.
• Python 3.10 or later
• A saved response that contains a proof, such as response.json from the previous section

StepsCopy Anchor Link

pip install "sglang[all]" toploc torch httpx numpy

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --enable-return-hidden-states \
  --disable-radix-cache \
  --port 30000

import json
import sys

import httpx
import numpy as np
import torch
from toploc import verify_proofs_base64

SERVER_URL = "http://127.0.0.1:30000"
THRESHOLD = 40

with open(sys.argv[1]) as f:
    proof = json.load(f)["proof"]

toploc = proof["toploc"]
prompt_ids = toploc["prompt_ids"]
n_tokens = toploc["n_tokens"]
output_ids = proof["output_token_ids"][:n_tokens]
full_ids = prompt_ids + output_ids

# Recompute the activations with a single prefill pass.
result = httpx.post(
    f"{SERVER_URL}/generate",
    json={
        "input_ids": full_ids,
        "sampling_params": {"temperature": 0.0, "max_new_tokens": 0},
        "return_hidden_states": True,
    },
    timeout=600,
).json()
result = result[0] if isinstance(result, list) else result
block = np.asarray(result["meta_info"]["hidden_states"][0], dtype=np.float32)

# The returned block covers the last positions of the sequence.
block_start = len(full_ids) - block.shape[0]
prompt_len = len(prompt_ids)
activations = [
    torch.from_numpy(
        np.ascontiguousarray(block[prompt_len - 1 + t - block_start])
    ).to(torch.bfloat16)
    for t in range(n_tokens)
]

results = verify_proofs_base64(
    activations,
    toploc["proofs"],
    decode_batching_size=toploc["decode_batching_size"],
    topk=toploc["topk"],
    skip_prefill=toploc["skip_prefill"],
)

mismatches = float(np.mean([r.exp_mismatches for r in results]))
status = "PASS" if mismatches <= THRESHOLD else "FAIL"
print(f"{status}: {mismatches:.1f} of {toploc['topk']} values mismatched")

python verify.py response.json

Interpret the resultCopy Anchor Link

The script reports the average number of mismatched activation values per proof segment, out of topk:

Small mismatches are expected, because different GPUs produce slightly different floating-point results. The default threshold of 40 separates the two cases with a wide margin.

LimitationsCopy Anchor Link

• Requesting a proof adds compute and bandwidth to the request, so proofs are not generated by default.
• Only workers that can produce proofs serve these requests. As a result, capacity for some models might be limited.
• Verification requires hardware that can run the full model.