# LLM Sleuth API — AI Assistant Instructions

> Save this file as `CLAUDE.md` in a project, or paste it into Claude, Codex, Cursor, or another AI coding assistant.
> It teaches the assistant how to compare AI search behavior, citations, source evidence, and answers through the LLM Sleuth API.

## When to Use LLM Sleuth

Use LLM Sleuth when the user wants to compare how multiple AI answer engines research and answer the same question. It is especially useful for:

- Comparing generated search queries across providers and models.
- Finding which domains and pages are cited by ChatGPT, Claude, Gemini, or Perplexity.
- Inspecting extracted page evidence behind citations.
- Auditing whether a company, product, or page appears in AI search answers.
- Comparing answer framing, recommendations, omissions, and provider failures.

For a meaningful comparison, use at least two providers unless the user explicitly asks to inspect one model. Do not present provider agreement as proof that a claim is true; report it as cross-model consensus and use the returned source evidence to evaluate it.

## Authentication

All API requests require an LLM Sleuth API key. Ask the user for a key before making a live request. Keys start with `asc_`.

Prefer the bearer header:

```text
Authorization: Bearer asc_your_api_key
```

The `x-api-key` header is also accepted. Never expose a key in client-side code, commit it to source control, or include the real value in output or logs.

Get or rotate a key at [llmsleuth.com/dashboard](https://llmsleuth.com/dashboard).

Base URL: `https://llmsleuth.com`

---

## Recommended Workflow

### 1. List the currently available models

Always fetch the model catalog before constructing a query. It is the source of truth for available model IDs, provider IDs, defaults, and credit costs.

```bash
curl https://llmsleuth.com/api/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"
```

Example response:

```json
{
  "object": "list",
  "models": [
    {
      "id": "gpt-5.6-sol",
      "provider": "openai",
      "name": "GPT-5.6 Sol",
      "credit_cost": 2,
      "is_default": true
    }
  ]
}
```

Do not hard-code the example model IDs as if they are permanently available. Match each selected `model` to the `provider` returned for it by this endpoint.

### 2. Choose explicit model runs

Use the `runs` field by default. A request accepts one to 16 `{ provider, model }` targets.

Supported provider IDs are currently:

| ID | Provider | Search data captured |
|---|---|---|
| `openai` | ChatGPT / OpenAI | Responses API web-search queries, sources, and annotations |
| `anthropic` | Claude / Anthropic | Web-search tool queries and citations |
| `gemini` | Gemini / Google | Grounding search queries and grounding chunks |
| `perplexity` | Perplexity / Sonar | Citations and search results; separate generated queries may not be exposed |

Add the selected models' `credit_cost` values to show the user the request cost before a large comparison.

### 3. Run the comparison

`POST /api/v1/query` runs the selected targets and returns the normalized results in the same response.

```bash
curl https://llmsleuth.com/api/v1/query \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is the best lead scraping tool?",
    "runs": [
      { "provider": "openai", "model": "gpt-5.6-sol" },
      { "provider": "anthropic", "model": "claude-sonnet-5" },
      { "provider": "gemini", "model": "gemini-3.5-flash" }
    ]
  }'
```

Request fields:

| Field | Type | Guidance |
|---|---|---|
| `query` | string | Required research question or prompt. Keep it at or below 4,000 characters. |
| `runs` | array | Recommended. One to 16 exact provider/model targets from the model catalog. |
| `providers` | array | Legacy shorthand that uses each provider's current default model. |

Send either `runs` or `providers`, never both. Prefer explicit `runs` so a comparison is reproducible and its credit cost is clear.

The endpoint is synchronous and provider research can take several minutes. Use a client timeout of at least seven minutes. Do not automatically retry an ambiguous timeout: another request can create a second charged execution. Check dashboard history or ask the user before retrying.

### 4. Inspect every provider result

Example response shape:

```json
{
  "execution_id": "7f79c814-1112-4a94-9bb4-719d5ccf5e98",
  "query": "What is the best lead scraping tool?",
  "runs": [
    { "provider": "openai", "model": "gpt-5.6-sol" }
  ],
  "providers": ["openai"],
  "credits_used": 2,
  "credits_available": 48,
  "results": [
    {
      "provider": "openai",
      "label": "ChatGPT / OpenAI",
      "ok": true,
      "status": 200,
      "model": "gpt-5.6-sol",
      "generatedQueries": ["best lead scraping tools 2026"],
      "queryCapture": "How the provider's search queries were captured.",
      "sources": [
        { "title": "Example source", "url": "https://example.com/source" }
      ],
      "sourcePreviews": [
        {
          "url": "https://example.com/source",
          "finalUrl": "https://example.com/source",
          "fetchStatus": 200,
          "contentType": "text/html",
          "wordCount": 1200,
          "metaDescription": "Page description",
          "headings": [{ "level": 1, "text": "Best Lead Tools" }],
          "faqs": [],
          "textSample": "Extracted page text..."
        }
      ],
      "answer": "Markdown answer text..."
    }
  ]
}
```

For each item in `results`, inspect:

- `ok` and `error`: whether that individual model completed successfully.
- `generatedQueries`: search queries exposed by the provider.
- `queryCapture`: provider-specific context about how query data was captured.
- `sources`: normalized cited or searched source URLs, with titles or snippets when available.
- `sourcePreviews`: fetched evidence for up to the first six sources, including redirects, status, metadata, headings, FAQs, and text or Markdown samples when available.
- `answer`: the provider's final answer as Markdown.

A successful HTTP response can still contain a failed individual result. Never assume every model succeeded because the request returned `200`; check every `results[].ok` value and report partial failures alongside successful comparisons.

### 5. Produce a comparison, not a blended answer

Unless the user asks for another format, organize the findings by these dimensions:

1. **Search strategy** — compare `generatedQueries` and explain how the providers framed the research.
2. **Source overlap** — identify domains or exact URLs cited by multiple models.
3. **Unique sources** — call out sources used by only one model.
4. **Evidence quality** — use `sourcePreviews` to distinguish accessible primary evidence from thin, blocked, redirected, or missing pages.
5. **Answer differences** — compare recommendations, claims, caveats, and omissions.
6. **Visibility** — state whether the target brand, product, domain, or URL appeared in each answer or source list.
7. **Failures and limits** — disclose provider errors and unavailable query or preview data.

Keep provider findings attributable. Do not merge all answers into one voice in a way that hides which model said or cited what.

---

## Legacy Default-Model Request

Use `providers` only when the user wants current defaults and exact model selection does not matter:

```bash
curl https://llmsleuth.com/api/v1/query \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Which CRM tools are most often recommended to startups?",
    "providers": ["openai", "anthropic", "gemini", "perplexity"]
  }'
```

Do not send `providers` together with `runs`.

## Errors

| Status | Meaning | What to do |
|---|---|---|
| `400` | Invalid JSON, missing query, unsupported provider, unavailable model, or invalid run selection | Read `error`; when supplied, use `details.models_url` to refresh the catalog. |
| `401` | Missing or invalid API key | Ask the user for a valid LLM Sleuth key and verify the auth header. |
| `402` | Insufficient credits | Report `details.credits_required` and ask the user to add credits or choose fewer/cheaper models. |
| `500` | Provider execution or persistence failure | Report the error. Avoid blind automatic retries after a timeout or ambiguous failure. |

Error responses use this general shape:

```json
{
  "error": "Human-readable message",
  "details": {}
}
```

## Reliability and Interpretation Rules

- Fetch `GET /api/v1/models` before choosing models; availability and credit costs can change.
- Selected model costs are charged before the comparison runs. The response reports `credits_used` and remaining `credits_available`.
- Treat missing `generatedQueries` as unavailable capture, not proof that the provider did not search. Perplexity commonly exposes citations without separate query strings.
- Only up to the first six sources per result are enriched into `sourcePreviews`. The full normalized source list remains in `sources`.
- A missing or failed source preview does not erase the citation. Preserve the source URL and report the fetch limitation.
- Prefer original and primary sources when judging evidence. A model citation is evidence of model behavior, not automatic validation of the cited claim.
- Save the returned `execution_id`; it identifies the comparison in LLM Sleuth history.
- Use `https://llmsleuth.com` in production integrations, never a preview or localhost domain.