Skip to content
Recipes
Skip navigation
Recipes

Exporting at volume

whisper.export pulls the reconciled threat corpus a page at a time, by label, with a cursor — for training a classifier or seeding a local store.

Published

View as Markdown
On this page (8)

Exporting at volume Documentation

whisper.export is the bulk read. It walks the reconciled threat corpus by label and hands back one row per indicator, paginated by cursor.

Key concepts: Threat Intelligence, Reconciled Verdict, Indicator of Compromise.


What it is for

Distilling a classifier, seeding a local indicator store, or taking a periodic snapshot of the corpus at a given label. It is not a substitute for a query: if you want indicators matching a condition, write Cypher. export is for the whole label.


The call takes one map, and the keys are a closed set

whisper.export takes exactly one map argument. The map accepts three keys and no others:

KeyRequiredMeaning
labelyeswhich corpus to read: malicious, ambiguous — coverage: ambiguous. In coverage, and the evidence points both ways. or benign-allowlisted
limitpass it on every callhow many rows this page returns
cursoron every page after the firstthe next_cursor you were handed by the previous page
cypher · runnablegraph.whisper.securitySign in to run
CALL whisper.export({label: "malicious", limit: 2})
YIELD host, label, ip, asn, last_seen, coverage, truncated, next_cursor
RETURN host, label, ip, coverage, truncated

Any other key, a missing label, or a label outside the three values is rejected before anything runs. unknown is not a label: it would be an unbounded scan of the whole graph rather than a read of a corpus, so there is no export for it.

The full row is wider than the columns above. YIELD any of host, label, ip, cidr, asn, url_paths, cert_shas, tls_fingerprints, dns, last_seen, coverage, truncated, supersedes, look_alike_negatives, next_cursor, and name the ones you read rather than taking the whole row.


Every row is reconciled before it is emitted

A candidate is emitted only when its reconciled label matches the one you asked for. That is what keeps a URL-scoped listing on a multi-tenant apex out of the malicious export: it reconciles to ambiguous — coverage: ambiguous. In coverage, and the evidence points both ways. and lands there instead.

Two of the columns are forward projections rather than facts about the indicator itself:

  • supersedes — the host's superseded registrar-lineage targets.
  • look_alike_negatives — confusable-but-clean neighbours. These are the hard negatives, and they are the reason the export is usable for distillation rather than only for blocklisting.

tls_fingerprints entries are family-tagged: ja3:, ja4: or jarm: followed by the hash.

CSR-derived fields are confirmed-only and come back null when absent, rather than guessed.


Paging: always pass limit, always continue from next_cursor

Each page carries an opaque next_cursor. Pass it back, with the same label and a limit, to continue:

cypher
CALL whisper.export({label: "malicious", limit: 500, cursor: "<the next_cursor from the previous page>"})
YIELD host, label, coverage, truncated, next_cursor
RETURN host, label, coverage, truncated, next_cursor

The loop is: call with limit, process the rows, take next_cursor off the page, call again with cursor set to it. Stop when a page comes back without a next_cursor.

The cursor is opaque. Do not parse it, do not construct one, and do not assume it survives a schema change; treat it as a token you received and hand back.


truncated is replicated on every row

truncated: true means the page you are holding is a prefix, not the whole answer. It is set on every row of a truncated page rather than once at the end, so a consumer that streams rows and never sees the last one still knows.

Resuming: keep the last next_cursor you successfully processed, not the last row. A cursor identifies a position; a row does not.


Choosing a page size

limit shapes the page. Smaller pages mean more round trips; larger pages mean fewer. Start small enough that a single call comfortably completes, then increase it until the round-trip count stops mattering. A page that comes back truncated is a prefix to continue from, not a failure to retry: take its next_cursor and keep going.


When to use the graph API instead

You wantUse
Every indicator at a labelwhisper.export
Indicators matching a pattern, a layer or a pivotCypher — see Recipes
One verdict for one indicatorwhisper.assess
A verdict for a URLwhisper.assessUrl