Webhooks

Postback and pingback#

You can set both on the same task. Webhooks are optional; tasks also appear in tasks_ready either way.

Postback#

When the task completes, the API sends POST to your postback_url with the same envelope that task_get would return, and these headers:

POST /webhooks/postback HTTP/1.1
Content-Type: application/json
User-Agent: Screaming Data webhooks
X-Signature: sha256=5d41402abc4b2a76b9719d911017c592…

Pingback#

A pingback is a lightweight GET to your pingback_url. Put the placeholders $id and $tag anywhere in the URL; they are replaced with the task id and your tag (URL-encoded):

https://example.com/webhooks/pingback?id=$id&tag=$tag
→ GET https://example.com/webhooks/pingback?id=09241235-4e1c-4b6a-9d8f-2c7a51f0e3b1&tag=catalog-sync

Verifying signatures#

Every delivery has an X-Signature header: sha256= followed by the hex HMAC-SHA256 of the signed message, keyed with your account’s webhook secret. For postbacks the message is the raw request body — verify it before parsing JSON. For pingbacks it is the full URL that was requested; behind a proxy, rebuild it from the original scheme, host, path and query. Always compare in constant time.

URLs are requested — and pingbacks signed — in their encoded form: an internationalised host name in punycode (пример.рф → xn--e1afmkfd.xn--p1ai), spaces and non-ASCII characters in the path and query percent-encoded (é → %C3%A9), the default port and any #fragment left out. The request line and Host header your server receives carry exactly that form, so the examples below rebuild the signed URL without any extra work. The data of task_get shows each URL in this form too.

import hashlib
import hmac
import os

from flask import Flask, abort, request

WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"].encode()
app = Flask(__name__)


def signature_is_valid(message: bytes, header: str) -> bool:
    expected = "sha256=" + hmac.new(WEBHOOK_SECRET, message, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header or "")


@app.post("/webhooks/postback")
def postback():
    # Postbacks: the signed message is the raw request body.
    if not signature_is_valid(request.get_data(), request.headers.get("X-Signature", "")):
        abort(401)
    envelope = request.get_json()
    for task in envelope["tasks"]:
        print(task["id"], task["status_code"], task["data"].get("tag"))
    return "", 204


@app.get("/webhooks/pingback")
def pingback():
    # Pingbacks: the signed message is the full URL that was requested.
    if not signature_is_valid(request.url.encode(), request.headers.get("X-Signature", "")):
        abort(401)
    print("task ready:", request.args["id"], request.args.get("tag"))
    return "", 204

import crypto from "node:crypto";
import express from "express";

const secret = process.env.WEBHOOK_SECRET;
const app = express();

function signatureIsValid(message, header = "") {
  const expected = "sha256=" + crypto.createHmac("sha256", secret).update(message).digest("hex");
  const a = Buffer.from(expected);
  const b = Buffer.from(header);
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

// Postbacks: the signed message is the raw request body.
app.post("/webhooks/postback", express.raw({ type: "application/json" }), (req, res) => {
  if (!signatureIsValid(req.body, req.get("X-Signature"))) return res.sendStatus(401);
  const envelope = JSON.parse(req.body.toString("utf8"));
  for (const task of envelope.tasks) console.log(task.id, task.status_code, task.data.tag);
  res.sendStatus(204);
});

// Pingbacks: the signed message is the full URL that was requested.
app.get("/webhooks/pingback", (req, res) => {
  const url = `${req.protocol}://${req.get("host")}${req.originalUrl}`;
  if (!signatureIsValid(url, req.get("X-Signature"))) return res.sendStatus(401);
  console.log("task ready:", req.query.id, req.query.tag);
  res.sendStatus(204);
});

app.listen(3000);

Delivery and retries#

• Answer with any 2xx status within a few seconds. Do heavy work asynchronously after responding.
• If your endpoint fails, times out (10 seconds) or answers with a non-2xx status, the delivery is retried up to 3 times: after 1, 5 and 15 minutes.
• Redirects are not followed: answer from the URL you configured.
• Deliveries can arrive more than once and out of order. Use the task id to make your handler idempotent.
• If all attempts fail, nothing is lost: the task stays in tasks_ready until you collect it, and results are kept for 30 days.

Security rules#

• Only http and https URLs without credentials are accepted. URLs whose host is a private, loopback or link-local address (for example 127.0.0.1, 10.0.0.0/8, 192.168.0.0/16, 169.254.0.0/16) or a local name such as localhost are refused when the task is posted — the task fails with 40501. So are URLs that could not be requested at all: control characters, or a host name with an empty label or a label longer than 63 characters.
• Host names are resolved at delivery time. If a name points to such an address, the delivery is refused and not retried; the result stays available through task_get.
• Use HTTPS endpoints so results and signatures are encrypted in transit.
• Reject requests without a valid signature.