Developer blog

Build a Production Polling Client for Video Dubbing Jobs

Every async API tutorial ends the same way:

while True:
    job = get(f"/jobs/{job_id}")
    if job["state"] == "done":
        break
    time.sleep(5)

That loop is fine for a demo and it fails in production for reasons that have nothing to do with dubbing. It holds the job id only in memory, so a deploy loses the work. It cannot tell a network blip from a job that genuinely failed. It polls at a fixed rate that gets you throttled the moment you run twenty jobs at once. And when the submit call times out, it has no idea whether it just spent your minutes.

The version below fixes those four, and every part of it is forced by something in the API contract rather than by taste.

The contract, as of today

Six endpoints matter. GET /api/v1/ping is unauthenticated and answers {"status":"ok"}, which makes it a usable liveness check. POST /api/v1/estimate reads a source's duration for free, without creating anything. POST /api/v1/dub and POST /api/v1/transcribe create work. GET /api/v1/jobs/{id} is the poll. GET /api/v1/me/usage reports your plan, quota and top-up balance.

Everything except ping wants Authorization: Bearer dv_live_…. A missing, unknown or revoked key gets exactly one answer, and I checked this against production while writing:

$ curl -s -X POST https://dubanyvideo.com/api/v1/dub -d '{}'
{"error":"unauthorized"}

Keys are created per integration and can expire on a date you choose. The list shows each key masked, with its last use, which is the cheapest way to notice a client you forgot was running.

Three error shapes are worth coding against, because they mean genuinely different things. A 401 {"error":"unauthorized"} is permanent: the same key will be rejected next time, so retrying is noise. A 422 {"error":"unprocessable","message":"…"} carries a human-readable reason (an unsupported language named explicitly, a source longer than the one-hour ceiling, a link that could not be probed, quota exhausted) and the same request will be rejected identically forever. A 429 {"error":"rate_limited"} is the only one that means "the same request, later".

Job state is a small closed set: queued, processing, then done or failed, plus expired and cancelled. Four of those are terminal. Nothing else exists, so a client that treats anything outside the set as terminal will not be surprised by a value invented next quarter.

There are no webhooks. The API is poll-only, so the quality of your loop is the quality of your integration.

Persist before you poll

The first rule has nothing to do with polling: write the row before you make the call that creates work.

import os, json, random, sqlite3, time
import requests

API = "https://dubanyvideo.com/api/v1"
KEY = os.environ["DV_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}

TERMINAL = {"done", "failed", "expired", "cancelled"}


class AuthError(RuntimeError):
    """401. Never retried: a bad key is bad on every attempt."""


class Rejected(RuntimeError):
    """422. The request itself is wrong; retrying sends the same wrong request."""


def db():
    conn = sqlite3.connect("dubs.db")
    conn.execute(
        """CREATE TABLE IF NOT EXISTS dub_job (
             local_ref TEXT PRIMARY KEY,
             job_id    INTEGER UNIQUE,
             lang      TEXT NOT NULL,
             state     TEXT NOT NULL,
             archive_url TEXT,
             expires_at  TEXT,
             downloaded_path TEXT)"""
    )
    return conn


def call(method, path, **kw):
    """One request. Classifies the outcome; never retries by itself."""
    r = requests.request(method, f"{API}{path}", headers=HEADERS, timeout=30, **kw)
    if r.status_code == 401:
        raise AuthError("key rejected")
    if r.status_code == 422:
        raise Rejected(r.json().get("message", "unprocessable"))
    if r.status_code == 429:
        return None                      # caller backs off
    if r.status_code >= 500:
        return None                      # caller backs off
    r.raise_for_status()
    return r.json()

local_ref is your own identifier for the thing being dubbed: a video id, a lesson id, a row in the table you already have. It is the primary key, not the job id, because it exists before the job does.

def submit(conn, local_ref, url, lang):
    """Submit at most once per local_ref. The row is written before the call."""
    row = conn.execute(
        "SELECT job_id FROM dub_job WHERE local_ref = ?", (local_ref,)
    ).fetchone()
    if row and row[0]:
        return row[0]                    # already submitted, do not spend again

    conn.execute(
        "INSERT OR IGNORE INTO dub_job (local_ref, lang, state) VALUES (?, ?, 'submitting')",
        (local_ref, lang),
    )
    conn.commit()

    body = {"source_type": "url", "input_url": url, "target_lang": lang}
    created = call("POST", "/dub", data=json.dumps(body))
    if created is None:
        raise RuntimeError("throttled or unavailable; retry this local_ref later")

    conn.execute(
        "UPDATE dub_job SET job_id = ?, state = ? WHERE local_ref = ?",
        (created["id"], created["state"], local_ref),
    )
    conn.commit()
    return created["id"]

The create response is deliberately minimal: {"id": …, "state": "queued"}. Ask for target_langs instead of target_lang and you get {"jobs": [{id, state, targetLang}, …]}, one job per language, with an all-or-nothing quota hold across the batch. Either way, the id is the only thing you cannot re-derive later, so it goes to disk immediately.

The idempotency boundary, stated plainly

There are no idempotency keys in this API. A submit that times out on your side may still have created a job and reserved minutes on ours.

That is why submit writes its row first and returns early when a job id is already recorded. What it deliberately does not do is retry a timed-out submit automatically, because that is exactly how one video becomes two jobs and twice the spend.

When I ran the listing above against production with a deliberately invalid key, the row stayed behind with job_id NULL:

submit -> AuthError: key rejected
rows: [('ref-1', None, 'submitting')]

That row is the honest representation of "we do not know". For a 401 the answer is easy, since nothing was created. For a timeout it is not, and the resolution is a human one: check GET /api/v1/me/usage for unexpected consumption, or look at the workspace, before you press submit again.

Backoff, jitter and the shape of the ceiling

Two rate limits apply, and the second one surprises people. Sixty requests per minute per key across all of /api/v1/*, and a separate submit ceiling per plan: two per minute on free, five on starter, ten on medium, twenty on super.

That first limit is shared by your polls. Twenty jobs polled every five seconds is 240 requests a minute, which is four times the ceiling, and the 429s that follow will delay the results you were impatient for. Polling every fifteen seconds with jitter puts the same twenty jobs at eighty requests a minute, which is still too many. Dubbing takes minutes, not seconds, so the sane interval starts around ten seconds for a single job and grows with the fleet.

def poll_once(conn, job_id):
    """One status read. Returns the job dict, or None when we were told to wait."""
    job = call("GET", f"/jobs/{job_id}")
    if job is None:
        return None
    conn.execute(
        "UPDATE dub_job SET state = ?, archive_url = ?, expires_at = ? WHERE job_id = ?",
        (job["state"], job.get("archiveUrl"), job.get("archiveExpiresAt"), job_id),
    )
    conn.commit()
    return job


def wait_for(conn, job_id, first_delay=10, max_delay=120, deadline_s=3 * 3600):
    """Bounded exponential backoff with jitter, stopping only on a terminal state."""
    delay, started = first_delay, time.monotonic()
    while time.monotonic() - started < deadline_s:
        job = poll_once(conn, job_id)
        if job and job["state"] in TERMINAL:
            return job
        time.sleep(delay * (0.5 + random.random()))   # jitter: 50%..150%
        delay = min(delay * 1.5, max_delay)
    raise TimeoutError(f"job {job_id} did not finish within the deadline")

The jitter matters more than the growth factor. Submit thirty jobs in a loop and, without it, thirty clients wake up in the same millisecond forever, which is a self-inflicted thundering herd against a per-key limit you now share with yourself.

Every poll writes what it learned. That is what makes the process restartable at any point rather than only between jobs.

Restart recovery

Once the state lives in the table, recovery is a query.

def resume(conn):
    """After a restart: everything unfinished is already in the table."""
    rows = conn.execute(
        "SELECT job_id FROM dub_job WHERE job_id IS NOT NULL "
        "AND state NOT IN ('done','failed','expired','cancelled')"
    ).fetchall()
    return [r[0] for r in rows]

On boot, poll everything resume returns before submitting anything new. The rows with a NULL job_id are the ambiguous ones from the previous section and should be reported, not retried automatically.

Download before the link expires

A finished job carries a primary archiveUrl and an artifacts object whose four keys are always present and null until they exist: dubbedVideoUrl, dubbedAudioUrl, translationSrtUrl, transcriptSrtUrl. An audio-only source has no dubbed video; a transcribe job has only the transcript.

It also carries archiveExpiresAt, an ISO 8601 timestamp. The files are cleaned up after it, so treat the field as authoritative and do not hardcode a retention window in your client.

def fetch_artifacts(conn, job):
    """Download before archiveExpiresAt. The URL is a plain CDN link."""
    if job["state"] != "done":
        raise RuntimeError(f"job {job['id']} ended as {job['state']}: {job.get('errorMessage')}")

    path = f"dub-{job['id']}-{job['targetLang']}.mp4"
    with requests.get(job["archiveUrl"], stream=True, timeout=300) as r:
        r.raise_for_status()
        with open(path, "wb") as f:
            for chunk in r.iter_content(1 << 20):
                f.write(chunk)

    conn.execute("UPDATE dub_job SET downloaded_path = ? WHERE job_id = ?", (path, job["id"]))
    conn.commit()
    return path

Stream it rather than reaching for r.content: a dubbed lesson runs to hundreds of megabytes, and the short version holds every one of them in memory, per worker.

If you store the file yourself, record its size and a checksum next to the path. The download is the one step where a partial success looks exactly like a success until someone plays the file.

What to log and when to page

Log the local ref, the job id and the state on every transition, and nothing on an unchanged poll, or the log becomes a heartbeat you stop reading.

Three conditions deserve an alert. A rising 429 rate means your poll interval no longer matches your fleet. Jobs sitting in queued far longer than usual is a pipeline signal rather than a client bug. And any row whose expires_at has passed with no downloaded_path is data you have already lost, which should be loud.

Terminal failed is not an alert by itself; it is a per-job outcome with errorMessage attached. Alert on the rate, not the event.

The failure matrix

What was and was not run

The transport layer above was executed against the live API on the day of writing: ping, the unauthorized submit, the unauthorized poll, and the state the table is left in afterwards are all real output. No job was created for this article, so the create response, the state transitions and the artifact URLs are read from the API's source rather than from a run of my own. If your client disagrees with something here, your run wins.

A few limits worth having in view before you build: one hour of media per job, one target language per job (with target_langs fanning out into several jobs), sources by link only as a direct media URL or an HLS playlist, and transcription billed at half the dubbing rate from the same balance.

Create a key under Account, then read the API reference for the field-by-field contract. If you want the small version first, the ten-line quickstart submits, polls and downloads in one screenful, and the pipeline walkthrough explains what those minutes are actually buying between queued and done.