kiln

Background jobs for Go, kept in PostgreSQL, MySQL, SQL Server or SQLite.

kiln runs background jobs for Go programs, the way Hangfire does for .NET. Jobs are rows in the database you already have, so they survive restarts and crashes, they can be enqueued in the same transaction as the data that produced them, and you can watch and retry them from a dashboard that comes with the library.

• Retries with exponential or custom backoff, snoozes, timeouts and permanent failures
• Workflows: jobs that wait for one or many other jobs and read their outputs, and batches, nested if needed, with a job that runs when the whole batch is done
• Limits per key, across every server: how many jobs run at once and how many start per second
• Recurring jobs from cron specs, with time zones, a policy for missed runs, and SyncRecurring to keep them in step with your code
• Job console: log lines and a progress bar from inside a handler, live in the dashboard
• Unique jobs, while a job is live or for a window of time, with replace and debounce
• Transactional enqueue: the job exists only if your transaction commits
• Cancellation of a running job from any process
• Dashboard in English or Brazilian Portuguese, and a JSON API, mounted on your own HTTP server
• OpenTelemetry traces from the request that enqueued a job to the handler that ran it
• PostgreSQL, MySQL, SQL Server and SQLite, plus an in-memory store for tests, all held to one conformance suite

Everything above is in this repository, under the MIT license. There is no paid edition.

Where to start

• Getting started: a working program in five minutes, on SQLite or PostgreSQL
• How it works: what a job is, and how servers share the work
• Concepts: states, enqueueing, retries, workflows, limits, recurring jobs and more
• Storage backends: what each database needs, and how to write your own store
• Coming from Hangfire: Hangfire calls and their kiln equivalents

Status

kiln runs in production at Sodexo, Zeep Labs and Starbem. If your team uses it too, open a pull request to add yourself here.

kiln is young, and its API can still change between minor versions before 1.0. Every change is listed in the changelog, and each release is tested against the previous one running on the same database, so a rolling upgrade from one version to the next keeps working.