Intermediate
You have a Python function that needs to run every hour. Maybe it fetches data from an API, sends a digest email, or clears out expired records in your database. Right now you are probably doing one of two things: running it manually whenever you remember, or wrestling with a cron job that works on your server but silently fails when your app restarts. There is a better way — one that lives inside your Python app, restarts with it, and gives you visibility into whether tasks actually ran.
That better way is celery-beat, the periodic task scheduler built on top of Celery. If you have already used Celery for background jobs, adding beat is straightforward: you define schedules in Python code, start the beat process alongside your worker, and never think about cron syntax again. If Celery is new to you, this article walks you through everything from installation to a working scheduler. You will need Python 3.8+, Redis (the simplest broker to set up locally), and about 15 minutes.
This article covers installation and configuration, defining tasks with @shared_task, setting schedules with crontab and timedelta, running the beat and worker processes, dynamic schedules stored in a database, a real-life example that ties it all together, and common pitfalls to avoid. By the end you will have a working periodic task pipeline you can adapt for any automation need.
celery-beat Quick Example
Here is the minimal setup that prints a timestamp every 30 seconds. We will go deep on each piece afterward — this is just to give you a win fast.
# celery_app.py
from celery import Celery
from celery.schedules import timedelta
app = Celery("quickstart", broker="redis://localhost:6379/0")
app.conf.beat_schedule = {
"print-every-30s": {
"task": "celery_app.print_timestamp",
"schedule": timedelta(seconds=30),
},
}
@app.task
def print_timestamp():
from datetime import datetime
print(f"[beat] tick at {datetime.now().isoformat()}")
Start Redis, then open two terminals:
# Terminal 1 -- the worker that runs tasks
celery -A celery_app worker --loglevel=info
# Terminal 2 -- the beat scheduler that sends tasks to the worker
celery -A celery_app beat --loglevel=info
Output (in the worker terminal, every 30 seconds):
[beat] tick at 2026-07-10T08:00:00.123456
[beat] tick at 2026-07-10T08:00:30.124512
[beat] tick at 2026-07-10T08:01:00.125103
The two key things here: beat_schedule is a dict where each entry names a scheduled job, and the beat process is separate from the worker. Beat is a clock — it sends tasks to the queue. The worker receives them and executes them. This separation is intentional and matters when you scale.
What Is celery-beat and Why Use It?
Celery is a distributed task queue — you push work onto a queue (backed by Redis or RabbitMQ) and worker processes pick it up and execute it asynchronously. Celery is for “do this now, in the background.” celery-beat adds “do this on a schedule.” Think of beat as the cron daemon for your Celery workers: it wakes up at the right time, publishes a task message to the queue, and the worker picks it up exactly as if you had triggered it manually.
Why choose celery-beat over cron?
| Feature | cron | celery-beat |
|---|---|---|
| Schedule defined in | crontab file (server-specific) | Python code (version-controlled) |
| Restarts with app | No (server-level) | Yes (process alongside worker) |
| Dynamic schedules | No | Yes (via database-backed scheduler) |
| Task result tracking | No | Yes (via Celery result backend) |
| Retry on failure | No | Yes (Celery retry mechanism) |
| Works across multiple servers | Requires coordination | Yes (one beat + many workers) |
The trade-off is setup cost: you need a broker (Redis or RabbitMQ) running. For single-server scripts that rarely change, cron is perfectly fine. For anything you are deploying as an application — especially if it already uses Celery for async tasks — celery-beat is the obvious choice.
Installation and Setup
Install Celery with the Redis transport. The [redis] extra pulls in the redis Python client:
# install_celery.sh -- run in your terminal
pip install "celery[redis]"
Output:
Successfully installed celery-5.4.0 redis-5.0.4 kombu-5.3.7 ...
Start Redis locally. If you have Docker:
# start_redis.sh
docker run -d -p 6379:6379 redis:7-alpine
Verify Redis is reachable:
# check_redis.py
import redis
r = redis.Redis(host="localhost", port=6379, db=0)
print(r.ping()) # True if Redis is up
Output:
True
Good. Redis is now ready to act as the Celery broker — the message bus that carries tasks from beat to worker.
Configuring Celery for a Real Project
In a real project you want your Celery configuration in a dedicated file, not scattered across scripts. The conventional layout uses a celery.py file at the same level as your app package. Here is a production-style config:
# myproject/celery.py
import os
from celery import Celery
from celery.schedules import crontab, timedelta
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
app = Celery(
"myproject",
broker="redis://localhost:6379/0",
backend="redis://localhost:6379/1", # stores task results
include=["myproject.tasks"], # modules to scan for tasks
)
app.conf.update(
task_serializer="json",
result_serializer="json",
accept_content=["json"],
timezone="Australia/Melbourne",
enable_utc=True,
beat_schedule_filename="celerybeat-schedule",
)
app.conf.beat_schedule = {
# Run every 5 minutes
"cleanup-temp-files": {
"task": "myproject.tasks.cleanup_temp_files",
"schedule": timedelta(minutes=5),
"args": ("/tmp/uploads",),
},
# Run at 7:00 AM Melbourne time, Monday through Friday
"send-morning-digest": {
"task": "myproject.tasks.send_digest_email",
"schedule": crontab(hour=7, minute=0, day_of_week="mon-fri"),
},
# Run on the 1st of every month at midnight
"monthly-report": {
"task": "myproject.tasks.generate_monthly_report",
"schedule": crontab(day_of_month=1, hour=0, minute=0),
},
}
Output (no output — this is a configuration module imported by Celery):
(module loaded successfully when imported by Celery CLI)
Notice beat_schedule_filename: beat writes a small file to track when each task last ran. This lets it survive restarts without re-firing tasks that already ran. Delete this file if you want beat to reset its schedule state.
Defining Periodic Tasks
Tasks are plain Python functions decorated with @app.task or @shared_task. Use @shared_task when your tasks live in a reusable app and should not import the Celery instance directly — this avoids circular import issues in larger projects:
# myproject/tasks.py
import os
import glob
from celery import shared_task
from datetime import datetime
@shared_task(name="myproject.tasks.cleanup_temp_files")
def cleanup_temp_files(directory):
now = datetime.now().timestamp()
removed = 0
pattern = os.path.join(directory, "*")
for path in glob.glob(pattern):
try:
age_seconds = now - os.path.getmtime(path)
if age_seconds > 3600: # older than 1 hour
os.remove(path)
removed += 1
except (OSError, PermissionError) as e:
print(f"Skipping {path}: {e}")
print(f"[cleanup] Removed {removed} old files from {directory}")
return removed
@shared_task(name="myproject.tasks.send_digest_email")
def send_digest_email():
print(f"[digest] Sending morning digest at {datetime.now():%Y-%m-%d %H:%M}")
# In production: connect to SMTP or use SendGrid/SES here
return "sent"
@shared_task(name="myproject.tasks.generate_monthly_report")
def generate_monthly_report():
month = datetime.now().strftime("%B %Y")
print(f"[report] Generating report for {month}")
return f"report-{month}.pdf"
Output (tasks run when triggered by beat):
[cleanup] Removed 3 old files from /tmp/uploads
[digest] Sending morning digest at 2026-07-10 07:00
[report] Generating report for July 2026
The name parameter in @shared_task is important: it must exactly match the string you put in beat_schedule. If they do not match, beat will silently queue tasks that never get picked up because no worker recognizes the task name.
Schedule Types: timedelta vs crontab
celery-beat offers two ways to express schedules. Here is when to use each:
# schedule_examples.py
from celery.schedules import crontab, timedelta
# timedelta -- interval-based ("every N units")
every_10_seconds = timedelta(seconds=10)
every_2_hours = timedelta(hours=2)
every_3_days = timedelta(days=3)
# crontab -- calendar-based ("at specific times")
every_minute = crontab() # * * * * *
every_hour = crontab(minute=0) # 0 * * * *
weekdays_9am = crontab(hour=9, minute=0,
day_of_week="mon-fri") # 0 9 * * 1-5
first_of_month = crontab(day_of_month=1,
hour=0, minute=0) # 0 0 1 * *
print("timedelta example:", every_2_hours)
print("crontab example:", weekdays_9am)
Output:
timedelta example: 2:00:00
crontab example: <crontab: 0 9 * * 1-5 (m/h/dM/MY/d)>
Use timedelta when you care about frequency (“run every hour”) and crontab when you care about calendar time (“run at 9 AM on weekdays”). A gotcha: timedelta(hours=1) fires every 60 minutes from when beat last sent the task — it drifts slightly. crontab(minute=0) fires on the hour, every hour, regardless of drift.
Running Beat and Worker
You need two processes running simultaneously. The worker handles task execution; beat handles scheduling. A common mistake is trying to run them together — celery-beat supports an embedded mode (-B flag) for development convenience, but it is not safe for production because a crashed worker also kills your scheduler.
Development (embedded beat, one terminal):
# dev_start.sh -- convenient for development, NOT for production
celery -A myproject.celery worker --beat --loglevel=info
Production (two separate processes):
# prod_worker.sh -- run in Terminal 1
celery -A myproject.celery worker --loglevel=info --concurrency=4
# prod_beat.sh -- run in Terminal 2
celery -A myproject.celery beat --loglevel=info --scheduler celery.beat.PersistentScheduler
Output (beat terminal):
celery beat v5.4.0 is starting.
LocalTime -> 2026-07-10 08:00:00
Configuration ->
. broker -> redis://localhost:6379/0
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. maxinterval -> 5.00 minutes (300s)
beat: Starting...
The maxinterval (5 minutes by default) is how often beat wakes up to check if any task is due. If you have tasks scheduled every 10 seconds, lower this: --max-interval=10. Otherwise beat might sleep for 5 minutes and miss triggers.
Dynamic Schedules with django-celery-beat
So far our schedule is defined in Python code — change it and you have to restart beat. For schedules that need to change at runtime (user-defined reminders, configurable intervals), use django-celery-beat, which stores the schedule in your database and checks for changes dynamically.
# Install the extension
# pip install django-celery-beat
# settings.py -- add to INSTALLED_APPS
INSTALLED_APPS = [
"django_celery_beat", # add this line
# ... your other apps
]
# Run migrations to create the schedule tables
# python manage.py migrate
# create_schedule.py -- run from Django shell or management command
from django_celery_beat.models import PeriodicTask, CrontabSchedule
import json
# Create a crontab schedule: every day at 3:00 AM
schedule, _ = CrontabSchedule.objects.get_or_create(
minute="0",
hour="3",
day_of_week="*",
day_of_month="*",
month_of_year="*",
)
# Create the periodic task record
task, created = PeriodicTask.objects.get_or_create(
name="nightly-data-sync",
defaults={
"task": "myproject.tasks.sync_data",
"crontab": schedule,
"args": json.dumps([]),
"kwargs": json.dumps({"source": "production_db"}),
"enabled": True,
},
)
print(f"Task {'created' if created else 'already exists'}: {task.name}")
print(f"Schedule: {task.crontab}")
Output:
Task created: nightly-data-sync
Schedule: <CrontabSchedule: 0 3 * * * (m/h/dM/MY/d)>
Update the celery.py to use the database scheduler:
# myproject/celery.py -- update the scheduler setting
app.conf.update(
beat_scheduler="django_celery_beat.schedulers:DatabaseScheduler",
)
With the DatabaseScheduler, beat polls the database every few seconds and picks up schedule changes without a restart. Django admin shows a clean UI for your periodic tasks out of the box — no code deploy needed to add or disable a scheduled job.
Real-Life Example: Automated API Price Monitor
This example polls a public API every minute, logs the latest data, and sends a simple alert if a value crosses a threshold. It uses jsonplaceholder.typicode.com as the data source so you can run it right now without any API keys. Swap in a real finance or monitoring API endpoint for production use.
# monitor/tasks.py
import requests
from celery import shared_task
from datetime import datetime
# Simple in-memory alert tracker; use Redis or a DB in production
_alert_sent = {}
@shared_task(name="monitor.tasks.check_metric",
bind=True,
max_retries=3,
default_retry_delay=10)
def check_metric(self, record_id, threshold):
try:
response = requests.get(
f"https://jsonplaceholder.typicode.com/posts/{record_id}",
timeout=5
)
response.raise_for_status()
data = response.json()
# Simulate a metric from the post ID field
metric_value = data.get("id", 0) * 2.75
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
print(f"[{timestamp}] Record #{record_id}: value={metric_value:.2f}")
# Fire alert once if value crosses threshold
if metric_value > threshold and not _alert_sent.get(record_id):
print(f" ALERT: {metric_value:.2f} crossed threshold {threshold:.2f}")
_alert_sent[record_id] = True
elif metric_value <= threshold:
_alert_sent[record_id] = False # reset when value drops
return {"id": record_id, "value": metric_value, "ts": timestamp}
except requests.RequestException as exc:
print(f"[check_metric] Request failed: {exc}")
raise self.retry(exc=exc)
# monitor/celery.py
from celery import Celery
from celery.schedules import timedelta
app = Celery("monitor", broker="redis://localhost:6379/0")
app.conf.include = ["monitor.tasks"]
app.conf.beat_schedule = {
"check-record-42-every-minute": {
"task": "monitor.tasks.check_metric",
"schedule": timedelta(minutes=1),
"args": (42, 50.0), # record_id=42, threshold=50.0
"options": {"expires": 55}, # drop if worker down >55s
},
"check-record-7-every-5min": {
"task": "monitor.tasks.check_metric",
"schedule": timedelta(minutes=5),
"args": (7, 25.0),
},
}
Output (worker terminal, running for 3 minutes):
[2026-07-10 08:00:01] Record #42: value=115.50
ALERT: 115.50 crossed threshold 50.00
[2026-07-10 08:01:01] Record #42: value=115.50
[2026-07-10 08:05:01] Record #7: value=19.25
[2026-07-10 08:05:01] Record #42: value=115.50
Key patterns worth keeping from this example: bind=True gives the task access to self for retries; max_retries=3 with default_retry_delay=10 automatically retries on network failure; options: {expires: 55} prevents a pile-up of stale tasks if your worker goes offline. Extend this by writing values to a PostgreSQL table, sending real alerts via Twilio or SendGrid, or plotting a rolling chart with matplotlib.
Frequently Asked Questions
Can I run multiple beat instances?
No -- run exactly one beat process per Celery application. Running two beat instances causes duplicate tasks: both instances send the same task at the same time and your workers execute it twice. If you need high availability, use django-celery-beat with a database-backed scheduler and a distributed lock (such as celery-redlock) to ensure only one beat fires at a time. For most applications, a single beat process managed by supervisord or systemd is sufficient and reliable.
What happens if beat was offline and missed a scheduled run?
celery-beat does not backfill missed runs by default. If beat was offline for 2 hours and you have an hourly task, those 2 missed runs are skipped -- beat resumes from the current time when it restarts. If you need catch-up behavior (the task must run even if it was missed), implement it inside the task itself: check the last successful run timestamp from your database and re-run for each missed interval. The beat_schedule_filename pickle file tracks last-run times but does not trigger catch-up execution automatically.
How do I prevent a task from running again before the previous run finishes?
celery-beat does not know when a task finishes -- it only knows when to send it. Use a distributed lock to prevent overlap. The simplest approach is celery-singleton, which raises Ignore() if the same task is already running. Alternatively, use Redis directly: if not redis_client.set("lock:my-task", 1, nx=True, ex=300): raise Ignore(). The ex=300 sets a 5-minute TTL so the lock auto-expires if the task crashes without releasing it.
My crontab task fires at the wrong time. What is wrong?
Set timezone and enable_utc=True explicitly in your Celery config. By default Celery uses UTC; if your crontab(hour=9) is meant for 9 AM local time, it will fire at 9 AM UTC instead. Set app.conf.timezone = "Australia/Melbourne" (or your local timezone) and Celery will interpret crontab times in that timezone. Verify with celery -A myapp inspect scheduled -- it shows the next scheduled run time in UTC so you can sanity-check the conversion.
My periodic task fails occasionally. How do I add retries?
Decorate with bind=True and call self.retry(exc=exc) inside the exception handler, as shown in the real-life example above. Set max_retries and default_retry_delay on the decorator. For exponential backoff, pass countdown=2 ** self.request.retries to self.retry(): this waits 1s, then 2s, then 4s between retries. Be careful: if a 1-minute task fails and retries 3 times with increasing delays, the final retry could land after the next scheduled run has already started.
How do I monitor which tasks ran and whether they succeeded?
Set a result backend (backend="redis://localhost:6379/1") and Celery stores task results keyed by task ID. Use celery -A myapp events for a live event stream, or run celery -A myapp flower (install flower separately) for a web dashboard that shows task history, run times, and failure rates. For production, integrate with your monitoring stack: Celery can emit metrics to Datadog, Prometheus (via celery-prometheus-exporter), or any StatsD-compatible collector.
Conclusion
celery-beat turns Celery from a "do this now" system into a "do this now and also every night at 3 AM" system. The core concepts are straightforward: configure a broker, define tasks with @shared_task, declare a beat_schedule using timedelta or crontab, and run the beat and worker processes separately. For schedules that need to change without a deployment, django-celery-beat's database scheduler gives you admin-editable periodic tasks with zero code changes.
Extend the API monitor from this article by wiring in a real data source (the yfinance library is a free option for stock prices), storing readings in a PostgreSQL table via SQLAlchemy, and sending real alerts via SMTP or a service like SendGrid. Once that is running you will have a complete scheduled data pipeline -- periodic fetch, persist, alert -- all managed by celery-beat with automatic retries and failure visibility.
For more detail on Celery configuration and advanced task options, see the official celery-beat documentation and the Celery task guide.