---
name: datap-keep-dataset-updated
description: Use when a seller needs to refresh a datap dataset on a schedule, asks where to run a collector, or wants to cut the cost of keeping data fresh. Ranks hosting options and sets up scheduled uploads.
---

# Keep a dataset updated

Buyers pay more for data that arrives on time. datap marks a dataset overdue when a check is later than its expected interval plus the larger of one hour or a quarter of that interval, and emails the seller.

## The run, whatever hosts it
1. Collect and validate. Empty or partial results are a failure.
2. If rows changed, upload them with the ingestion key (https://datap.com/docs/ingestion). The same primary key updates a row; new keys add rows.
3. If nothing changed, report that so the dataset stays fresh: POST https://datap.com/api/v1/ingest/DATASET_ID/check with {"key": "<new uuid>", "result": "no_changes"}.
4. Exit non-zero on failure so the scheduler can alert.
Use a plain script on a timer. Do not put a model in the loop for routine runs: it costs money every run and fails in new ways. Have the agent write and test the script once.

## Where to run it, best first
1. **A hosted scheduler** (GitHub Actions on a cron, a Cloudflare Worker cron trigger, or a cloud scheduler with a small function). Usually free at this scale, nothing to keep switched on, logs and failure emails included. Best for API calls and light fetching that finish in minutes. Limits: start times can drift by several minutes, some schedulers pause on inactive repositories, and shared IP addresses are blocked by some sites.
2. **A small always-on server** (a basic VPS, about the price of a coffee a month) with a systemd timer or cron. Best when the job needs a browser, runs long, keeps state between runs or needs a steady IP. Add a dead-man alert: a free uptime service that expects a ping after each successful run.
3. **An agent product with its own always-on machine or scheduled tasks.** Easiest to set up if the person already pays for one. Have it run the script, not re-reason the task each time, or token costs will exceed the server it replaces. Check how it stores the ingestion key.
4. **The person's own computer** (launchd on macOS, Task Scheduler on Windows, cron or a systemd user timer on Linux). Free, and fine for weekly or monthly data. Set it to run at wake or login as well as on the timer, so a missed run is caught up. A laptop that sleeps cannot promise hourly or daily freshness.

Choose the highest option the job fits. Move down the list only for a reason: a browser, a long run, a blocked IP.

## Keep it cheap
- Fetch only what changed: use since-parameters, ETags, sitemaps or feeds before full crawls.
- Match the schedule to how often the source really changes.
- Upload changed rows only, and send "no_changes" otherwise.
- Cap runtime and spend per run, and stop on repeated failures instead of retrying all night.

## Tell datap the promise
`datatap_set_update_schedule` sets the expected interval. Promise what the setup can keep: a daily job on a hosted scheduler can promise daily; a laptop should promise weekly. `datatap_get_my_dataset` shows the last successful check and recent ingestion jobs. Pause monitoring with the same tool during planned downtime.

Feeds larger than the published limits (`datatap_get_ingest_limits`): https://datap.com/provider/ingest

These steps use the datap connector (https://datap.com/mcp/private). Setup: https://datap.com/docs/agents
