KEEP GOOD DATA FLOWING

Connect your collection pipeline.

Generate a scoped ingestion key in your dataset settings. Use it only in your server or scraper environment.

A stable record contract

{"id":"profile-123","followers":12800,"_version":2,"_op":"upsert"}

The primary key identifies the entity or event. Increase the integer _version when its source changes. A repeated identity and version must have identical business content. Use delete for a tombstone or correction for a free repair. Older source versions cannot overwrite newer data.

Connect in Python or JavaScript

Download the Python client or Node.js client. Both upload in bounded chunks, resume from a local checkpoint and wait for full validation. Keep the checkpoint to resume; remove it when deliberately starting a new source run. Set DATATAP_URL and DATATAP_INGEST_KEY in your server environment.

python ingest.py DATASET_ID data.ndjson --format ndjson
node ingest.mjs DATASET_ID data.ndjson ndjson

# After a successful source check with no changes:
python ingest.py DATASET_ID --no-changes

Corrections require a descriptive _correction_reason. CSV uses \\N for null; an empty string remains an empty string. Dates use ISO dates, timestamps include a timezone, and large identifiers or exact decimals should be strings. Your first completed upload is the connection test; check the job result before scheduling another run.

Resumable upload protocol

  1. POST /api/v1/seller/datasets/:id/uploads with action: start, file bytes, format and a UUID idempotency key.
  2. Request each signed part URL with action: part, upload ID and part number. PUT the file chunk to that URL and keep its ETag.
  3. POST action: complete, upload ID, and the ordered part numbers and ETags.
  4. The response contains a job ID. Full validation must finish before the new version is available.

Send Authorization: Bearer YOUR_INGESTION_KEY on the API requests. Each key can upload only to its assigned dataset. Accepted formats: CSV, NDJSON, JSON arrays and Parquet.

Limits

  • Each upload: up to 5 GiB and 4,000,000 rows.
  • Each dataset: up to 50,000,000 stored rows.
  • Each dataset: up to 24 uploads in any 24 hours, one processing at a time.
  • Buyer and seller refresh intervals: at least one hour.
  • Processing pauses new uploads while 20 jobs or 10 GiB are waiting; retry later.

An upload over a limit is refused with HTTP 413, 429 or 503, a limit code and, where waiting helps, a Retry-After header. Rows over the per-upload or per-dataset limit fail validation with the reason. Higher-rate feeds: Upload a batch file (Parquet is smallest) at most once an hour instead of streaming records. Aggregate or sample high-rate events before upload, for example per-minute counts. Split very large histories across datasets, such as one per year or region. Describe the feed for direct supply, where buyers receive data from your systems through a datap proxy. Describe a high-volume feed.

A successful check with no changes

POST https://datap.com/api/v1/ingest/DATASET_ID/check
Authorization: Bearer YOUR_INGESTION_KEY
Content-Type: application/json

{"key":"A_NEW_UUID_FOR_THIS_RUN","result":"no_changes"}

Send this after your scraper successfully checks its source and finds no changes. A process keepalive does not prove a source check. datap tracks source contact, successful checks and material changes separately.

Simple freshness rules

A source becomes overdue after its expected interval plus the greater of one hour or 25% of that interval. A weekly source has 8 days and 18 hours. Monitoring can be paused without deleting data or cancelling buyer access.