RedNote API Best Practices: Rate Limits, Retries, Pagination & Dedup

Taking RedNote (Xiaohongshu) data collection from "it runs" to "stable at scale" comes down to a few engineering details: rate limiting, retries, pagination, deduplication, and cost control. Here's a copy-paste-ready set of best practices for use with the Rnote API.

1. Rate limiting & concurrency

Cap concurrency and throttle your requests instead of maxing out immediately. On 429 (rate limited), back off and retry — a smooth request curve is more stable and less stressful than bursty floods.

2. Retries & backoff

Retry only on network errors and 5xx, using exponential backoff (1s, 2s, 4s…). The good news: the Rnote API bills only successful (HTTP 2xx) requests, so retries on failures cost nothing — add retry logic with confidence.

3. Pagination: cursors differ by endpoint

Always carry the cursor fields from the previous response into the next request to keep pagination consistent.

4. Deduplication

Across pages, sort orders, and jobs you'll see duplicates. Deduplicate by note_id / user_id (a set or a unique DB index) — it saves credits and keeps your data clean.

5. Cost & budget control

• Only successful requests are billed — failures are free.
• Validate your logic and fields on a small batch first, then scale.
• Set a budget cap per job so a misbehaving script can't run away.

A robust paginated collection skeleton (Python)

import time, requests

API = "https://rnote.dev/api/v2/crawler"
H = {"X-API-Key": "YOUR_API_KEY"}

def get(path, params, retries=3):
    for i in range(retries):
        r = requests.get(f"{API}/{path}", params=params, headers=H, timeout=30)
        if r.status_code == 200:
            return r.json()
        if r.status_code == 429:
            time.sleep(2 ** i)          # rate-limit backoff
        elif r.status_code >= 500:
            time.sleep(2 ** i)          # server-error backoff
        else:
            r.raise_for_status()        # don't retry 4xx
    raise RuntimeError("retries exhausted")

def search_all(keyword, max_pages=10):
    seen = set()
    for page in range(1, max_pages + 1):
        data = get("search/notes", {"keyword": keyword, "page": page})
        items = extract_items(data)     # pull the list per response shape
        if not items:
            break
        for it in items:
            nid = it["note_id"]
            if nid not in seen:         # dedup
                seen.add(nid)
                yield it

6. Error classes: what to retry and what not to

Backoff only helps once you've decided the error is worth retrying. The cost of getting this wrong is asymmetric: failing to retry something retryable loses data, while retrying something hopeless burns quota and buries your logs.

The third class is where people slip. Treating 402 (insufficient balance) as an ordinary failure is the classic case — the balance does not grow because you retried, and you end up with tens of thousands of useless requests hiding the real problem. Those two codes should raise, not continue.

7. Idempotency and resumability

Batch jobs will be interrupted: network, deploy, an accidental Ctrl+C. Making them resumable takes one principle — persist "done" to disk, not to memory:

import pathlib

state = pathlib.Path("done.txt")
done = set(state.read_text().split()) if state.exists() else set()

with state.open("a") as f:
    for note_id in all_ids:
        if note_id in done:       # already fetched — skip, don't pay again
            continue
        if not process(note_id):  # leave failures for round two
            continue
        f.write(note_id + "\n")
        f.flush()                 # without this, a kill loses the buffered entries

That flush() is the line most often dropped, and it fails at exactly the moment it was needed.

Pair it with two passes: run everything in pass one, recording failures without retrying; run only the failure list in pass two. Pass one never stalls behind a few stubborn items, and pass two tends to succeed because the rate-limit window has moved on and upstream has recovered. Whatever still fails is usually genuinely gone (note deleted, account closed) — move it to a "confirmed unavailable" list rather than looping on it forever.

8. Separate keys for separate jobs

One key for every environment is the most expensive habit a team can pick up: one runaway test script eats the production quota and rate limit together. Rate limits are per key and independent, so splitting them is itself a fuse:

• prod — production services, full limit
• staging — pre-release, half
• dev-<name> / batch — development and batch jobs, lowest

The value shows up on the bad day: a leaked or misused key gets disabled on its own while everything else keeps running. More in managing API keys.

Get started

Drop these practices into your collection jobs and they'll run reliably at volume with little change. Sign up free for an API key, get going with the Python / Go tutorials, and see the API docs and pricing.