How to poll a crypto market API without hitting rate limits: ETag, gzip, filters and retries after 429
In short
- An API usually has three limits: requests per second, a call quota per period and download volume. On Genesis API that is 1 to 15 requests per second, 500K to 10M calls per 30 days and 60 MB of compressed data per 10 seconds per key.
- Set the polling interval from
meta.freshness_sec: polling faster returns the same snapshot.If-None-Matchturns a repeat into a free304, gzip and filters cut traffic, and one full dataset instead of a request per coin saves calls. - Retry
429and503afterRetry-Afterseconds or, without that header, with exponential backoff plus random jitter. A client-side token bucket keeps your pace under the limit, and theX-RateLimit-*andX-Quota-*headers show the headroom left.
To poll an API without 429 errors, shape your requests around three limits at once: requests per second, the call quota for the period and the volume of data you download. Five techniques cover it: an interval no shorter than the data refresh, conditional requests with ETag, gzip compression, filters instead of oversized downloads, and retries after the pause the server asks for. The numbers and code below are for Genesis API, a REST API for crypto market data, but the techniques work with any HTTP API, exchange APIs included.
Three API limits: requests per second, quota and volume
A per-second rate limit catches bursts: a batch of requests sent at once. The quota catches your average pace over the whole period, and the volume limit catches heavy downloads even when there are few requests.
| Limit | What it counts | Genesis API | Over the limit |
|---|---|---|---|
| Requests per second | Every request, including 304 and errors | 1 on Trial, 2 on Start, 5 on Pro, 15 on Max | 429 rate_limited, Retry-After: 1 |
| Quota per period | Only 200 responses | 500K, 2M, 5M, 10M calls per 30 days | 429 rate_limited, Retry-After until the quota resets |
| Download volume | Compressed responses of 64 KB and up | 60 MB per 10 seconds per key, on every plan | 429 download_limited until the window ends |
Genesis API never bills requests over a limit: a 429 uses no quota and costs nothing. All the rules are in the Genesis API limits guide and in the Plans & limits section of the docs.
Exchange APIs can be more involved. Binance counts request weight per IP address, and every endpoint has its own weight. A client that keeps sending requests after a 429 gets an automated IP ban with status 418, and repeat offenders get longer bans (Binance docs).
Polling interval: how many calls you use in 30 days
Polling faster than the data refreshes brings nothing but repeats. The Genesis Arbitrage core refreshes data every second, while the API output for your plan refreshes once every meta.freshness_sec seconds: 60 on Trial, 30 on Start, 10 on Pro and 5 on Max. That is your plan's data freshness and the floor for your polling interval. Genesis API has no WebSocket or webhooks, so you get data by polling over REST.
Illustrative example: a bot on the Start plan — $50 a month, 2M calls, 2 requests per second, 30 s freshness — polls four endpoints: /arbitrage/spot-spot, /arbitrage/spot-futures, /funding/spread and /market/open-interest.
| Interval | Calls per 30 days | Share of Start quota | Repeats of one snapshot |
|---|---|---|---|
| 60 s | 172,800 | 8.64% | none |
| 30 s | 345,600 | 17.28% | none |
| 10 s | 1,036,800 | 51.84% | 2 of 3 |
| 5 s | 2,073,600 | 103.68% | 5 of 6 |
Polling every 5 seconds runs the quota out on day 29: 4 × 86,400 / 5 = 69,120 calls a day, and 2,000,000 / 69,120 ≈ 28.9 days. Five of every six responses repeat one snapshot with the same meta.updated_at, and without If-None-Match each of them costs a call.
At one poll every 30 seconds the average pace is 4 / 30 ≈ 0.13 requests per second. But four requests sent at once land in the same second, and two of them get a 429. If you need data fresher than 30 seconds, the answer is a plan with a lower freshness_sec: Pro refreshes the snapshot every 10 seconds, Max every 5.
Conditional requests: ETag, If-None-Match and 304
An ETag is a version tag on a response. The client stores it and sends it back in the If-None-Match header of its next request to the same URL; if the version has not changed, the server replies 304 Not Modified. Under RFC 9110 a 304 ends after the headers and carries no body, so you use your own copy of the data.
On Genesis API a 304 uses no quota but still takes a slot in the requests-per-second limit. Only full datasets without filters get a tag. Responses to requests with coin, exchange, min_spread or other filters, as well as /orderbooks/book and /orderbooks/coin/{coin}, are built per request and come without an ETag.
ETag saves quota where data changes less often than you ask for it. The tag on spreads changes with every snapshot, so when you poll once per freshness_sec, almost every response is a 200.
Illustrative example on Pro: the daily sums of funding analytics, /funding/analytics/daily, are regenerated hourly; once a year of history has built up, the full section is about 12 MB gzip, and without coin it is served at most once per 300 seconds. Polling every 300 seconds makes 2,592,000 / 300 = 8,640 requests in 30 days. With If-None-Match the section downloads 30 × 24 = 720 times, and the other 8,640 − 720 = 7,920 responses come back as 304.
The result: 720 calls instead of 8,640, and about 8.6 GB of traffic instead of 104 GB. 304 responses do not count toward the full-download limit.
Filters and coin: what saves calls and what saves traffic
A filter shrinks the response but not the number of calls: the quota counts responses, and a request with coin=BTC costs as much as the full dataset. So the choice depends on which runs out first, calls or traffic.
Illustrative example on Start: a bot tracks funding for five coins. Five requests to /funding/rates with coin every 30 seconds come to 5 × 86,400 = 432,000 calls in 30 days, plus five requests per cycle against a 2-per-second limit. One full /funding/rates with the coins picked out in code costs 86,400 calls, five times fewer, and it gets an ETag.
| Task | How to request | What it saves |
|---|---|---|
| Several coins from one dataset | Full dataset with If-None-Match, filter in code | Calls: one request instead of several |
| One coin from a heavy section | The coin parameter | Traffic and the volume limit |
| Rows above a threshold | min_spread, min_volume | Traffic, but every poll is a call |
| A slow section in full | Full dataset with If-None-Match | Calls and traffic |
The heaviest sections belong to funding analytics: the full summary is about 600 KB gzip, the 7-day payouts about 1 MB and the daily sums up to 12 MB (Funding endpoints). A single-coin slice with coin is exempt from the full-download limit, which is once per 60 seconds for the summary and payouts and once per 300 for the daily sums.
Gzip compression
Compression cuts the number of bytes on the wire. The client asks for it with Accept-Encoding: gzip, and the server replies with a compressed body and a Content-Encoding: gzip header. Python's requests adds gzip to Accept-Encoding and decompresses the response on its own; curl does the same with --compressed.
On Genesis API compression is tied to a limit: download volume is counted in compressed data, 60 MB per 10 seconds per key, and responses under 64 KB are not counted. Full datasets and large responses come gzipped when the client asks for it (Genesis API responses and errors). To see how many body bytes actually crossed the network, use curl's size_download variable:
# body bytes on the wire with and without gzip; each request is a call
URL="https://api.genesisarbitrage.com/v1/funding/rates"
curl -s -o /dev/null --compressed -w "gzip: %{size_download}\n" \
-H "X-API-Key: $GENESIS_API_KEY" "$URL"
curl -s -o /dev/null -w "uncompressed: %{size_download}\n" \
-H "X-API-Key: $GENESIS_API_KEY" "$URL"
Retrying after 429: Retry-After and backoff with jitter
HTTP 429 Too Many Requests is the server's answer to a client that went over a limit. Under RFC 6585 the response may carry a Retry-After header, and under RFC 9110 its value is either a number of seconds or a date. Genesis API sends Retry-After with every 429 and 503, in seconds.
A retry without a pause nearly always gets the same error and takes another slot in the per-second window. If there is a Retry-After, wait as long as the server asks. If there is not, wait exponentially longer after each failure: up to 1, 2, 4, 8 seconds.
Jitter, a random addition to the pause, matters when there are many clients: without it, everyone who got an error in the same second retries in the same second too. The backoff function picks a random pause between zero and the current step and adds a fraction of a second to Retry-After. fetch retries 429, 500 and 503, and stops when the pause is longer than 10 minutes: that is what an exhausted quota looks like.
import os
import random
import time
import requests
BASE = "https://api.genesisarbitrage.com/v1"
api = requests.Session() # reuses the TCP connection, gzip is on
api.headers["X-API-Key"] = os.environ["GENESIS_API_KEY"]
cache = {} # path -> (ETag, data): full datasets only
def backoff(attempt, retry_after=None, cap=60.0):
"""Pause before a retry, in seconds."""
if retry_after and retry_after.isdigit():
return int(retry_after) + random.uniform(0, 1)
return random.uniform(0, min(cap, 2 ** attempt))
def fetch(path, params=None, tries=5):
"""GET from the API: returns (data, response headers)."""
for attempt in range(tries):
tag = None if params else cache.get(path, (None,))[0]
try:
r = api.get(BASE + path, params=params, timeout=15,
headers={"If-None-Match": tag} if tag else {})
except (requests.ConnectionError, requests.Timeout):
time.sleep(backoff(attempt))
continue
if r.status_code == 304: # same snapshot, no quota used
return cache[path][1], r.headers
if r.status_code in (429, 500, 503):
wait = backoff(attempt, r.headers.get("Retry-After"))
if wait > 600: # this is what an exhausted quota looks like
raise RuntimeError(f"quota resets in {wait:.0f} s")
time.sleep(wait)
continue
r.raise_for_status() # 400, 401, 403, 404, 410: do not retry
data = r.json()["data"]
if not params and "ETag" in r.headers:
cache[path] = (r.headers["ETag"], data)
return data, r.headers
raise RuntimeError(f"{path}: no response after {tries} tries")
| Response | Pause | Retry |
|---|---|---|
429 rate_limited, per second | Retry-After: 1 plus a fraction of a second | Yes |
429 rate_limited, quota | Until the quota resets, can be days | No |
429 download_limited | Retry-After, until the window ends | Yes, with narrower requests |
503 snapshot_not_ready | Retry-After | Yes |
500, timeout, dropped connection | Exponential with jitter | Yes |
400, 401, 403, 404, 410 | — | No, fix the request |
You tell an exhausted quota from the per-second limit by Retry-After: one second versus hours or days until the reset, with Monthly call quota exceeded. in the response body. Error codes and bodies are in the Errors section of the docs.
Client-side rate limiter: token bucket
A retry deals with a 429; a client-side limiter prevents it. A token bucket works like this: tokens arrive at rate per second, each request takes one, and an empty bucket makes you wait. The capacity, burst, sets how many requests can go out back to back without a pause.
import time
class TokenBucket:
"""At most rate requests per second, up to burst back to back."""
def __init__(self, rate, burst=1):
self.rate, self.burst = rate, burst
self.tokens = float(burst)
self.last = time.monotonic()
def take(self):
while True:
now = time.monotonic()
self.tokens = min(self.burst, self.tokens
+ (now - self.last) * self.rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return
time.sleep((1 - self.tokens) / self.rate)
Take the limit from the rps_limit field of /account/usage (the Usage section) or from the X-RateLimit-Limit header. Set the rate a little below the limit, for example 0.9 * rps_limit: network latency is uneven, so requests sent half a second apart can reach the server closer together.
Monitoring: the X-RateLimit-* and X-Quota-* headers
A data response carries the state of your limits: X-RateLimit-Remaining and X-RateLimit-Reset give what is left in the per-second window and when it resets, while X-Quota-Remaining, X-Quota-Limit and X-Quota-Reset give the remaining quota, its size and the unix time it resets. More useful than any single remainder is the pace: what share of the quota is gone at this point in the period.
PERIOD = 30 * 86400 # Genesis API counts quota per 30 days
def watch(h):
"""Checks limit headroom from the response headers."""
if h.get("X-RateLimit-Remaining") == "0": # window used up
reset = float(h.get("X-RateLimit-Reset", 0))
time.sleep(max(0.0, reset - time.time()))
try:
left = int(h["X-Quota-Remaining"])
limit = int(h["X-Quota-Limit"])
reset = int(h["X-Quota-Reset"])
except (KeyError, ValueError):
return # 429 responses carry no X-Quota-* headers
used = 1 - left / limit
passed = 1 - (reset - time.time()) / PERIOD
if used > passed + 0.05:
print(f"quota is running ahead: {used:.0%} used"
f" with {passed:.0%} of the period gone")
The function waits for the window to reset when its limit is used up, and warns when quota use runs ahead of the calendar by more than 5 percentage points. Illustrative example: 70% of the quota is gone by day 15, so at the same pace the calls run out 15 / 0.7 ≈ 21.4 days into the period. /account/usage gives the same numbers without headers: calls_used, calls_limit and period_resets_at.
A schedule for several endpoints
With several endpoints, each gets its own interval, and the first requests are spread out so they do not go out as a batch. Pick the interval for the task, but never below freshness_sec: spreads need your plan's full rate, while for many strategies a minute is enough for funding and open interest.
import heapq
usage, _ = fetch("/account/usage") # does not use a call
fresh = usage["freshness_sec"]
bucket = TokenBucket(rate=0.9 * usage["rps_limit"])
JOBS = { # path -> polling interval, s
"/arbitrage/spot-spot": fresh,
"/funding/spread": max(fresh, 60),
"/market/open-interest": max(fresh, 60),
}
t0 = time.monotonic()
queue = [(t0 + i * fresh / len(JOBS), path, every)
for i, (path, every) in enumerate(JOBS.items())]
heapq.heapify(queue)
while True:
due, path, every = heapq.heappop(queue)
time.sleep(max(0.0, due - time.monotonic()))
bucket.take()
data, hdrs = fetch(path)
watch(hdrs)
print(path, len(data))
heapq.heappush(queue, (due + every, path, every))
On Start that is 86,400 + 43,200 + 43,200 = 172,800 calls in 30 days, 8.64% of the quota: spreads every 30 seconds, funding and open interest every 60. The first requests go out 30 / 3 = 10 seconds apart, after which each endpoint keeps its own rhythm, and bucket.take() guards against collisions. A bot built on this polling, with Telegram alerts, is in Crypto arbitrage bot in Python.
What to do on 503 and server errors
Under RFC 9110, 503 Service Unavailable means a temporary overload or maintenance, and the server may suggest a pause in Retry-After. On Genesis API this is 503 snapshot_not_ready: the data is being rebuilt or its source is restarting. Retry after Retry-After seconds; the pause is usually a few seconds.
While there is no fresh response, your bot still holds the data from the last poll, and that data is aging. Before you send a signal, compare meta.updated_at with the current time: a pair from a minute-old snapshot may already be gone on the exchange. Other risks are covered in the Automation risks lesson.
Retry 500 internal_error, timeouts and dropped connections with exponential backoff. If the error persists, contact support with the X-Request-Id value, the request time in UTC, the endpoint and the status code. Never send your key.
Questions and answers
How many requests per second can I send to Genesis API?
As many as your plan allows: 1 on Trial, 2 on Start, 5 on Pro and 15 on Max. The limit is per key and includes 304 responses and errors. Your plan's value is in the X-RateLimit-Limit header and in the rps_limit field of /account/usage.
Do errors and 304 responses count against my quota?
No. Only a 200 response uses quota; 304, 429, 503 and other errors are free and cost nothing. They do take slots in the requests-per-second limit, though, so retries without a pause trigger new 429 responses.
How do I stay under 60 MB per 10 seconds?
Turn on gzip: the limit counts compressed data. Send If-None-Match with full datasets: a 304 has no body. Take heavy sections such as funding analytics as a coin slice, and do not download the same full dataset more often than freshness_sec. Responses under 64 KB do not count, and according to the docs, polling every dataset at your plan's freshness fits under the limit with room to spare.
What does a 429 Too Many Requests error mean?
The server rejected the request because the client went over a limit within a time window. The response may carry Retry-After, the number of seconds to wait; without it, double the pause after each failure. On Genesis API a 429 comes with the code rate_limited or download_limited.