Genesis Arbitrage

Genesis Arbitrage · · 15 min read

How to poll a crypto market API without hitting rate limits: ETag, gzip, filters and retries after 429

In short

  • An API usually has three limits: requests per second, a call quota per period and download volume. On Genesis API that is 1 to 15 requests per second, 500K to 10M calls per 30 days and 60 MB of compressed data per 10 seconds per key.
  • Set the polling interval from meta.freshness_sec: polling faster returns the same snapshot. If-None-Match turns a repeat into a free 304, gzip and filters cut traffic, and one full dataset instead of a request per coin saves calls.
  • Retry 429 and 503 after Retry-After seconds or, without that header, with exponential backoff plus random jitter. A client-side token bucket keeps your pace under the limit, and the X-RateLimit-* and X-Quota-* headers show the headroom left.

To poll an API without 429 errors, shape your requests around three limits at once: requests per second, the call quota for the period and the volume of data you download. Five techniques cover it: an interval no shorter than the data refresh, conditional requests with ETag, gzip compression, filters instead of oversized downloads, and retries after the pause the server asks for. The numbers and code below are for Genesis API, a REST API for crypto market data, but the techniques work with any HTTP API, exchange APIs included.

Three API limits: requests per second, quota and volume

A per-second rate limit catches bursts: a batch of requests sent at once. The quota catches your average pace over the whole period, and the volume limit catches heavy downloads even when there are few requests.

LimitWhat it countsGenesis APIOver the limit
Requests per secondEvery request, including 304 and errors1 on Trial, 2 on Start, 5 on Pro, 15 on Max429 rate_limited, Retry-After: 1
Quota per periodOnly 200 responses500K, 2M, 5M, 10M calls per 30 days429 rate_limited, Retry-After until the quota resets
Download volumeCompressed responses of 64 KB and up60 MB per 10 seconds per key, on every plan429 download_limited until the window ends

Genesis API never bills requests over a limit: a 429 uses no quota and costs nothing. All the rules are in the Genesis API limits guide and in the Plans & limits section of the docs.

Exchange APIs can be more involved. Binance counts request weight per IP address, and every endpoint has its own weight. A client that keeps sending requests after a 429 gets an automated IP ban with status 418, and repeat offenders get longer bans (Binance docs).

Polling interval: how many calls you use in 30 days

Polling faster than the data refreshes brings nothing but repeats. The Genesis Arbitrage core refreshes data every second, while the API output for your plan refreshes once every meta.freshness_sec seconds: 60 on Trial, 30 on Start, 10 on Pro and 5 on Max. That is your plan's data freshness and the floor for your polling interval. Genesis API has no WebSocket or webhooks, so you get data by polling over REST.

Illustrative example: a bot on the Start plan — $50 a month, 2M calls, 2 requests per second, 30 s freshness — polls four endpoints: /arbitrage/spot-spot, /arbitrage/spot-futures, /funding/spread and /market/open-interest.

IntervalCalls per 30 daysShare of Start quotaRepeats of one snapshot
60 s172,8008.64%none
30 s345,60017.28%none
10 s1,036,80051.84%2 of 3
5 s2,073,600103.68%5 of 6

Polling every 5 seconds runs the quota out on day 29: 4 × 86,400 / 5 = 69,120 calls a day, and 2,000,000 / 69,120 ≈ 28.9 days. Five of every six responses repeat one snapshot with the same meta.updated_at, and without If-None-Match each of them costs a call.

At one poll every 30 seconds the average pace is 4 / 30 ≈ 0.13 requests per second. But four requests sent at once land in the same second, and two of them get a 429. If you need data fresher than 30 seconds, the answer is a plan with a lower freshness_sec: Pro refreshes the snapshot every 10 seconds, Max every 5.

Conditional requests: ETag, If-None-Match and 304

An ETag is a version tag on a response. The client stores it and sends it back in the If-None-Match header of its next request to the same URL; if the version has not changed, the server replies 304 Not Modified. Under RFC 9110 a 304 ends after the headers and carries no body, so you use your own copy of the data.

On Genesis API a 304 uses no quota but still takes a slot in the requests-per-second limit. Only full datasets without filters get a tag. Responses to requests with coin, exchange, min_spread or other filters, as well as /orderbooks/book and /orderbooks/coin/{coin}, are built per request and come without an ETag.

ETag saves quota where data changes less often than you ask for it. The tag on spreads changes with every snapshot, so when you poll once per freshness_sec, almost every response is a 200.

Illustrative example on Pro: the daily sums of funding analytics, /funding/analytics/daily, are regenerated hourly; once a year of history has built up, the full section is about 12 MB gzip, and without coin it is served at most once per 300 seconds. Polling every 300 seconds makes 2,592,000 / 300 = 8,640 requests in 30 days. With If-None-Match the section downloads 30 × 24 = 720 times, and the other 8,640 − 720 = 7,920 responses come back as 304.

The result: 720 calls instead of 8,640, and about 8.6 GB of traffic instead of 104 GB. 304 responses do not count toward the full-download limit.

Filters and coin: what saves calls and what saves traffic

A filter shrinks the response but not the number of calls: the quota counts responses, and a request with coin=BTC costs as much as the full dataset. So the choice depends on which runs out first, calls or traffic.

Illustrative example on Start: a bot tracks funding for five coins. Five requests to /funding/rates with coin every 30 seconds come to 5 × 86,400 = 432,000 calls in 30 days, plus five requests per cycle against a 2-per-second limit. One full /funding/rates with the coins picked out in code costs 86,400 calls, five times fewer, and it gets an ETag.

TaskHow to requestWhat it saves
Several coins from one datasetFull dataset with If-None-Match, filter in codeCalls: one request instead of several
One coin from a heavy sectionThe coin parameterTraffic and the volume limit
Rows above a thresholdmin_spread, min_volumeTraffic, but every poll is a call
A slow section in fullFull dataset with If-None-MatchCalls and traffic

The heaviest sections belong to funding analytics: the full summary is about 600 KB gzip, the 7-day payouts about 1 MB and the daily sums up to 12 MB (Funding endpoints). A single-coin slice with coin is exempt from the full-download limit, which is once per 60 seconds for the summary and payouts and once per 300 for the daily sums.

Gzip compression

Compression cuts the number of bytes on the wire. The client asks for it with Accept-Encoding: gzip, and the server replies with a compressed body and a Content-Encoding: gzip header. Python's requests adds gzip to Accept-Encoding and decompresses the response on its own; curl does the same with --compressed.

On Genesis API compression is tied to a limit: download volume is counted in compressed data, 60 MB per 10 seconds per key, and responses under 64 KB are not counted. Full datasets and large responses come gzipped when the client asks for it (Genesis API responses and errors). To see how many body bytes actually crossed the network, use curl's size_download variable:

# body bytes on the wire with and without gzip; each request is a call
URL="https://api.genesisarbitrage.com/v1/funding/rates"
curl -s -o /dev/null --compressed -w "gzip: %{size_download}\n" \
  -H "X-API-Key: $GENESIS_API_KEY" "$URL"
curl -s -o /dev/null -w "uncompressed: %{size_download}\n" \
  -H "X-API-Key: $GENESIS_API_KEY" "$URL"

Retrying after 429: Retry-After and backoff with jitter

HTTP 429 Too Many Requests is the server's answer to a client that went over a limit. Under RFC 6585 the response may carry a Retry-After header, and under RFC 9110 its value is either a number of seconds or a date. Genesis API sends Retry-After with every 429 and 503, in seconds.

A retry without a pause nearly always gets the same error and takes another slot in the per-second window. If there is a Retry-After, wait as long as the server asks. If there is not, wait exponentially longer after each failure: up to 1, 2, 4, 8 seconds.

Jitter, a random addition to the pause, matters when there are many clients: without it, everyone who got an error in the same second retries in the same second too. The backoff function picks a random pause between zero and the current step and adds a fraction of a second to Retry-After. fetch retries 429, 500 and 503, and stops when the pause is longer than 10 minutes: that is what an exhausted quota looks like.

import os
import random
import time

import requests

BASE = "https://api.genesisarbitrage.com/v1"
api = requests.Session()  # reuses the TCP connection, gzip is on
api.headers["X-API-Key"] = os.environ["GENESIS_API_KEY"]
cache = {}  # path -> (ETag, data): full datasets only


def backoff(attempt, retry_after=None, cap=60.0):
    """Pause before a retry, in seconds."""
    if retry_after and retry_after.isdigit():
        return int(retry_after) + random.uniform(0, 1)
    return random.uniform(0, min(cap, 2 ** attempt))


def fetch(path, params=None, tries=5):
    """GET from the API: returns (data, response headers)."""
    for attempt in range(tries):
        tag = None if params else cache.get(path, (None,))[0]
        try:
            r = api.get(BASE + path, params=params, timeout=15,
                        headers={"If-None-Match": tag} if tag else {})
        except (requests.ConnectionError, requests.Timeout):
            time.sleep(backoff(attempt))
            continue
        if r.status_code == 304:  # same snapshot, no quota used
            return cache[path][1], r.headers
        if r.status_code in (429, 500, 503):
            wait = backoff(attempt, r.headers.get("Retry-After"))
            if wait > 600:  # this is what an exhausted quota looks like
                raise RuntimeError(f"quota resets in {wait:.0f} s")
            time.sleep(wait)
            continue
        r.raise_for_status()  # 400, 401, 403, 404, 410: do not retry
        data = r.json()["data"]
        if not params and "ETag" in r.headers:
            cache[path] = (r.headers["ETag"], data)
        return data, r.headers
    raise RuntimeError(f"{path}: no response after {tries} tries")
ResponsePauseRetry
429 rate_limited, per secondRetry-After: 1 plus a fraction of a secondYes
429 rate_limited, quotaUntil the quota resets, can be daysNo
429 download_limitedRetry-After, until the window endsYes, with narrower requests
503 snapshot_not_readyRetry-AfterYes
500, timeout, dropped connectionExponential with jitterYes
400, 401, 403, 404, 410—No, fix the request

You tell an exhausted quota from the per-second limit by Retry-After: one second versus hours or days until the reset, with Monthly call quota exceeded. in the response body. Error codes and bodies are in the Errors section of the docs.

Client-side rate limiter: token bucket

A retry deals with a 429; a client-side limiter prevents it. A token bucket works like this: tokens arrive at rate per second, each request takes one, and an empty bucket makes you wait. The capacity, burst, sets how many requests can go out back to back without a pause.

import time


class TokenBucket:
    """At most rate requests per second, up to burst back to back."""

    def __init__(self, rate, burst=1):
        self.rate, self.burst = rate, burst
        self.tokens = float(burst)
        self.last = time.monotonic()

    def take(self):
        while True:
            now = time.monotonic()
            self.tokens = min(self.burst, self.tokens
                              + (now - self.last) * self.rate)
            self.last = now
            if self.tokens >= 1:
                self.tokens -= 1
                return
            time.sleep((1 - self.tokens) / self.rate)

Take the limit from the rps_limit field of /account/usage (the Usage section) or from the X-RateLimit-Limit header. Set the rate a little below the limit, for example 0.9 * rps_limit: network latency is uneven, so requests sent half a second apart can reach the server closer together.

Monitoring: the X-RateLimit-* and X-Quota-* headers

A data response carries the state of your limits: X-RateLimit-Remaining and X-RateLimit-Reset give what is left in the per-second window and when it resets, while X-Quota-Remaining, X-Quota-Limit and X-Quota-Reset give the remaining quota, its size and the unix time it resets. More useful than any single remainder is the pace: what share of the quota is gone at this point in the period.

PERIOD = 30 * 86400  # Genesis API counts quota per 30 days


def watch(h):
    """Checks limit headroom from the response headers."""
    if h.get("X-RateLimit-Remaining") == "0":  # window used up
        reset = float(h.get("X-RateLimit-Reset", 0))
        time.sleep(max(0.0, reset - time.time()))
    try:
        left = int(h["X-Quota-Remaining"])
        limit = int(h["X-Quota-Limit"])
        reset = int(h["X-Quota-Reset"])
    except (KeyError, ValueError):
        return  # 429 responses carry no X-Quota-* headers
    used = 1 - left / limit
    passed = 1 - (reset - time.time()) / PERIOD
    if used > passed + 0.05:
        print(f"quota is running ahead: {used:.0%} used"
              f" with {passed:.0%} of the period gone")

The function waits for the window to reset when its limit is used up, and warns when quota use runs ahead of the calendar by more than 5 percentage points. Illustrative example: 70% of the quota is gone by day 15, so at the same pace the calls run out 15 / 0.7 ≈ 21.4 days into the period. /account/usage gives the same numbers without headers: calls_used, calls_limit and period_resets_at.

A schedule for several endpoints

With several endpoints, each gets its own interval, and the first requests are spread out so they do not go out as a batch. Pick the interval for the task, but never below freshness_sec: spreads need your plan's full rate, while for many strategies a minute is enough for funding and open interest.

import heapq

usage, _ = fetch("/account/usage")  # does not use a call
fresh = usage["freshness_sec"]
bucket = TokenBucket(rate=0.9 * usage["rps_limit"])

JOBS = {  # path -> polling interval, s
    "/arbitrage/spot-spot": fresh,
    "/funding/spread": max(fresh, 60),
    "/market/open-interest": max(fresh, 60),
}
t0 = time.monotonic()
queue = [(t0 + i * fresh / len(JOBS), path, every)
         for i, (path, every) in enumerate(JOBS.items())]
heapq.heapify(queue)

while True:
    due, path, every = heapq.heappop(queue)
    time.sleep(max(0.0, due - time.monotonic()))
    bucket.take()
    data, hdrs = fetch(path)
    watch(hdrs)
    print(path, len(data))
    heapq.heappush(queue, (due + every, path, every))

On Start that is 86,400 + 43,200 + 43,200 = 172,800 calls in 30 days, 8.64% of the quota: spreads every 30 seconds, funding and open interest every 60. The first requests go out 30 / 3 = 10 seconds apart, after which each endpoint keeps its own rhythm, and bucket.take() guards against collisions. A bot built on this polling, with Telegram alerts, is in Crypto arbitrage bot in Python.

What to do on 503 and server errors

Under RFC 9110, 503 Service Unavailable means a temporary overload or maintenance, and the server may suggest a pause in Retry-After. On Genesis API this is 503 snapshot_not_ready: the data is being rebuilt or its source is restarting. Retry after Retry-After seconds; the pause is usually a few seconds.

While there is no fresh response, your bot still holds the data from the last poll, and that data is aging. Before you send a signal, compare meta.updated_at with the current time: a pair from a minute-old snapshot may already be gone on the exchange. Other risks are covered in the Automation risks lesson.

Retry 500 internal_error, timeouts and dropped connections with exponential backoff. If the error persists, contact support with the X-Request-Id value, the request time in UTC, the endpoint and the status code. Never send your key.

Questions and answers

How many requests per second can I send to Genesis API?

As many as your plan allows: 1 on Trial, 2 on Start, 5 on Pro and 15 on Max. The limit is per key and includes 304 responses and errors. Your plan's value is in the X-RateLimit-Limit header and in the rps_limit field of /account/usage.

Do errors and 304 responses count against my quota?

No. Only a 200 response uses quota; 304, 429, 503 and other errors are free and cost nothing. They do take slots in the requests-per-second limit, though, so retries without a pause trigger new 429 responses.

How do I stay under 60 MB per 10 seconds?

Turn on gzip: the limit counts compressed data. Send If-None-Match with full datasets: a 304 has no body. Take heavy sections such as funding analytics as a coin slice, and do not download the same full dataset more often than freshness_sec. Responses under 64 KB do not count, and according to the docs, polling every dataset at your plan's freshness fits under the limit with room to spare.

What does a 429 Too Many Requests error mean?

The server rejected the request because the client went over a limit within a time window. The response may carry Retry-After, the number of seconds to wait; without it, double the pause after each failure. On Genesis API a 429 comes with the code rate_limited or download_limited.