Troubleshooting 403 Blocks and 429 Rate Limits

Overview


If your application starts receiving 403 Forbidden or 429 Too Many Requests responses, two different systems are involved: the Repliers API Firewall, which blocks requests from addresses it considers abusive, and our rate limiter, which governs how quickly requests may be made. They are related — how the rate limiter treats your traffic depends on whether the firewall can see it — but they fail in different ways and are fixed by different changes. This article explains how each behaves, how to tell them apart, and what to send us when you need our team to investigate.


Before you begin, it helps to know two things about your account:


  • Throughput is not a fixed number of requests per second — it depends on whether your solution implements the Repliers firewall. Without it, your requests are subject to our generic traffic-pattern analysis and are counted against the entire API key. With it, traffic is assessed per end user. This is documented in API Usage Limits and Overages.
  • Requests blocked by our firewall, and requests from allowed bots such as well-known search engine crawlers, do not count toward your monthly quota. Traffic that never reaches our application servers is not billed to you, and neither is crawler traffic we recognise as allowed.


Understanding 403 Responses From the API Firewall


A 403 returned by our edge means the Repliers API Firewall (WAF) rejected the request before it reached our application. This only happens on requests that carry the X-Repliers-Forwarded-For header, because that header is what tells us which end-user IP address to evaluate.


What triggers a block


Our firewall evaluates forwarded IP addresses against several signals:


  • Country of origin. Traffic from countries outside your service area can be filtered.
  • A maintained blocklist of roughly 100,000 known abusive IP addresses.
  • Real-time reputation data from AbuseIPDB.
  • Live traffic-pattern analysis, which identifies scraping and automated behaviour as it happens.


In some subscriber accounts, the firewall filters out as much as 75% of incoming traffic — traffic that would otherwise have consumed your throughput and your monthly quota.


When a legitimate visitor is blocked


IP addresses rotate. An address that was flagged for abuse months ago may since have been reassigned to an ordinary residential or mobile customer, and that customer may then see broken search results on your site.


We do not yet have an automated appeal process. To have an address reviewed, contact support with:


  • The IP address, if you retain it, or
  • The exact UTC timestamp, the endpoint path, and the API key used, so we can locate the request in our own WAF logs.


Note: Because reputation data is refreshed continuously, an address we remove from the blocklist may be re-added if new abuse reports appear for it on AbuseIPDB. If an address has an active high-confidence abuse score, removing it is a temporary measure rather than a permanent fix.


Before reporting a false positive


Check the request pattern for the affected visitor first. Traffic that looks abusive to our firewall sometimes turns out to be:


  • A single page view issuing many API calls. If one property-detail page fires a dozen requests, one visitor can look like a scraper.
  • Intermittent header forwarding. If some of your routes send X-Repliers-Forwarded-For and others do not, the same visitor will succeed on some pages and receive a 403 on others. This is the most common explanation for a visitor who "keeps coming back and keeps getting blocked."


Understanding 429 Responses


A 429 means the rate limit was exceeded for the traffic that request belonged to. What "that traffic" means is the important part, and it is determined by whether the request carried a forwarded end-user IP address.


The firewall determines how throughput is counted


This is the single most important point to design around.


Without the firewall, your requests are throttled more aggressively than protected requests, because we cannot tell where the traffic is coming from. Throttling is applied at the level of your entire solution. Our systems analyse the request patterns arriving on your API key, and when that combined traffic matches abusive or automated behaviour, the resulting throttling applies to the key as a whole. Every workload draws from the same allowance: your interactive map traffic, your property-detail pages, your ad landing pages, your nightly batch jobs, and any crawler traffic that reaches your application. A single misbehaving consumer — one scraper, one runaway job, one badly configured sitemap crawl — can therefore throttle every legitimate user of your application at once.


With the firewall implemented, we apply advanced traffic filtration and enforce rate limiting per end user rather than across your whole account. Scraping activity and known abusive addresses are identified and addressed at the source, so:


  • Only the visitor generating excessive requests is throttled; everyone else continues to be served normally.
  • Traffic our firewall rejects never reaches your allowance at all.
  • Legitimate growth in visitor numbers no longer pushes your account toward a solution-wide throttle.


Implementing X-Repliers-Forwarded-For consistently is therefore the most effective single change you can make to sustain higher throughput. It is not only a security measure — it is what makes per-user enforcement possible in the first place.


Two limits, measured differently


It is worth separating the two allowances, because they fail in different ways and are fixed by different changes.



Rate limit

Monthly quota

Scope

Per end user when the firewall is implemented; across the entire API key when it is not

Per account, per MLS board

Measured

Continuously, from live request patterns

Per billing period

Symptom when exceeded

429 Too Many Requests

No interruption — overage is billed linearly

Usual fix

Firewall coverage, pacing, fewer calls per page view, workload isolation

Firewall coverage, crawler control, fewer calls per page view


A 429 is always the rate limit, never the monthly quota. Exceeding your monthly allowance does not stop your key from working.


Requests we cannot attribute to a user


Some traffic has no end user behind it, and it is always assessed at the account level:


  • Background and batch jobs. There is no visitor to attribute them to.
  • Server-rendered pages where the header was not applied. If your application calls our API from the server without forwarding the visitor's address, we see only your hosting platform's egress address. On shared platforms this means all of your server-side traffic appears to originate from a small pool of addresses, and it is assessed as a single stream rather than as many individual visitors. Forward the header on server-rendered routes as well as client-side ones.
  • Internal tooling, health checks, and monitoring.


This traffic is exactly what benefits from being isolated on its own API key, so that it cannot throttle your live application.


Working out your real capacity


Whatever allowance applies to your account, the arithmetic behind a page view matters more than most teams expect:


  • A map UI that issues 2 calls per interaction — for example, one for listings and one for counts or statistics — halves your effective concurrent user count.
  • A property-detail page that fetches the listing, its history, and nearby comparables issues 3 calls for a single page view.
  • A visitor panning and zooming a map generates interactions every few seconds, not every few minutes.


Count the actual calls behind your busiest page and multiply by your peak concurrency. If the result looks heavy, no retry strategy will save you — the work has to be removed by consolidating calls and cutting the requests each page makes.


To measure rather than estimate, use the developer portal at login.repliers.com:


  1. The Usage tab shows consumption over time and will reveal daily peaks.
  2. Live Tail shows individual requests as they arrive, which is the fastest way to discover that one page is quietly making five calls instead of one.
  3. Security Insights shows the protected/unprotected split, and therefore how much of your traffic is eligible for per-user treatment rather than being pooled against the key.


Bursts count, not just averages


The limiter measures requests over very short intervals. A modest average will still produce 429s if those requests arrive as bursts.


A batch job that issues several calls back to back and then pauses averages out to a comfortable figure, but the burst itself may momentarily exceed the ceiling — and if any live traffic arrives during that burst, it is throttled along with the job.


Pace evenly instead. Space requests at a fixed interval rather than alternating bursts with sleeps, and issue them single-threaded so that the interval is actually observed.


Enforce a limit on your side


Rather than discovering the ceiling through 429 responses, keep your own requests below it:


  • Use a token-bucket or leaky-bucket limiter in front of your Repliers client, sized with headroom for the calls you did not anticipate.
  • On serverless platforms, the limiter must be shared. Each function instance has its own memory, so an in-process limiter is enforced per instance rather than per account, and ten concurrent instances will happily send ten times your intended rate. Back the limiter with Redis or an equivalent shared store.
  • Debounce map interactions. Waiting 300–500 ms after the user stops panning or zooming, and cancelling superseded requests, typically removes the majority of calls generated by an interactive map.


Isolate background jobs on a separate API key


Create a dedicated API key for background work — nightly statistics runs, data synchronisation, report generation — and do not send X-Repliers-Forwarded-For on those requests, since there is no end user to attribute them to.


This does three things:


  • It stops a batch job from starving your live website of throughput.
  • It makes both traffic streams far easier to diagnose, since a 429 on one key no longer tells you nothing about which workload caused it.
  • It keeps your Security Insights protected/unprotected figures meaningful, since unforwarded background traffic is no longer counted against your live traffic's coverage.


Schedule batch work outside your peak hours, and pace it as described above rather than running it as fast as the limiter allows.


Back off correctly


Standard exponential backoff schedules are usually too aggressive for our limiter. A retry ladder of 2s, 4s, 8s, and 16s will typically continue to fail, because the limiter is still counting your earlier attempts.


Do not retry requests made on behalf of an end user. A 403 or 429 on a user-facing request is final for that visitor: show a graceful message rather than retrying. A throttled visitor is throttled because of their own request pattern, and retrying on their behalf only adds to it. Never resend the request without the X-Repliers-Forwarded-For header to get around a block or throttle; doing so circumvents the firewall.


Pass the 429 through to the client. Requests usually reach us through your application's backend, so a 429 from Repliers only reaches the client if your backend passes it on. Return a 429 from your own route, with the Retry-After header if one was sent, rather than a 200 with an empty page, a 500, or a different 4xx. Ethical bots such as Googlebot treat a 429 as a signal to slow down; if they never see it, they keep crawling at the same rate. Other 4xx codes such as 403 or 404 don't slow them down and can get your pages dropped from search results.


For people, pair the 429 with a challenge. Instead of a dead end, serve the 429 page with a CAPTCHA or your bot-management service's challenge (see Protect your own pages from bots), and use the Retry-After value to tell the visitor when they can continue. Let them resume browsing only once they have passed the challenge and the Retry-After period has elapsed. Passing the challenge confirms a person is there, but it does not lift our throttle, so resuming early will just produce another 429. The visitor chooses when to continue; your application doesn't retry on their behalf. Don't challenge verified search engine and advertising crawlers: send them the plain 429 with Retry-After.


The rules below are for background and batch jobs on their own API key. For those, follow these rules instead of a standard backoff schedule:


  1. Honour the Retry-After header returned with the 429 response rather than using a fixed schedule.
  2. Wait at least 60 seconds after the most recent 429 before resuming. The clock restarts on every additional 429 you receive, so retrying early extends the throttling rather than resolving it.
  3. Add jitter. Without it, every queued request retries at the same instant and immediately re-triggers the limit.
  4. Drop concurrency to one while recovering, then ramp back up gradually once requests are succeeding again.


Also check the x-ratelimit-remaining header on successful responses — it is a useful signal for pacing a long-running job before it starts failing, and it tells you how close you are running to the ceiling under normal load.


Do not cache Repliers responses


We ask that you do not cache Repliers API responses — not in Redis or another key-value store, not in your framework's data cache, not as pre-rendered pages (for example, Next.js ISR), and not at a CDN in front of pages that display Repliers data.


A request served from your cache never reaches us, so the Repliers firewall cannot evaluate the visitor behind it. That has two consequences:


  • Blocked users keep getting data. A scraper or abusive visitor that our firewall has identified and blocked can continue to retrieve listing data from your cache, because the request that would have been rejected is never made.
  • Abuse becomes invisible. The live traffic-pattern analysis that detects scraping depends on seeing each visitor's requests. Cached responses hide those patterns, so abusive behaviour is detected later, or not at all.


In other words, caching works against the per-user protection described above. Pooling many visitors behind a single upstream call — for example, coalescing identical in-flight requests and fanning out the result — has the same effect and should also be avoided.


To reduce pressure on your allowance without caching:


  • Forward X-Repliers-Forwarded-For on every client-facing request. Traffic our firewall rejects never reaches your allowance or your monthly quota, which is often the largest saving available.
  • Cut the calls behind each page view. Consolidate requests where you can, and avoid fetching data a page does not display.
  • Debounce map interactions and cancel superseded requests, as described above.
  • Pace background work and isolate it on its own API key.
  • Keep unwanted crawlers away from pages that call our API, and put your own bot protection in front of them (see below).


Control crawler and AI bot traffic


Automated crawlers — search engines, advertising verifiers, and increasingly AI training and answer bots — can consume a large share of your throughput at exactly the wrong moment, because they do not pause during your peak hours.


  • Traffic filtered by the Repliers firewall never reaches your allowance at all. This only applies to requests forwarded with X-Repliers-Forwarded-For, which is a further reason to apply the header consistently.
  • Unwanted bots that crawl your own pages spend your budget, because your server makes the resulting Repliers calls under your key. Without forwarding, those calls are indistinguishable from your live user traffic and are pooled with it. Filtering has to happen at your edge, before the page is rendered.
  • Use robots.txt to keep crawlers away from API routes and low-value paths, and disallow AI crawlers you do not wish to serve. Well-behaved bots honour it; the rest need blocking or challenging at your edge (see Protect your own pages from bots).
  • Forward the crawler's address too. When a page is fetched by a bot, forward its IP address with X-Repliers-Forwarded-For just as you would for a human visitor, so the bot is assessed as an individual client rather than pooled with the rest of your traffic.
  • Allowed bots, such as well-known search engine and advertising crawlers, get higher rate limits than a regular visitor, and their requests do not count toward your monthly quota. We can only recognise an allowed bot when its IP address is forwarded with X-Repliers-Forwarded-For, which is a further reason to forward the header for crawler traffic.
  • Pay particular attention to ad landing pages. Advertising crawlers (for example, Google AdsBot) verify these pages regularly. When AdsBot's IP address is forwarded, we recognise it as an allowed bot and give it the higher limit. When it is not forwarded, its calls are pooled with the rest of your unprotected traffic and are far more likely to fail with 429, which can hurt your ad quality scores. Forward the header on every landing-page route, and keep the number of Repliers calls behind each landing page to a minimum.


Protect your own pages from bots


The Repliers firewall protects our API, but it only sees a request once your application has decided to call us. A bot that loads your pages directly causes those calls to be made on its behalf. Because you should not cache Repliers responses, every page a bot loads turns into requests against your allowance. We therefore recommend putting your own bot protection in front of your application:


  • Use a bot-management or challenge service at your edge. Most CDNs, hosting platforms, and web application firewalls offer bot detection, CAPTCHA-style or invisible challenges, and IP or ASN reputation filtering. Apply it to the pages and routes that trigger Repliers calls, such as search, map, and listing-detail pages.
  • Protect your own API routes as well as your pages. If your front end calls an internal endpoint that proxies to Repliers, scrapers will often call that endpoint directly. Challenge or authenticate those routes, and validate that the challenge was actually passed before calling us.
  • Rate-limit at your edge. Per-IP or per-session limits on your own routes stop a single client from generating bursts of upstream calls.
  • Allow the crawlers you want. Make sure verified search engine and advertising crawlers are not challenged, or your search visibility and ad quality scores can suffer.


Your own bot protection works alongside the Repliers firewall rather than replacing it. Continue to forward X-Repliers-Forwarded-For on every client-facing request, so that anything your edge lets through can still be assessed and throttled per user.


Verifying That Your Traffic Is Protected


You can confirm how much of your traffic reaches the firewall yourself, without contacting support:


  1. Log in at login.repliers.com.
  2. Open Security Insights from the dashboard.
  3. Review the split between Protected and Unprotected traffic.


Requests are counted as Protected when they carry the X-Repliers-Forwarded-For header, and Unprotected when they do not. Only protected traffic is eligible for per-user rate limiting; unprotected traffic is pooled and assessed against the key as a whole.


If you have deployed forwarding and still see substantial unprotected traffic, the header is likely missing on a subset of your routes — server-rendered pages, API routes added later, or endpoints other than /listings are common gaps.


Requesting a Higher Rate Limit


Throughput can be increased. Before we do so, we ask that all client-facing traffic on the account is routed through the Repliers firewall, verified through Security Insights.


The reason is practical rather than procedural. On an unprotected key we cannot distinguish one visitor from another, so raising the ceiling raises it for bots as much as for your users, and a single abusive consumer can still throttle everyone. Once forwarding is in place, unwanted traffic is filtered before it consumes your allowance, throttling can be targeted at the individual responsible for it, and the additional headroom reaches the people you intended it for.


Once all user traffic is protected, you can request further increases as demand for your application grows.


Background jobs on a separate, unforwarded key do not affect this assessment.


When you request an increase, tell us your peak concurrent user count and your expected growth, so we can size the limit appropriately.


Quick Reference


Symptom

Most likely cause

First action

403 on requests with a forwarded IP

Firewall blocked the end-user IP

Check whether forwarding is applied consistently, then send us the timestamp and path

403 on some pages but not others

X-Repliers-Forwarded-For missing on some routes

Review Security Insights for unprotected traffic

429 affecting all users at once

Unprotected traffic throttled at the solution level

Implement X-Repliers-Forwarded-For across all routes

429 during a batch window

Batch job sharing a key with live traffic

Move background jobs to a separate key

429 on light interactive traffic

Another workload consuming the shared, unprotected budget

Implement forwarding; check for crawler traffic reaching your app

429 persisting through a background job's retries

Backoff too short

Honour Retry-After; wait at least 60 seconds after the last 429

429 despite a modest average

Requests arriving in bursts

Pace evenly instead of alternating bursts and sleeps

429 that appears only under load on serverless

Per-instance limiter, enforced many times over

Move the limiter to a shared store such as Redis

429 on server-rendered pages only

Header not forwarded server-side; hosting egress seen as one client

Apply forwarding on server-rendered routes

Crawlers keep hitting 429 without slowing down

Your backend returns the 429 as a 200, 500, or other error

Pass the 429 and Retry-After through to the client

High request volume from bots loading your pages

No bot protection in front of your application

Add bot management or challenges at your edge




What questions does this article answer?


  • Why is the Repliers API returning 403 for my legitimate end users?
  • What triggers an IP block, and how do I request a review?
  • How does implementing the Repliers firewall change the way rate limiting is applied?
  • Does sending X-Repliers-Forwarded-For change how my throughput is counted?
  • Why does one abusive visitor or bot throttle my entire application?
  • What is the difference between per-user and account-level rate limiting?
  • Why am I receiving 429 responses at low request volumes?
  • What is the difference between the rate limit and the monthly quota?
  • Why do I get 429 responses when my average request rate looks modest?
  • How do I enforce a rate limit on my own side, and what changes on serverless platforms?
  • Why do my server-rendered pages get throttled when my client-side calls do not?
  • How should I implement backoff when I hit the rate limit?
  • Should my background jobs use a different API key?
  • Should I cache Repliers API responses?
  • How can I reduce my request volume without caching?
  • How do I stop crawlers and AI bots from consuming my throughput?
  • Should I have my own bot protection in front of my application?
  • Do requests from search engine crawlers and other allowed bots count toward my monthly quota?
  • How do I check whether my traffic is going through the Repliers firewall?
  • What do I need to do before requesting a higher rate limit?

Updated on: 04/11/2026

Was this article helpful?

Share your feedback

Cancel

Thank you!