> ## Documentation Index > Fetch the complete documentation index at: https://qwady.wiki/llms.txt > Use this file to discover all available pages before exploring further. # Data Freshness > How Borough keeps data fresh and what staleness means for your tier ## Overview Borough caches data in a D1 database and refreshes it on a schedule. Every API response includes baseline freshness metadata so you can see how old the data is and how it was served. Freshness-sensitive endpoints may also include refresh-decision metadata when tier-based refresh logic applies. ## Response metadata Freshness-aware JSON responses include these baseline fields in `meta`: | Field | Type | Description | | -------------------------------------------------------- | ------ | ----------------------------------------------------- | | `dataAge` | string | When the data was last refreshed (ISO 8601 timestamp) | | `source` | string | How the data was served: `cached`, `stale`, or `live` | | Some freshness-sensitive endpoints additionally include: | | | | Field | Type | Description | | -------------------- | ------- | --------------------------------------------------- | | `freshnessThreshold` | integer | Max allowed age in minutes before refresh triggers | | `refreshTriggered` | boolean | Whether this request triggered a background refresh | `GET /v1/property/{id}/history`, `/fees`, and `/open-houses` additionally include: | Field | Type | Description | | --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `detailScraped` | boolean | Whether a detail scrape has ever run for this listing. `false` means the `data: []` you got back means "never checked," not "checked, none found." See [Detail-only fields and coverage](#detail-only-fields-and-coverage) below. When `false`, `dataAge` is also `null` for the same reason. | ### Response headers Two freshness headers are also included on metered JSON `/v1` responses: ``` X-Data-Age: 2026-02-16T14:30:00Z X-Data-Source: cached ``` ## Source values | Value | Meaning | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `cached` | Data is within the freshness threshold for your tier | | `stale` | Data exceeds the freshness threshold for your tier. On property and building detail this also queues a background refresh; on search it is a staleness signal only (see below) | | `live` | Fetched live from the source just now (Business/Internal tiers on property and building detail) | When `source` is `stale`, the API still returns the data immediately. On property and building detail a refresh is queued asynchronously, so a subsequent request gets fresher data. ## Freshness thresholds by tier Data is considered stale past these age thresholds. On property and building detail, paid tiers additionally queue a background refresh (search does not; see below): ### Search results | Tier | Threshold | | -------- | --------- | | Starter | 8 hours | | Pro | 8 hours | | Business | 8 hours | ### Listing details | Tier | Threshold | | -------- | ---------- | | Starter | 30 minutes | | Pro | 15 minutes | | Business | 10 minutes | ### Building details | Tier | Threshold | | -------- | --------- | | Starter | 6 hours | | Pro | 3 hours | | Business | 2 hours | ## Free tier behavior Free-tier requests always serve cached data and never trigger background refreshes. On freshness-sensitive endpoints, `freshnessThreshold` and `refreshTriggered` are `null` for free-tier responses. Cached data is refreshed on a fixed schedule regardless of API traffic: | Data type | Schedule | | --------------------- | ------------------------------------------------------------------------------------------------------ | | Rental search index | Every 6 hours | | Sale search index | Every 8 hours | | Listing detail sweeps | Every 12 hours | | Building details | Daily (03:00 UTC) | | Area boundaries | Rarely change; updated on demand | | Broadband enrichment | Checked monthly; the underlying FCC dataset is published twice a year, so most checks find nothing new | | Market snapshots | Daily at 05:00 UTC | ## How async refresh works When a paid-tier request for **property or building detail** hits stale data: 1. The API returns the stale data immediately with `"source": "stale"` 2. A background job is triggered to fetch fresh data from the source 3. The fresh data is written to the database 4. Your next request returns the updated data with `"source": "cached"` This means you never wait for a refresh: you always get a response immediately. The tradeoff is that one response may contain slightly older data. ### Search is different Search results are kept fresh by the scheduled crawl, not by per-request refreshes. A single search spans many listings and areas, so there is no one upstream fetch to trigger. A `"source": "stale"` on search therefore means the data is past the 8-hour threshold (typically because a scheduled crawl was missed), not that a refresh has been queued. It resolves on the next crawl. ## Live-first data (Business / Internal) Business and Internal tier requests for **property detail** and **building detail** endpoints get live data automatically: * If cached data is older than **5 minutes** (listings) or **30 minutes** (buildings), the API fetches fresh data from the source synchronously * The live data is returned immediately with `"source": "live"` * The fresh data is also written back to the cache, keeping it warm for all tiers * If the live fetch fails or times out (8-10 seconds), the API falls back to cached data with the standard async refresh behavior This applies to: * `GET /v1/property/{id}` * `GET /v1/property/by-url` * `GET /v1/building/{id}` Search, market, and area endpoints always serve from cache regardless of tier. ## Listing availability Search endpoints (`/v1/search/rentals`, `/v1/search/sales`) and `GET /v1/building/{id}/listings?status=ACTIVE` return only listings that are currently on the market. When a listing comes off the market, Borough sets its `status` to `OFF_MARKET` and removes it from active search results. Webhook subscribers receive a [`listing.expired`](/borough/guides/webhooks) event; that event is delivered to webhooks only and is not part of `GET /v1/property/{id}/history`, which returns price events (`LISTED`, `PRICE_DROP`, `PRICE_INCREASE`) exclusively. Without a webhook subscription, read off-market status from `data.status` on `GET /v1/property/{id}`. Availability is reconciled against the source on each search refresh (rentals every 6 hours, sales every 8 hours), so a listing that leaves the market is reflected within roughly that window. A property detail lookup (`GET /v1/property/{id}`) still resolves for an off-market listing, so check `data.status` to confirm availability rather than assuming a successful response means the unit is active. ## Listed date accuracy Every listing carries `listedAt` (on-market date) and a `daysOnMarket` value derived from it. Until a listing has been detail-scraped, `listedAt` is the date Borough **first observed** the listing in the feed (an estimate accurate to within one search cycle) rather than the authoritative on-market date. The `listedAtIsApproximate` boolean tells you which you have: * `listedAtIsApproximate: true`: `listedAt` is the feed-first-seen estimate (detail not yet scraped). * `listedAtIsApproximate: false`: `listedAt` is the authoritative date from the listing detail page. `daysOnMarket` inherits the same accuracy. Check the flag before relying on `listedAt` or `daysOnMarket` for date filtering (`listedAfter` / `listedBefore`) or time-on-market analysis. A listing's flag flips to `false` once its detail page is scraped, but scraping is a continuous, prioritized queue, not a fixed-latency job. The listings scraper *runs* every 12 hours; that's how often the cron fires, not a bound on how long any individual listing waits to be detail-scraped. See [Detail-only fields and coverage](#detail-only-fields-and-coverage) below for why many listings wait far longer than 12 hours, some indefinitely. ## Detail-only fields and coverage Some listing fields are only ever populated by a **detail scrape** (a fetch of the individual listing's page), as distinct from the search-feed sweep that keeps most of Borough's data warm. Until a listing has been detail-scraped, these fields read as `null`, not `false`, `0`, or `[]`: * Listing booleans: `noFee`, `furnished`, `has3dTour`, `hasVideo` * Other detail-only listing fields: `sqft`, `petsAllowed`, `amenities` (listing-level), `maintenance`, `commonCharges`, `propertyTax`, `floorPlanKey`, `virtualTourUrl` * The sub-resources `GET /v1/property/{id}/history`, `/fees`, and `/open-houses`: an empty `data: []` from these on an undetailed listing means "never checked," not "checked, none found." Confirm via `meta.detailScraped` (see the table above) `detailScraped` tells you whether a detail scrape has run for this listing. It's on every listing object: search results (`/v1/search/rentals`, `/v1/search/sales`), `GET /v1/building/{id}/listings`, `GET /v1/property/{id}/comparables`, and `GET /v1/property/{id}`, plus `meta.detailScraped` on the three sub-resources above. But it's a weaker signal than it looks. A detail scrape that ran but failed to parse one particular field (for example, an upstream key rename) still sets `detailScraped: true` while that one field stays `null`. Read `detailScraped: true` as "Borough attempted a detail scrape," not "every detail-only field on this listing was observed." **Coverage today:** currently around 55% of ACTIVE listings have never been detail-scraped. Treat that figure as approximate and time-varying — it moves as the backlog drains and as new listings arrive — but the shape holds: this is the majority case, not an edge case, so expect most of the fields above to be `null` on most listings, most of the time. Detail scraping is an always-running pipeline, not a one-time backfill, but new listings enter the feed at roughly the rate the backlog drains, so the gap does not reliably close with time. If your workflow needs confirmed values, gate on the field being non-`null` (or on `detailScraped`) rather than assuming that waiting resolves it. This coverage gap is also why `noFee=false`, `has3dTour=false`, and `hasOpenHouse=false` are rejected outright (`400 INVALID_PARAMS`) rather than silently matched. See the [filtering guide](/borough/guides/filtering). ## Best practices * Check `meta.source` to know if you're reading stale data * For time-sensitive detail lookups, use Pro or Business tier for tighter listing/building freshness thresholds * Business tier gets live data on property and building detail: no need for SSE streams for single lookups * If you need the freshest available Borough path on Pro, use the [listing stream](/borough/guides/streaming) endpoint for live SSE delivery. Like any live fetch, it still depends on upstream availability and response time.