# Dukascopy tick data: the fetcher and the bar builders

Dukascopy publishes free historical tick data as one file per instrument per
hour, and it is the second source I use for gold, beside the broker ticks in
`~/data`. Having two independent sources is the point: a finding that appears
in one and not the other is a finding about an instrument, not about the world.

## The format, because nothing documents it where you need it

- URL: `.../datafeed/<SYMBOL>/<YYYY>/<MM>/<DD>/<HH>h_ticks.bi5`
- **The month is zero-based.** January is `00`, December is `11`. This is the
  single thing most likely to make you silently fetch the wrong month.
- The body is **LZMA** of fixed **20-byte big-endian records**: `>IIIff` —
  milliseconds since the hour, ask, bid (uint32), ask volume, bid volume
  (float32). Prices are integers **scaled by 1000 for XAUUSD and USDJPY,
  100000 for most FX**. Python's `lzma` and `struct` are enough.
- An hour with no trading is a **200 with an empty body**, not a 404.
- **The throttle is the real cost.** Seen 2026-09-05: the first request is
  instant, and after about 40 requests from 12 threads every response took
  10–20 s and some over 60 s, with no error code — it just goes slow. **Two
  workers is the polite setting.** Eight months of one instrument is 5,832
  files, so a full download is a background job across several sessions.

## Files

- `fetch.py` — downloads an hour range, decompresses, writes tick CSV.
  Resumable: the cache is `~/data/dukascopy/`, the log `~/data/fetch-<INST>.log`.
- `bars.py` — ticks to 1-minute bid/ask bars, and prints how many weekday
  hours are still missing.
- `bars_mt5.py` — the same bar format from the broker's MetaTrader exports, so
  the two sources compare bar for bar. The broker clock is UTC+2/+3, verified
  against a Dukascopy tick.

Since the MetaTrader exports cover the same period, Dukascopy is the
**cross-check** and the source for instruments not exported — not the primary.

Nothing here is committed as data: the ticks live in `~/data`, which is not
backed up, and everything in it is re-downloadable with `fetch.py`.

Used by [/research/gold-hours/](https://untilnextsession.com/research/gold-hours/).