Download MUF Data

Downloads and prepares training data for a neural network that predicts the Maximum Usable Frequency (MUF) for ionospheric radio propagation. The pipeline fetches ionosonde measurements from INPE's EMBRACE network and co-locates them with geomagnetic and solar space indices.

Background

MUF is the highest radio frequency that can be reflected back to Earth by the ionosphere for a given path. Predicting it requires both ionospheric measurements (from ionosondes) and geophysical drivers (geomagnetic activity, solar flux). This project collects both and stores them in SQLite databases ready for training.

Two stations are supported:

Station	Code	Location
Boa Vista	BVJ03	Boa Vista, Roraima, Brazil
Cachoeira Paulista	CAJ2M	Cachoeira Paulista, São Paulo, Brazil

Data Sources

• MUF data — .SAO files from the EMBRACE/INPE ionosonde network. Each file contains a single measurement: timestamp, foF2, and MUF.
• Space indices — fetched via SpaceIndices.jl:
  • Dst — Disturbance Storm Time index (geomagnetic activity)
  • Kp — Planetary K-index (geomagnetic activity, 3-hour resolution)
  • F10obs — Observed F10.7 solar flux
  • F10avg — 81-day average F10.7 solar flux

Requirements

• Julia ≥ 1.10
• sqlite3 CLI (for the merge step)

Julia dependencies are declared in Project.toml and resolved automatically when running with --project=./.

Usage

Run targets in order for each station. Boa Vista is used as an example; replace boa-vista with cachoeira-paulista for the other station.

1. Download MUF data

make download-muf-boa-vista

Fetches .SAO files from the EMBRACE website and stores parsed timestamps and MUF values in bvj03-muf.db. Downloads are incremental — already-fetched files are skipped. Progress is written to download_muf.log.

2. Download space indices

make download-space-indices-boa-vista

Reads all timestamps from bvj03-muf.db, builds a 5-minute time grid spanning the same range, and fetches Dst, Kp, F10obs, and F10avg for every point. Results go into space_indices-bvj03.db. Already-present timestamps are skipped.

3. Merge into training database

make merge-training-data-boa-vista

Combines bvj03-muf.db and space_indices-bvj03.db into a single bvj03.db by copying the MUF database and importing the space_indices table into it.

All targets

make help

Target	Description
download-muf-boa-vista	Download MUF data for BVJ03
download-muf-cachoeira-paulista	Download MUF data for CAJ2M
download-space-indices-boa-vista	Download space indices for BVJ03 timestamps
download-space-indices-cachoeira-paulista	Download space indices for CAJ2M timestamps
merge-training-data-boa-vista	Merge MUF + space indices into bvj03.db
merge-training-data-cachoeira-paulista	Merge MUF + space indices into caj2m.db

Output Databases

bvj03-muf.db / caj2m-muf.db

Table: muf

Column	Type	Description
filename	TEXT (PK)	Source .SAO filename
station	TEXT	Station code (e.g. BVJ03)
timestamp	TEXT	Measurement datetime (ISO 8601)
MUF	REAL	Maximum Usable Frequency in MHz; NULL if measurement failed or unreliable

MUF is set to NULL when: the raw value is ≥ 9999 MHz (sensor failure flag), or foF2 < 3 MHz during nighttime hours (unreliable measurement).

space_indices-bvj03.db / space_indices-caj2m.db

Table: space_indices

Column	Type	Description
timestamp	TEXT (PK)	Datetime (ISO 8601), on a 5-minute grid
Dst	REAL	Dst index (nT)
Kp	REAL	Kp index (3-hour interval value)
F10obs	REAL	Observed F10.7 solar flux (sfu)
F10avg	REAL	81-day average F10.7 solar flux (sfu)

bvj03.db / caj2m.db (training databases)

Final merged databases containing both the muf and space_indices tables, ready for training.

Implementation Notes

• Downloads are parallelized with nworkers=4 concurrent threads per day-of-year directory. All SQLite writes are serialized through a ReentrantLock and batched in per-day transactions to minimize disk fsyncs.
• The MUF downloader resumes from the latest stored timestamp, skipping year/DOY directories that precede it while still checking for gaps within the latest DOY.
• Space indices are computed over the union of actual MUF timestamps and a 5-minute background grid, so the neural network can be evaluated at arbitrary times within the covered range.