# IGRA format to FSL format converter
**Scott Hamilton, November 21st, 2024**

## Why?
The radiosonde database at NOAA was recently closed, which is pretty unfortunate as that's where RapidAIR sourced its .FSL files for driving AERMET.

The IGRA database is an alternative source of upper air soundings, but the data is in a different format. In the NOAA database, files were organised by site and year. In the IGRA database all soundings for each site are in a single massive text file (but they're still in the wrong format- annoying am I right?).

## What?
Two scripts have been prepared which a) acquire the files from the IGRA website automatically, and b) convert the files to .FSL format data that AERMET understands.

1) The downloader- `download_igra.py`
2) The converter- `convert_igra_to_fsl.py`

The logic in the downloader is straightforward enough. 

The converter.... less so.... soundings data is.... messy (or so it appears, it actually follows a very rigid pattern).

To create the logic to make the conversion I used the fortran source code for AERMET to understand how it reads FSL files at run time, and using some methods similar to the fortran in MMIF I wrote python code to similar rules to enable the translation between formats. This gave a row by row 'fixed width format' structure that must be followed if the data is to work with AERMET. Basically each header and subsequent 4 rows follows a rigid pattern made up of dates, site details and the like. After these rows the radiosonde data is arranged in a table, following a similar columnar structure to the header information. 

## How?
To **run the downloader**, open the `download_igra.py` script in a python environment with `pandas`, and add the country code you want at the bottom (e.g. UK). Then run from the command line, the tool will download all IGRA files with the UK country code. 

To **run the converter**, at the bottom of the `convert_igra_to_fsl.py` script, change the file list step to the folder with your IGRA files. Set the other variables for output_folder and station metadata file (which is beside the scripts). The tool will convert all available years in the IGRA file into individual FSL files you can use with AERMET. We will run this each year instead of downloading all the FSL data from NOAA, which was kinda faffy anyway.

**There is a demo dataset** in the `demo_dataset` folder you can try the converter out on. Pass that folder into the arguments at the bottom of the script and it will make all the FSL files that can be made from that single (massive) IGRA file.
 
## Do I have to?
Well, not really if you only want the files for RapidAIR. I'll be adding the 2023 data very soon from this converter and the 2024 data early in the new year to the database RapidAIR tirelessly searches on your behalf when you run the model- so you won't have to do anything to adopt this new data. 

`ta`

`scott.hamilton@ricardo.com`

`November 21st 2024`