Measured data upload from stations uses python software data-uploader. It can be used to upload data of Bolidozor and Ionozor projects.

Upload to central server works via SSH protocol. It is secured via an ssh-key.

The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database.

Overall it provides a solution for the following tasks:

• Sorting the data to folders
• Data upload to server space.astro.cz
• Checking the free space on the disk (not yet solved)

The whole script is ready to be used with radio-observer - thus it is run with parameter of path to the config file of radio-observer. See an example below:

./repos/data-uploader/dataUpload.py ~/bolidozor/station/Bolidozor.json

The source files can be find on Githubbolidozor/data-uploader.

Data-uploader requires the following dependencies:

sudo apt install python-paramiko

Auxiliary libraries:

git clone https://github.com/MLAB-project/python-mlab-utils.git
cd python-mlab-utils/
sudo python setup.py install develop

Download the script using:

git clone https://github.com/bolidozor/data-uploader.git
cd data-uploader

After that, you need to have the right key phrases in Bolidozor.json.

In order for data-uploader to function properly, you have to add the following parameters into its configuration file:

"project": "bolidozor",
"project_home_folder": "/home/odroid/bolidozor/station/",
"storage_hostname": "space.astro.cz",
"storage_username": "obsjaromer",
"storage_stationpath": "/storage/bolidozor/",
"storage_protocol": "ssh",

In some cases, usually observatories with slow data connection, it is advantageous to limit the upload speed from station. You can use a trickle programme to limit the bandwidth of data-uploader.

trickle -u 200 ./dataUpload.py ~/bolidozor/station/Bolidozor.json

To check the outcome, you can use, for example, bmon or iftop software.

sudo apt-get install bmon

cd ~/repos/
mv RMDS-data-uploader data-uploader
cd ~/repos/data-uploader/
git pull

cd ~/repos/station-supervisor/
git pull

sudo apt install python-paramiko

Add the following parameters to configuration file in front of ‘’”frontend“: “tcp_raw”,'':

"project": "bolidozor",
"project_home_folder": "/home/odroid/bolidozor/station/",
"storage_hostname": "space.astro.cz",
"storage_username": "<< uzivatel >>",
"storage_stationpath": "/storage/bolidozor/",
"storage_protocol": "ssh",

"HWversion": "RMDS02D",

"observatory": [
    {
        "name": "svakov",
        "owner.login": "kaklik", // GitHub login
        "lat": 49.260658,
        "lon": 14.691537,
        "alt": 445.0,
        "location": "Svakov",	//City, local name
    }
],

The sorting takes place based on the name of the original file. The resulting structure looks as follows:

/station/data_type/year/month/day/[hour]/file.record

The structure acts in accordance with Bolidozor network standards.

During data sorting, the meta-data .cvs records of an current one-hour time interval, are kept on the original place and are only copied to the storage place as the file may be overwritten. Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru.

Upload to central server is done via scp calling with a request for a check sum M5 on the side of the data server.

The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps.

1. Synchronising the spectrum snapshots
2. Synchronising the meta-data files
3. Synchronising the station’s config files
4. Synchronising the RAW meteor records