User Tools

Site Tools

Translations of this page?:

en:data-uploader

This is an old revision of the document!


FIXME This page is not fully translated, yet. Please help completing the translation.
(remove this paragraph once the translation is finished) ??

Upload system for data measured by stations

Measured data upload from stations uses python software data-uploader. It can be used to upload data of Bolidozor and Ionozor projects.

Upload to central server works via SSH protocol. It is secured via an ssh-key.

The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database.

Overall it provides a solution for the following tasks:

The whole script is ready to be used with radio-observer - thus it is run with parameter of path to the config file of radio-observer. See an example below:

./repos/data-uploader/dataUpload.py ~/bolidozor/station/Bolidozor.json

Installation

The source files can be find on Githubbolidozor/data-uploader.

Data-uploader requires the following dependencies:

sudo apt install python-paramiko

Auxiliary libraries:

git clone https://github.com/MLAB-project/python-mlab-utils.git
cd python-mlab-utils/
sudo python setup.py install develop

Download the script using:

git clone https://github.com/bolidozor/data-uploader.git
cd data-uploader

After that, you need to have the right key phrases in Bolidozor.json.

Configuration

In order for data-uploader to function properly, you have to add the following parameters into its configuration file:

"project": "bolidozor",
"project_home_folder": "/home/odroid/bolidozor/station/",
"storage_hostname": "space.astro.cz",
"storage_username": "obsjaromer",
"storage_stationpath": "/storage/bolidozor/",
"storage_protocol": "ssh",

Limiting the data flow

In some cases, usually observatories with slow data connection, it is advantageous to limit the upload speed from station. You can use a trickle programme to limit the bandwidth of data-uploader.

trickle -u 200 ./dataUpload.py ~/bolidozor/station/Bolidozor.json

To check the outcome, you can use, for example, bmon or iftop software.

sudo apt-get install bmon

Actualisation of data-uploader from RMDS-data

cd ~/repos/
mv RMDS-data-uploader data-uploader
cd ~/repos/data-uploader/
git pull
cd ~/repos/station-supervisor/
git pull
sudo apt install python-paramiko

Add the following parameters to configuration file in front of ‘’”frontend“: “tcp_raw”,'':

"project": "bolidozor",
"project_home_folder": "/home/odroid/bolidozor/station/",
"storage_hostname": "space.astro.cz",
"storage_username": "<< uzivatel >>",
"storage_stationpath": "/storage/bolidozor/",
"storage_protocol": "ssh",

"HWversion": "RMDS02D",

"observatory": [
    {
        "name": "svakov",
        "owner.login": "kaklik", // GitHub login
        "lat": 49.260658,
        "lon": 14.691537,
        "alt": 445.0,
        "location": "Svakov",	//City, local name
    }
],

Sorting data to folders

The sorting takes place based on the name of the original file. The resulting structure looks as follows:

/station/data_type/year/month/day/[hour]/file.record

The structure acts in accordance with Bolidozor network standards.

During data sorting, the meta-data .cvs records of an current one-hour time interval, are kept on the original place and are only copied to the storage place as the file may be overwritten. Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru.

Upload of data to server

Upload to central server is done via scp calling with a request for a check sum M5 on the side of the data server.

The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps.

  1. Synchronising the spectrum snapshots
  2. Synchronising the meta-data files
  3. Synchronising the station’s config files
  4. Synchronising the RAW meteor records
en/data-uploader.1563021486.txt.gz · Last modified: 2019/07/13 14:38 by fluktuacia