====== Upload system for data measured by stations ======

Measured data upload from stations uses python software [[https://github.com/bolidozor/data-uploader|data-uploader]]. It can be used to upload data of Bolidozor and [[http://wiki.ionozor.eu|Ionozor]] projects. 

Upload to central server works via SSH protocol. It is secured via an ssh-key.

The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database.

Overall it provides a solution for the following tasks:

  * Sorting the [[en:convention|data to folders]]
  * Data upload to server [[http://space.astro.cz/bolidozor/|space.astro.cz]]
  * Checking the free space on the disk (not yet solved)

The whole script is ready to be used with [[en:radio-observer|radio-observer]] - thus it is run with parameter of path to the config file of [[en:radio-observer|radio-observer]]. See an example below:
  ./repos/data-uploader/dataUpload.py ~/bolidozor/station/Bolidozor.json
  
======Installation======

The source files can be find on Github[[https://github.com/bolidozor/RMDS-data-uploader|bolidozor/data-uploader]].

Data-uploader requires the following dependencies:
  
  sudo apt install python-paramiko

Auxiliary libraries: 
  git clone https://github.com/MLAB-project/python-mlab-utils.git
  cd python-mlab-utils/
  sudo python setup.py install develop

Download the script using:
  git clone https://github.com/bolidozor/data-uploader.git
  cd data-uploader
  
After that, you need to have the right key phrases in Bolidozor.json.

==== Configuration ====

In order for data-uploader to function properly, you have to add the following parameters into its configuration file:

  "project": "bolidozor",
  "project_home_folder": "/home/odroid/bolidozor/station/",
  "storage_hostname": "space.astro.cz",
  "storage_username": "obsjaromer",
  "storage_stationpath": "/storage/bolidozor/",
  "storage_protocol": "ssh",

==== Limiting the data flow ====

In some cases, usually observatories with slow data connection, it is advantageous to limit the upload speed from station. You can use a trickle programme to limit the bandwidth of data-uploader.

  trickle -u 200 ./dataUpload.py ~/bolidozor/station/Bolidozor.json

To check the outcome, you can use, for example, bmon or iftop software.

  sudo apt-get install bmon

==== Actualisation of data-uploader from RMDS-data ====
  
  cd ~/repos/
  mv RMDS-data-uploader data-uploader
  cd ~/repos/data-uploader/
  git pull

  cd ~/repos/station-supervisor/
  git pull

  sudo apt install python-paramiko

Add the following parameters to configuration file in front of ‘’”frontend": "tcp_raw",'':

  "project": "bolidozor",
  "project_home_folder": "/home/odroid/bolidozor/station/",
  "storage_hostname": "space.astro.cz",
  "storage_username": "<< uzivatel >>",
  "storage_stationpath": "/storage/bolidozor/",
  "storage_protocol": "ssh",
  
  "HWversion": "RMDS02D",
  
  "observatory": [
      {
          "name": "svakov",
          "owner.login": "kaklik", // GitHub login
          "lat": 49.260658,
          "lon": 14.691537,
          "alt": 445.0,
          "location": "Svakov",	//City, local name
      }
  ],

==== Sorting data to folders ====

The sorting takes place based on the name of the original file. The resulting structure looks as follows:

  /station/data_type/year/month/day/[hour]/file.record

The structure acts in accordance with [[en:convention|Bolidozor network standards]].

During data sorting, the meta-data .cvs records of an current one-hour time interval, are kept on the original place and are only copied to the storage place as the file may be overwritten. 
Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru.
  
==== Upload of data to server ====

Upload to central server is done via scp calling with a request for a check sum M5 on the side of the data server.

The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps. 

  - Synchronising the spectrum snapshots 
  - Synchronising the meta-data files
  - Synchronising the station’s config files
  - Synchronising the RAW meteor records