Page locked

This page is currently locked for editing by another user. You have to wait until this user finishes editing or the lock expires.
Currently locked by: 13.58.28.53
Lock expires at: 2025/04/12 10:52 (7 min)
This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
FIXME **This page is not fully translated, yet. Please help completing the translation.**\\ //(remove this paragraph once the translation is finished)//

====== Upload system for data measured by stations ======

Measured data upload from stations uses python software [[https://github.com/bolidozor/data-uploader|data-uploader]]. It can be used to upload data of Bolidozor and [[http://wiki.ionozor.eu|Ionozor]] projects. 

Upload to central server works via SSH protocol. It is secured via an ssh-key.

The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database (?? database index, indexační databáze).

Overall it provides a solution for the following tasks:

  * Sorting the [[en:convention|data to folders]]
  * Data upload to server [[http://space.astro.cz/bolidozor/|space.astro.cz]]
  * Checking the free space on the disk (not yet solved)

The whole script is ready to be used with [[en:radio-observer|radio-observer]] - thus it is run with parameter of path to the config file of [[en:radio-observer|radio-observer]]. See an example below:
  ./repos/data-uploader/dataUpload.py ~/bolidozor/station/Bolidozor.json
  
======Installation======

The source files can be find on Github[[https://github.com/bolidozor/RMDS-data-uploader|bolidozor/data-uploader]].

Data-uploader requires the following dependencies:
  
  sudo apt install python-paramiko

Auxiliary libraries: 
  git clone https://github.com/MLAB-project/python-mlab-utils.git
  cd python-mlab-utils/
  sudo python setup.py install develop

Download the script using:
  git clone https://github.com/bolidozor/data-uploader.git
  cd data-uploader
  
After that, you need to have the right key phrases in Bolidozor.json (?? kličové výrazy).

==== Configuration ====

In order for data-uploader to function properly, you have to add the following parameters into its configuration file:

  "project": "bolidozor",
  "project_home_folder": "/home/odroid/bolidozor/station/",
  "storage_hostname": "space.astro.cz",
  "storage_username": "obsjaromer",
  "storage_stationpath": "/storage/bolidozor/",
  "storage_protocol": "ssh",

==== Limiting the data flow ====

In some cases, usually observatories with slow data connection, it is advantageous to limit the upload speed from station. You can use a trickle programme to limit the bandwidth of data-uploader.

  trickle -u 200 ./dataUpload.py ~/bolidozor/station/Bolidozor.json

To check the outcome, you can use, for example, bmon or iftop software.

  sudo apt-get install bmon

==== Actualisation of data-uploader from RMDS-data ====
  
  cd ~/repos/
  mv RMDS-data-uploader data-uploader
  cd ~/repos/data-uploader/
  git pull

  cd ~/repos/station-supervisor/
  git pull

  sudo apt install python-paramiko

Add the following parameters to configuration file in front of ‘’”frontend": "tcp_raw",'':

  "project": "bolidozor",
  "project_home_folder": "/home/odroid/bolidozor/station/",
  "storage_hostname": "space.astro.cz",
  "storage_username": "<< uzivatel >>",
  "storage_stationpath": "/storage/bolidozor/",
  "storage_protocol": "ssh",
  
  "HWversion": "RMDS02D",
  
  "observatory": [
      {
          "name": "svakov",
          "owner.login": "kaklik", // GitHub login
          "lat": 49.260658,
          "lon": 14.691537,
          "alt": 445.0,
          "location": "Svakov",	//City, local name
      }
  ],

==== Sorting the data to folders ====

The sorting takes place based on the name of the original file. The resulting structure looks as follows:

  /station/typ dat/rok/měsíc/den/[hodina]/soubor.záznamu

The structure acts in accordance with [[en:convention|Bolidozor network standards]].

During data sorting, the meta-data .cvs records of an current time (??z aktuální hodiny) are kept on the original place and are only copied to the storage place as the file may be overwritten. 
Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru.
  
==== Upload of data to server ====

Upload to central server is done via scp calling with a request for a check sum M5 on the side of the data server.

The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps. 
Skript každé 2 minuty zkontroluje připojeni k internetu (dostupnost space.astro.cz) a v případě autentizace se spustí synchronizace dat. Synchronizace probíhá postupně ve čtyřech krocích:

  - Synchronising the spectrum snapshots 
  - Synchronising the meta-data files
  - Synchronising the station’s config files
  - Synchronising the RAW meteor records