Bolidozor wiki

User Tools

Site Tools

Trace:
en:data-uploader

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
en:data-uploader [2019/02/28 07:49] – created fluktuaciaen:data-uploader [2019/07/13 12:38] (current) – fluktuacia
Line 1: Line 1:
-FIXME **This page is not fully translated, yet. Please help completing the translation.**\\ //(remove this paragraph once the translation is finished)// +====== Upload system for data measured by stations ======
- +
-====== Upload system for measured data from stations ======+
  
 Measured data upload from stations uses python software [[https://github.com/bolidozor/data-uploader|data-uploader]]. It can be used to upload data of Bolidozor and [[http://wiki.ionozor.eu|Ionozor]] projects.  Measured data upload from stations uses python software [[https://github.com/bolidozor/data-uploader|data-uploader]]. It can be used to upload data of Bolidozor and [[http://wiki.ionozor.eu|Ionozor]] projects. 
Line 7: Line 5:
 Upload to central server works via SSH protocol. It is secured via an ssh-key. Upload to central server works via SSH protocol. It is secured via an ssh-key.
  
-The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database (?? database index, indexační databáze).+The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database.
  
 Overall it provides a solution for the following tasks: Overall it provides a solution for the following tasks:
Line 35: Line 33:
   cd data-uploader   cd data-uploader
      
-After that, you need to have the right key phrases in Bolidozor.json (?? kličové výrazy).+After that, you need to have the right key phrases in Bolidozor.json.
  
 ==== Configuration ==== ==== Configuration ====
  
-In order to data-uploader function properly, you have to add the following parameters in its configuration file:+In order for data-uploader to function properly, you have to add the following parameters into its configuration file:
  
   "project": "bolidozor",   "project": "bolidozor",
Line 50: Line 48:
 ==== Limiting the data flow ==== ==== Limiting the data flow ====
  
-In some cases, usually observatories with slow data connection, it is suitable to limit the upload speed from station. You can use trickle programme to limit the bandwidth of data-uploader.+In some cases, usually observatories with slow data connection, it is advantageous to limit the upload speed from station. You can use a trickle programme to limit the bandwidth of data-uploader.
  
   trickle -u 200 ./dataUpload.py ~/bolidozor/station/Bolidozor.json   trickle -u 200 ./dataUpload.py ~/bolidozor/station/Bolidozor.json
  
-To check the outcome, you can use for example bmon or iftop programmes.+To check the outcome, you can use, for example, bmon or iftop software.
  
   sudo apt-get install bmon   sudo apt-get install bmon
  
-==== Actualisation of data-uploader from RMDS-data-uploaderu====+==== Actualisation of data-uploader from RMDS-data ====
      
   cd ~/repos/   cd ~/repos/
Line 70: Line 68:
   sudo apt install python-paramiko   sudo apt install python-paramiko
  
-Add the following parameters to configuration file before ‘’”frontend": "tcp_raw",'':+Add the following parameters to configuration file in front of ‘’”frontend": "tcp_raw",'':
  
   "project": "bolidozor",   "project": "bolidozor",
Line 92: Line 90:
   ],   ],
  
-==== Sorting the data to folders ====+==== Sorting data to folders ====
  
 The sorting takes place based on the name of the original file. The resulting structure looks as follows: The sorting takes place based on the name of the original file. The resulting structure looks as follows:
  
-  /station/typ dat/rok/měsíc/den/[hodina]/soubor.záznamu+  /station/data_type/year/month/day/[hour]/file.record
  
 The structure acts in accordance with [[en:convention|Bolidozor network standards]]. The structure acts in accordance with [[en:convention|Bolidozor network standards]].
  
-During data sorting, the meta-data .cvs records of an current time (??z aktuální hodiny) are kept on the original place and are only copied to the storage place as the file may be overwritten. +During data sorting, the meta-data .cvs records of an current one-hour time interval, are kept on the original place and are only copied to the storage place as the file may be overwritten. 
 Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru. Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru.
      
Line 108: Line 106:
  
 The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps.  The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps. 
-Skript každé 2 minuty zkontroluje připojeni k internetu (dostupnost space.astro.cz) a v případě autentizace se spustí synchronizace dat. Synchronizace probíhá postupně ve čtyřech krocích: 
  
   - Synchronising the spectrum snapshots    - Synchronising the spectrum snapshots 
en/data-uploader.1551340187.txt.gz · Last modified: 2019/02/28 07:49 by fluktuacia