en:data-uploader
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
en:data-uploader [2019/04/29 13:25] – [Configuration] fluktuacia | en:data-uploader [2019/07/13 12:38] (current) – fluktuacia | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | FIXME **This page is not fully translated, yet. Please help completing the translation.**\\ //(remove this paragraph once the translation is finished)// | ||
- | |||
====== Upload system for data measured by stations ====== | ====== Upload system for data measured by stations ====== | ||
Line 7: | Line 5: | ||
Upload to central server works via SSH protocol. It is secured via an ssh-key. | Upload to central server works via SSH protocol. It is secured via an ssh-key. | ||
- | The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database | + | The software checks the integrity of an uploaded file during the process of data upload using a check sums. At the same time it provides an information about the new file on server to the indexing database. |
Overall it provides a solution for the following tasks: | Overall it provides a solution for the following tasks: | ||
Line 35: | Line 33: | ||
cd data-uploader | cd data-uploader | ||
| | ||
- | After that, you need to have the right key phrases in Bolidozor.json | + | After that, you need to have the right key phrases in Bolidozor.json. |
==== Configuration ==== | ==== Configuration ==== | ||
Line 50: | Line 48: | ||
==== Limiting the data flow ==== | ==== Limiting the data flow ==== | ||
- | In some cases, usually observatories with slow data connection, it is suitable | + | In some cases, usually observatories with slow data connection, it is advantageous |
trickle -u 200 ./ | trickle -u 200 ./ | ||
- | To check the outcome, you can use for example bmon or iftop programmes. | + | To check the outcome, you can use, for example, bmon or iftop software. |
sudo apt-get install bmon | sudo apt-get install bmon | ||
- | ==== Actualisation of data-uploader from RMDS-data-uploaderu==== | + | ==== Actualisation of data-uploader from RMDS-data ==== |
| | ||
cd ~/repos/ | cd ~/repos/ | ||
Line 70: | Line 68: | ||
sudo apt install python-paramiko | sudo apt install python-paramiko | ||
- | Add the following parameters to configuration file before | + | Add the following parameters to configuration file in front of ‘’”frontend": |
" | " | ||
Line 92: | Line 90: | ||
], | ], | ||
- | ==== Sorting | + | ==== Sorting data to folders ==== |
The sorting takes place based on the name of the original file. The resulting structure looks as follows: | The sorting takes place based on the name of the original file. The resulting structure looks as follows: | ||
- | /station/typ dat/rok/měsíc/den/[hodina]/soubor.záznamu | + | /station/data_type/year/month/day/[hour]/file.record |
The structure acts in accordance with [[en: | The structure acts in accordance with [[en: | ||
- | During data sorting, the meta-data .cvs records of an current time (??z aktuální hodiny) | + | During data sorting, the meta-data .cvs records of an current |
Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru. | Při rozřazování dat jsou meta-datové .csv záznamy z aktuální hodiny zachovány na původním místě a pouze překopírovány na úložiště kvůli možnosti dalšího zápisu do tohoto souboru. | ||
| | ||
Line 108: | Line 106: | ||
The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps. | The script checks the internet connection (availability of space.astro.cz) every 2 minutes and in case of authentication it begins the data synchronisation. The synchronisation takes place in four steps. | ||
- | Skript každé 2 minuty zkontroluje připojeni k internetu (dostupnost space.astro.cz) a v případě autentizace se spustí synchronizace dat. Synchronizace probíhá postupně ve čtyřech krocích: | ||
- Synchronising the spectrum snapshots | - Synchronising the spectrum snapshots |
en/data-uploader.1556544341.txt.gz · Last modified: 2019/04/29 13:25 by fluktuacia