Hasil Pencarian  ::  Simpan CSV :: Kembali

Abstrak :
The dynamic development of digital technologies, especially those dedicated to devices generating large data streams, such as all kinds of measurement equipment temperature and humidity sensors, cameras, radio telescopes and satellites Internet of Things enables more in depth analysis of the surrounding reality, including better understanding of various natural phenomenon, starting from atomic level reactions, through macroscopic processes e.g. meteorology to observation of the Earth and the outer space. On the other hand such a large quantitative improvement requires a great number of processing and storage resources, resulting in the recent rapid development of Big Data technologies. Since 2015, the European Space Agency ESA has been providing a great amount of data gathered by exploratory equipment a collection of Sentinel satellites which perform Earth observation using various measurement techniques. For example Sentinel 2 provides a stream of digital photos, including images of the Baltic Sea and the whole territory of Poland. This data is used in an experimental installation of a Big Data processing system based on the open source software at the Academic Computer Center in Gdansk. The center has one of the most powerful supercomputers in Poland the Tryton computing cluster, consisting of 1600 nodes interconnected by a fast Infiniband network 56 Gbps and over 6 PB of storage. Some of these nodes are used as a computational cloud supervised by an OpenStack platform, where the Sentinel 2 data is processed. A subsystem of the automatic, perpetual data download to object storage based on Swift is deployed, the required software libraries for the image processing are configured and the Apache Spark cluster has been set up. The above system enables gathering and analysis of the recorded satellite images and the associated metadata, benefiting from the parallel computation mechanisms. This paper describes the above solution including its technical aspects.

Abstrak :
Penelitian ini bertujuan untuk menganalisis throughput dan latensi Kafka dan RabbitMQ sebagai message broker pada proyek Mata Elang. Percobaan dilakukan dengan scenario 10 kali putaran dengan menggunakan file PCAP yang terdiri dari smallflow.PCAP dan bigflow.PCAP. Perbedaan nilai throughput pada pengujian menggunakan Kafka dan RabbitMQ didapatkan cukup signifikan baik pada scenario pengujian menggunakan smallflow.PCAP (p= 0,002) dan bigflow.PCAP (p=0,003). Pada pengujian dengan scenario menggunakan smallflow.PCAP didapatkan rata-rata throughput untuk Kafka dan RabbitMQ masing-masing sebesar 0,13 ± 0,03 pps dan 0,10 ± 0,01 pps. Sementara itu pada scenario pengujian menggunakan bigflow.PCAP didapatkan rata-rata throughput untuk Kafka dan RabbitMQ masing-masing sebesar 0,21 ± 0,07 dan 0,11 ± 0,02. Perbedaan nilai latensi pada pengujian menggunakan Kafka dan RabbitMQ didapatkan cukup signifikan baik pada scenario pengujian menggunakan smallflow.PCAP (p= 0,002) dan bigflow.PCAP (p=0,003). Pada pengujian dengan scenario menggunakan smallflow.PCAP didapatkan rata-rata latensi untuk Kafka dan RabbitMQ masing-masing sebesar 8,26 ± 3,51 sekon dan 9,73 ± 0,95 sekon. Sementara itu pada scenario pengujian menggunakan bigflow.PCAP didapatkan rata-rata throughput untuk Kafka dan RabbitMQ masing-masin sebesar 5,06 ± 1,23 sekon dan 7,20 ± 0,47 sekon. ...... This study aims to analyze the throughput and latency of Kafka and RabbitMQ as message brokers in the Mata Elang project. Experiments were conducted with 10 rounds of testing using PCAP files consisting of smallflow.PCAP and bigflow.PCAP. The difference in throughput values in the testing using Kafka and RabbitMQ was found to be significant in both the smallflow.PCAP scenario (p=0.002) and the bigflow.PCAP scenario (p=0.003). In the testing scenario using smallflow.PCAP, the average throughput for Kafka and RabbitMQ was 0.13 ± 0.03 pps and 0.10 ± 0.01 pps, respectively. Meanwhile, in the testing scenario using bigflow.PCAP, the average throughput for Kafka and RabbitMQ was 0.21 ± 0.07 pps and 0.11 ± 0.02 pps, respectively. The difference in latency values in the testing using Kafka and RabbitMQ was found to be significant in both the smallflow.PCAP scenario (p=0.002) and the bigflow.PCAP scenario (p=0.003). In the testing scenario using smallflow.PCAP, the average latency for Kafka and RabbitMQ was 8.26 ± 3.51 seconds and 9.73 ± 0.95 seconds, respectively. Meanwhile, in the testing scenario using bigflow.PCAP, the average latency for Kafka and RabbitMQ was 5.06 ± 1.23 seconds and 7.20 ± 0.47 seconds.