Overview
EMQ X Kuiper is an edge lightweight IoT data analytics / streaming software implemented by Golang, and it can be run at all kinds of resource constrained edge devices. One goal of Kuiper is to migrate the cloud streaming software frameworks (such as Apache Spark, Apache Storm and Apache Flink) to edge side. Kuiper references these cloud streaming frameworks, and also considered special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sink, rule engine is used for developing streaming applications at edge side.
User scenarios
It can be run at various IoT edge use scenarios, such as real-time processing of production line data in the IoT; Gateway of Connected Vehicle analyze the data from data-bus in real time; Real-time analysis of urban facility data in smart city scenarios. Kuiper processing at the edge can reduce system response latency, save network bandwidth and storage costs, and improve system security
Features
•
Lightweight
◦
Core server package is only about 4.5M, initial memory footprint is about 10MB
•
Cross-platform
◦
CPU Arch: X86 AMD * 32, X86 AMD * 64; ARM * 32, ARM * 64; PPC
◦
The popular Linux distribuitions, OpenWrt Linux, MacOS and Docker
◦
Industrial PC, Raspberry Pi, Industrial gateway, home gateway, MEC edge cloud server
•
Data analysis support
◦
Support data extract, transform and filter through SQL
◦
Data order, group, aggregation and join
◦
60+ functions, includes mathematical, string, aggregate and hash etc
◦
4 time windows & count window
•
Highly extensibile
Plugin system is provided, and it supports to extend at Source, SQL functions and Sink.
◦
Source: embedded supoort for MQTT, and provide extension points for sources
◦
Sink: embedded support for MQTT and HTTP, and provide extension points for sinks
◦
UDF functions: embedded support for 60+ functions, and provide extension points for SQL functions.
•
Management
◦
A web based management dashboard for nodes, plugins, streams & rules management
◦
Plugins, streams and rules management through CLI & REST API
◦
Easily be integrate with KubeEdge, K3s and Baetyl, which bases Kubernetes
Quick start
Performance test result
MQTT throughput test
•
Using JMeter MQTT plugin to send simulation data to EMQ X Broker, such as: {"temperature": 10, "humidity" : 90}, the value of temperature and humidity are random integer between 0 - 100.
•
Kuiper subscribe from EMQ X Broker, and analyze data with SQL: SELECT * FROM demo WHERE temperature > 50
•
EdgeX throughput test
생략.
Max number of rules support
•
8000 rules with 800 message/second
•
Configurations
◦
2 core * 4GB memory in AWS
◦
Ubuntu
•
Resource usage
◦
Memory: 72% ~ 89%
◦
CPU: 25%
◦
400KB - 500KB / rule
•
Rule
◦
Source: MQTT
◦
SQL: SELECT temperature FROM source WHERE temperature > 20 (90% data are filtered)
◦
Sink: Log
Documents
Build from source
Preparation
•
Go version >= 1.13
Compile
•
Binary:
◦
Binary: $ make
◦
Binary files that support EdgeX: $ make build_with_edgex
•
Packages: $ make pkg
◦
Packages: $ make pkg
◦
Packages files that support EdgeX: $ make pkg_with_edgex
•
Docker images: $ make docker
Docker images support EdgeX by default
