LF Edge eKuiper - An edge lightweight IoT data analytics software

English | 简体中文

Overview

LF Edge eKuiper is an edge lightweight IoT data analytics / streaming software implemented by Golang, and it can be run at all kinds of resource constrained edge devices. One goal of eKuiper is to migrate the cloud streaming software frameworks (such as Apache Spark，Apache Storm and Apache Flink) to edge side. eKuiper references these cloud streaming frameworks, and also considered special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sink, rule engine is used for developing streaming applications at edge side.

User scenarios

It can be run at various IoT edge use scenarios, such as real-time processing of production line data in the IIoT; Gateway of Connected Vehicle analyze the data from data-bus in real time; Real-time analysis of urban facility data in smart city scenarios. eKuiper processing at the edge can reduce system response latency, save network bandwidth and storage costs, and improve system security.

Features

• Lightweight

  • Core server package is only about 4.5M, initial memory footprint is about 10MB

• Cross-platform

  • CPU Arch：X86 AMD * 32, X86 AMD * 64; ARM * 32, ARM * 64; PPC
  • The popular Linux distributions, OpenWrt Linux, MacOS and Docker
  • Industrial PC, Raspberry Pi, industrial gateway, home gateway, MEC edge cloud server

• Data analysis support

  • Support data extract, transform and filter through SQL
  • Data order, group, aggregation and join
  • 60+ functions, includes mathematical, string, aggregate and hash etc
  • 4 time windows & count window

• Highly extensibile

  Plugin system is provided, and it supports to extend at Source, SQL functions and Sink.

  • Source: embedded support for MQTT, and provide extension points for sources
  • Sink: embedded support for MQTT and HTTP, and provide extension points for sinks
  • UDF functions: embedded support for 60+ functions, and provide extension points for SQL functions

• Management

  • A web based management dashboard for nodes, plugins, streams & rules management
  • Plugins, streams and rules management through CLI & REST API
  • Easily be integrate with KubeEdge, K3s and Baetyl, which bases Kubernetes

• Integration with EMQ X Edge

  Seamless integration with EMQ X Neuron & EMQ X Edge, and provided an end to end solution from messaging to analytics.

Quick start

• eKuiper 5 minutes quick start
• EdgeX rule engine tutorial

Community

Join our Slack, and then join ekuiper or ekuiper-user channel.

Meeting

Subscribe to community events calendar.

Weekly community meeting at Friday 10:30AM GMT+8:

• zoom link
• Meeting minutes

Contributing

Thank you for your contribution! Please refer to the CONTRIBUTING.md for more information.

Performance test result

MQTT throughput test

• Using JMeter MQTT plugin to send simulation data to EMQ X Broker, such as: {"temperature": 10, "humidity" : 90}, the value of temperature and humidity are random integer between 0 - 100.
• eKuiper subscribe from EMQ X Broker, and analyze data with SQL: SELECT * FROM demo WHERE temperature > 50
• The analysis result are wrote to local file by using file sink plugin.

Devices	Message # per second	CPU usage	Memory usage
Raspberry Pi 3B+	12k	sys+user: 70%	20M
AWS t2.micro( 1 Core * 1 GB) Ubuntu18.04	10k	sys+user: 25%	20M

EdgeX throughput test

• A Go application is wrote to send data to ZeroMQ message bus, the data is as following.

  {
    "Device": "demo", "Created": 000, …
    "readings": 
    [
       {"Name": "Temperature", value: "30", "Created":123 …},
       {"Name": "Humidity", value: "20", "Created":456 …}
    ]
  }
  

• eKuiper subscribe from EdgeX ZeroMQ message bus, and analyze data with SQL: SELECT * FROM demo WHERE temperature > 50. 90% of data will be filtered by the rule.

• The analysis result are sent to nop sink, all of the result data will be ignored.

	Message # per second	CPU usage	Memory usage
AWS t2.micro( 1 Core * 1 GB) Ubuntu18.04	11.4 k	sys+user: 75%	32M

Max number of rules support

• 8000 rules with 800 message/second in total
• Configurations
  • 2 core * 4GB memory in AWS
  • Ubuntu
• Resource usage
  • Memory: 89% ~ 72%
  • CPU: 25%
  • 400KB - 500KB / rule
• Rule
  • Source: MQTT
  • SQL: SELECT temperature FROM source WHERE temperature > 20 (90% data are filtered)
  • Sink: Log

Multiple rules with shared source instance

• 300 rules with a shared mqtt stream instance.
  • 500 messages/second in the mqtt source
  • 150,000 message processing per second in total
• Configurations:
  • 2 Core * 2GB memory in AWS
  • Ubuntu
• Resource usage
  • Memory: 95MB
  • CPU: 50%
• Rule
  • Source: MQTT
  • SQL: SELECT temperature FROM source WHERE temperature > 20 (90% data are filtered)
  • Sink: 90% nop and 10% MQTT

To run this benchmark by yourself, please check the instruction.

Documents

• Getting started

• Reference guide

  • Install and operation
  • Command line interface tools - CLI
  • eKuiper SQL reference
  • Rules
  • Extend eKuiper
  • Plugins

Build from source

Preparation

• Go version >= 1.13

Compile

• Binary:

  • Binary: $ make

  • Binary files that support EdgeX: $ make build_with_edgex

  • Minimal binary file with core runtime only: $ make build_core

• Packages: $ make pkg

  • Packages: $ make pkg

  • Packages files that support EdgeX: $ make pkg_with_edgex

• Docker images: $ make docker

  Docker images support EdgeX by default

Prebuilt binaries are provided in the release assets. If using os or arch which does not have prebuilt binaries, please use cross-compilation, refer to this doc.

During compilation, features can be selected through go build tags so that users can build a customized product with only the desired feature set to reduce binary size. This is critical when the target deployment environment has resource constraint. Please refer to features for more detail.

Open source license

Apache 2.0