English | 中文
Linkis builds a layer of computation middleware between upper applications and underlying engines. By using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc., and achieve the intercommunication of user resources like unified variables, scripts, UDFs, functions and resource files at the same time.
As a computation middleware, Linkis provides powerful connectivity, reuse, expansion, and computation governance capabilities. By decoupling the application layer and the engine layer, it simplifies the complex network call relationship, and thus reduces the overall complexity and saves the development and maintenance costs as well.
Since the first release of Linkis in 2019, it has accumulated more than 700 trial companies and 1000+ sandbox trial users, which involving diverse industries, from finance, banking, tele-communication, to manufactory, internet companies and so on. Lots of companies have already used Linkis as a unified entrance for the underlying computation and storage engines of the big data platform.
Based on the concept of the computation middleware architecture of Linkis, we have built a large amount of applications and systems on top of it. The following is the currently available open-source projects:
More upcoming tools ready to be open-sourced, please stay tuned!
| Engine | Description | Contributor | Version | Released? | User Manual |
|---|---|---|---|---|---|
| Flink | Flink EngineConn. Supports the submission of FlinkSQL code, and also supports the submission of Flink Jar to Linkis Manager to start a new Yarn application. | hui zhu & WeDataSphere | Linkis-1.1.0 | No, Expected in mid-2021 | To be perfected |
| Impala | Impala EngineConn. Supports the submission of Impala SQL code. | hui zhu | Linkis-0.12.0 | No, Expected in April | To be perfected |
| Presto | Presto EngineConn. Supports the submission of Presto SQL code. | wForget | Linkis-0.11.0 | Open sourced | Presto User Manual |
| ElasticSearch | ElasticSearch EngineConn. Supports the submission of SQL and DSL code. | wForget | Linkis-0.11.0 | Open sourced | ElasticSearch User Manual |
| Shell | Shell EngineConn. Supports the submission of shell code. | WeDataSphere | Linkis-0.9.3 | Open sourced | Shell User Manual |
| MLSQL | MLSQL EngineConn. Supports the submission of MLSQL code. | WilliamZhu | Linkis-0.9.1 | Open sourced | MLSQL User Manual |
| JDBC | JDBC EngineConn. Supports the submission of MySQL and HiveQL code. | WeDataSphere | Linkis-0.9.0 | Open sourced | JDBC User Manual |
| Spark | Spark EngineConn. Supports the submission of SQL, Scala, Pyspark and R code. | WeDataSphere | Linkis-0.5.0 | Open sourced | Spark User Manual |
| Hive | Hive EngineConn. Supports the submission of HiveQL code. | WeDataSphere | Linkis-0.5.0 | Open sourced | Hive User Manual |
| Python | Python EngineConn. Supports the submission of python code. | WeDataSphere | Linkis-0.5.0 | Open sourced | Python User Manual |
| TiSpark | TiSpark EngineConn. Support querying TiDB data by SparkSQL. | WeDataSphere | Linkis-0.5.0 | Open sourced | To be perfected |
Linkis uses GitBook for management, and the entire project will be organized into a GitBook e-book, which is convenient for everyone to download and use.
WeDataSphere will provide a unified document reading entrance in the future. For the usage of GitBook, please refer to: GitBook User Manual
Before contributing, please read the Linkis Contribution Specification to understand how to contribute and submit your contribution.
Linkis is under the Apache 2.0 license. See the License file for details.