[中文版]
Forward is a library for high-performance deep learning inference on NVIDIA GPUs. It provides a well-designed scheme that directly parses Tensorflow/PyTorch/Keras/ONNX models to the high-performance engine based on TensorRT. Compared to TensorRT, it is easy-to-use and easy-to-expand. So far, Forward supports not only mainstream deep learning models in CV, NLP, and Recommend fields, but also some advanced models such as BERT, GAN, FaceSwap, StyleTransfer.
- High-performance optimization: utilize TensorRT API and customized operators for high-performance deep learning inference;
- Wide range support: support advanced models such as BERT, GAN, FaceSwap, and StyleTransfer besides the mainstream deep learning models in CV, NLP, and Recommend fields;
- Multiple modes: support FLOAT/HALF/INT8 infer modes;
- Easy to use: load Tensorflow(.pb)/PyTorch(.pth)/Keras(.h5)/ONNX(.onnx) models directly, followed by inferencing with TensorRT;
- Easy to expand: register customized layers according to add_support_op.md;
- Provide C++ and Python interfaces.
- NVIDIA CUDA >= 10.0, CuDNN >= 7 (Recommended version: >= CUDA 10.2)
- TensorRT >= 7.0.0.11 (Recommended version: TensorRT-7.2.1.6)
- CMake >= 3.12.2
- GCC >= 5.4.0, ld >= 2.26.1
- PyTorch >= 1.7.0
- TensorFlow >= 1.15.0 (For Linux users, download Tensorflow 1.15.0 and unzip all
.so
files underForward/source/third_party/tensorflow/lib
folder) - Keras HDF5 (Built from
Forward/source/third_party/hdf5
as default)
Use CMake to generate Makefiles or Visual Studio project (Windows). Upon the usage, Forward is able to be built for different frameworks, such as Fwd-Torch, Fwd-Python-Torch, Fwd-Tf, Fwd-Python-Tf, Fwd-Keras, Fwd-Python-Keras, Fwd-Onnx, and Fwd-Python-Onnx.
The example below shows the building workflow under the Linux system,
Step 1: clone the project
1 git clone https://github.com/Tencent/Forward.git
Step 2: download Tensorflow 1.15.0
(only needed when using Forward with Tensorflow framework under Linux)
1 cd Forward/source/third_party/tensorflow/
2 wget https://github.com/neargye-forks/tensorflow/releases/download/v1.15.0/libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz
3 tar -xvf libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz
Step 3: create build
folder
1 cd ~/Forward/
2 rm -rf build
3 mkdir -p build
4 cd build/
Step 4: run cmake
to create dependencies and TensorRT_ROOT
installation path should be specified
1 cmake .. -DTensorRT_ROOT=<path_to_TensorRT> -DENABLE_TENSORFLOW=ON -DENABLE_UNIT_TESTS=ON
Step 5: run make
to build the project
1 make -j
Step 6: run unit_test
to check if the project has been built built successfully
cd bin/
./unit_test --gtest_filter=TestTfNodes.*
# The following lines show the success of building the project
# [ OK ] TestTfNodes.ZeroPadding (347 ms)
# [----------] 22 tests from TestTfNodes (17555 ms total)
# [----------] Global test environment tear-down
# [==========] 22 tests from 1 test case ran. (17555 ms total)
# [ PASSED ] 22 tests.
For more information, please refer to CMake Build.
Refer to Demo for using Forward-Cpp in Linux
Refer to Demo for using Forward-Python
Refer to Demo for using Forward-Bert
Notice: the model INPUT name can be viewed by model viewers, such as Netron.
Forward use easylogging++ for logging, and use forward_log.conf
as the log configuration file.
- If
forward_log.conf
is existed under the workspace directory, Forward will use this file as the configuration file. For more information, please refer to Using-configuration-file; - If
forward_log.conf
is not existed under the workspace directory, Forward will use its default settings and save logging information inlogs/myeasylog.log
.
Example for the log configuration file forward_log.conf
* GLOBAL:
FORMAT = "[%level] %datetime %fbase(%line): %msg"
FILENAME = "Forward.log"
ENABLED = true
TO_FILE = true
TO_STANDARD_OUTPUT = true
PERFORMANCE_TRACKING = true
MAX_LOG_FILE_SIZE = 2097152 ## 2MB - Comment starts with two hashes (##)
LOG_FLUSH_THRESHOLD = 100 ## Flush after every 100 logs
All the models and operators supported by Forward are listed below. If the one you are looking for is not listed here, please create a new Issue. We also welcome developers to contribute to this project together. Please refer to CONTRIBUTING.md.
- Welcome to join our discussion group, Tencent QQ Group ID: 776314438.
- Please refer to CONTRIBUTING.md about how to contribute to this project.
Aster JIAN |
Zexi YUAN |
Ao LI |
Paul LU |
Zhaoyi LUO |
Jett Hu |
Ryosuke1eep |
Any form of contribution is welcome. The above contributors have been officially released by Tencent.
We very much welcome developers to contribute to Tencent's open source, and we will also give them incentives to acknowledge and thank them. Here we provide an official description of Tencent's open source contribution. Specific contribution rules for each project are formulated by the project team. Developers can choose the appropriate project and participate according to the corresponding rules. The Tencent Project Management Committee will report regularly to qualified contributors and awards will be issued by the official contact.