Skip to content

Commit 1bea5e0

Browse files
Akanksha-kediaAkanksha-kedia
committed
BIGTOP-3959: RPM packaging for TRINO
1 parent 9fca8d8 commit 1bea5e0

File tree

21 files changed

+1166
-10
lines changed

21 files changed

+1166
-10
lines changed

‎bigtop-packages/src/common/ranger/install_ranger.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@ else
146146
COMPONENT_LIB_DIR=${PREFIX}/${RANGER_COMPONENT_DIR}/tomcat/webapps/kylin/WEB-INF/lib
147147
elif [ "${RANGER_COMPONENT}" = "elasticsearch" ]; then
148148
COMPONENT_LIB_DIR=${PREFIX}/${RANGER_COMPONENT_DIR}/plugins
149-
elif [ "${RANGER_COMPONENT}" = "presto" ]; then
149+
elif [ "${RANGER_COMPONENT}" = "trino" ]; then
150150
COMPONENT_LIB_DIR=${PREFIX}/${RANGER_COMPONENT_DIR}/plugin/ranger
151151
if [ ! -d "${COMPONENT_LIB_DIR}" ]; then
152152
echo "INFO: Creating ${COMPONENT_LIB_DIR}"

‎bigtop-packages/src/common/trino/developers.rst

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
For Developers: Building a component from Git repository
2+
3+
Prerequisites
4+
5+
You will need git installed.
6+
You will need java 17 installed for Trino
7+
Java 8 for bigtop
8+
You will need to use gradlew which is included in the source code. (Right in the root of the project folder)
9+
This project's gradlew has more documentation here
10+
Use git to download BigTop :
11+
git clone https://github.com/apache/bigtop.git
12+
13+
move into the root project folder:
14+
cd bigtop
15+
16+
To fetch source from a Git repository, there're two ways to achieve this: a). modify ./bigtop.bom and add JSON snippets to your component/package, or b). specify properties at command line
17+
18+
bigtop.bom
19+
Add following JSON snippets to the desired component/package:
20+
21+
git { repo = ""; ref = ""; dir = ""; commit_hash = "" }
22+
repo - SSH, HTTP or local path to Git repo.
23+
ref - branch, tag or commit hash to check out.
24+
dir - [OPTIONAL] directory name to write source into.
25+
commit_hash - [OPTIONAL] a commit hash to reset to.
26+
Some packages have different names for source directory and source tarball (hbase-0.98.5-src.tar.gz contains hbase-0.98.5 directory). By default source will be fetched in a directory named by tarball { source = TARBALL_SRC } without .t* extension. To explicitly set directory name use the dir option.
27+
28+
When commit_hash specified, the repo to build the package will be reset to the commit hash.

‎bigtop-packages/src/common/trino/do-component-build

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/bin/bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
set -ex
18+
19+
20+
. `dirname $0`/bigtop.bom
21+
mkdir trino
22+
export trino_java_url=https://artifactory.trusted.visa.com/middleware-release/JAVA/OPENJDK/17.0.7/bellsoft-jdk17.0.7+7-linux-amd64.tar.gz
23+
wget --user="${PIPER_CUSTOM_USER}" --password="${PIPER_CUSTOM_PASSWORD}" -O java.tar.gz "${trino_java_url}" --no-check-certificate
24+
tar zxvf java.tar.gz && rm -rf java.tar.gz
25+
OLD_JAVA=$JAVA_HOME
26+
echo JAVA_HOME=`pwd`/`ls | grep "jdk-*"` >> ~/.bash_profile
27+
source ~/.bash_profile
28+
29+
# Setting versions for trino build
30+
mvn clean -pl !plugin/trino-kafka,!testing/trino-benchto-benchmarks,!docs install -DskipTests -Dcheckstyle.skipExec -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Djavax.net.ssl.trustStore=/home/akedia/bigtop/cacerts -Djavax.net.ssl.trustStorePassword=changeit
31+
sed -i '$ d' ~/.bash_profile
32+
source ~/.bash_profile
33+
rm -rf `pwd`/`ls | grep "jdk-*"`
34+
35+
mkdir -p build/trino
36+
mkdir -p build/trino-cli
37+
38+
# Server
39+
tar -C build/trino --strip-components=1 -xzf core/trino-server/target/*.tar.gz
40+
# CLI
41+
cp -ra client/trino-cli/target/*-executable.jar build/trino-cli/trino

‎bigtop-packages/src/common/trino/getting-started.rst

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
Getting started with bigtop packaging.
2+
3+
1) You will have to package rpm.
4+
5+
2) Here are some steps you can follow to bring a new bigtop package in.
6+
7+
determine where your source is, and add it to bigtop.bom
8+
update the bigtop-packages/src/common/<your-package> folder to have your component,
9+
and the do-component-build for it (which usually just builds a jar).
10+
Why is there a "common" directory? Simply because deb and rpm packaging share some tasks (like do-component-build,
11+
which just usually runs a mvn or gradle command), and so we keep a common install directory which they can both leverage for packaging.
12+
(for RPM) now add a .spec file into bigtop-packages/src/<your-package>/... into the appropriate directory
13+
(i.e. bigtop-packages/src/rpm/tachyon/SPECS/tachyon.spec).
14+
Obviously, your tachyon.spec file will use whats in common/ in a RPM specific way, to install the RPM package.
15+
create a rules file using do-component-build
16+
17+
Test it with gradle <your-package>-rpm for the others.
18+
Finally add a smoke test! This is as easy as adding a new groovy file to bitop-tests/smoke-tests/<your-package>/TestThisStuff.groovy,
19+
following conventions that others have created.
20+
3) As always, we will improve on the directions above, but this should help to get you started. .

‎bigtop-packages/src/common/trino/install_trino.sh

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
#!/bin/bash
2+
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
set -e
19+
20+
usage() {
21+
echo "
22+
usage: $0 <options>
23+
Required not-so-options:
24+
--build-dir=DIR path to dist.dir
25+
--source-dir=DIR path to package shared files dir
26+
--prefix=PREFIX path to install into
27+
Optional options:
28+
--doc-dir=DIR path to install docs into [/usr/share/doc/trino]
29+
--lib-dir=DIR path to install trino home [/usr/lib/trino]
30+
--installed-lib-dir=DIR path where lib-dir will end up on target system
31+
--bin-dir=DIR path to install bins [/usr/bin]
32+
... [ see source for more similar options ]
33+
"
34+
exit 1
35+
}
36+
37+
OPTS=$(getopt \
38+
-n $0 \
39+
-o '' \
40+
-l 'prefix:' \
41+
-l 'lib-dir:' \
42+
-l 'installed-lib-dir:' \
43+
-l 'bin-dir:' \
44+
-l 'trino-version:' \
45+
-l 'source-dir:' \
46+
-l 'cli-dir:' \
47+
-l 'build-dir:' -- "$@")
48+
49+
if [ $? != 0 ] ; then
50+
usage
51+
fi
52+
53+
eval set -- "$OPTS"
54+
while true ; do
55+
case "$1" in
56+
--prefix)
57+
PREFIX=$2 ; shift 2
58+
;;
59+
--build-dir)
60+
BUILD_DIR=$2 ; shift 2
61+
;;
62+
--source-dir)
63+
SOURCE_DIR=$2 ; shift 2
64+
;;
65+
--cli-dir)
66+
CLI_BUILD_DIR=$2 ; shift 2
67+
;;
68+
--lib-dir)
69+
LIB_DIR=$2 ; shift 2
70+
;;
71+
--installed-lib-dir)
72+
INSTALLED_LIB_DIR=$2 ; shift 2
73+
;;
74+
--bin-dir)
75+
BIN_DIR=$2 ; shift 2
76+
;;
77+
--)
78+
shift ; break
79+
;;
80+
*)
81+
echo "Unknown option: $1"
82+
usage
83+
exit 1
84+
;;
85+
esac
86+
done
87+
88+
for var in PREFIX BUILD_DIR SOURCE_DIR ; do
89+
if [ -z "$(eval "echo \$$var")" ]; then
90+
echo Missing param: $var
91+
usage
92+
fi
93+
done
94+
95+
if [ -f "$SOURCE_DIR/bigtop.bom" ]; then
96+
. $SOURCE_DIR/bigtop.bom
97+
fi
98+
99+
MAN_DIR=${MAN_DIR:-/usr/share/man}/man1
100+
DOC_DIR=${DOC_DIR:-/usr/share/doc/trino}
101+
CLI_DIR=${CLI_DIR:-/usr/lib/trino-cli}
102+
LIB_DIR=${TRINO_DIR:-/usr/lib/trino}
103+
VAR_DIR=${VAR_DIR:-/var/lib/trino}
104+
LOG_DIR=${LOG_DIR:-/var/log/trino}
105+
RUN_DIR=${RUN_DIR:-/var/run/trino}
106+
INSTALLED_LIB_DIR=${INSTALLED_LIB_DIR:-/usr/lib/trino}
107+
BIN_DIR=${BIN_DIR:-/usr/bin}
108+
CONF_DIR=${CONF_DIR:-/etc/trino}
109+
CONF_DIST_DIR=${CONF_DIST_DIR:-/etc/trino.dist}
110+
DEFAULT_DIR=${DEFAULT_DIR:-/etc/default}
111+
112+
install -d -m 0755 $PREFIX/$CONF_DIST_DIR
113+
install -d -m 0755 $PREFIX/$LIB_DIR
114+
install -d -m 0755 $PREFIX/$CLI_DIR
115+
install -d -m 0755 $PREFIX/$DOC_DIR
116+
install -d -m 0755 $PREFIX/$VAR_DIR
117+
install -d -m 0755 $PREFIX/$LOG_DIR
118+
install -d -m 0755 $PREFIX/$RUN_DIR
119+
install -d -m 0755 $PREFIX/$DEFAULT_DIR
120+
121+
cp -ra ${BUILD_DIR}/* $PREFIX/$LIB_DIR/
122+
cp -ra ${CLI_BUILD_DIR}/* $PREFIX/$CLI_DIR/
123+
124+
chmod +x $PREFIX/$LIB_DIR/bin/launcher
125+
126+
install -d -m 0755 $PREFIX/$CONF_DIST_DIR
127+
install -d -m 0755 $PREFIX/$CONF_DIST_DIR/catalog
128+
129+
cat > $PREFIX/$CONF_DIST_DIR/node.properties <<EOF
130+
node.environment=production
131+
node.id=ffffffff-ffff-ffff-ffff-ffffffffffff
132+
node.data-dir=/var/trino/data
133+
EOF
134+
135+
cat > $PREFIX/$CONF_DIST_DIR/jvm.config <<EOF
136+
-server
137+
-Xmx16G
138+
-XX:G1HeapRegionSize=32M
139+
-XX:+ExplicitGCInvokesConcurrent
140+
-XX:+ExitOnOutOfMemoryError
141+
-XX:+HeapDumpOnOutOfMemoryError
142+
-XX:-OmitStackTraceInFastThrow
143+
-XX:ReservedCodeCacheSize=512M
144+
-XX:PerMethodRecompilationCutoff=10000
145+
-XX:PerBytecodeRecompilationCutoff=10000
146+
-Djdk.attach.allowAttachSelf=true
147+
-Djdk.nio.maxCachedBufferSize=2000000
148+
-XX:+UnlockDiagnosticVMOptions
149+
EOF
150+
151+
cat > $PREFIX/$CONF_DIST_DIR/config.properties <<EOF
152+
# A single machine for testing that will function as both a coordinator and worker
153+
coordinator=true
154+
node-scheduler.include-coordinator=true
155+
http-server.http.port=8080
156+
query.max-memory=4GB
157+
query.max-memory-per-node=1GB
158+
discovery-server.enabled=true
159+
discovery.uri=http://localhost:8080
160+
# Minimal configuration for the coordinator:
161+
#coordinator=true
162+
#node-scheduler.include-coordinator=false
163+
#http-server.http.port=8080
164+
#query.max-memory=4GB
165+
#query.max-memory-per-node=1GB
166+
#discovery-server.enabled=true
167+
#discovery.uri=http://example.net:8080
168+
# Minimal configuration for the workers:
169+
#coordinator=false
170+
#http-server.http.port=8080
171+
#query.max-memory=4GB
172+
#query.max-memory-per-node=1GB
173+
#discovery.uri=http://example.net:8080
174+
http-server.log.path=/var/log/presto/http-request.log
175+
EOF
176+
177+
cat > $PREFIX/$CONF_DIST_DIR/log.properties <<EOF
178+
io.trino=INFO
179+
EOF
180+
181+
cat > $PREFIX/$CONF_DIST_DIR/catalog/tpch.properties <<EOF
182+
connector.name=tpch
183+
EOF
184+
185+
cat > $PREFIX/$CONF_DIST_DIR/catalog/tpcds.properties <<EOF
186+
connector.name=tpcds
187+
EOF
188+
189+
cat > $PREFIX/$CONF_DIST_DIR/catalog/jmx.properties <<EOF
190+
connector.name=jmx
191+
EOF
192+
193+
ln -s ${CONF_DIR} $PREFIX/$LIB_DIR/etc

0 commit comments

Comments
 (0)