Install python3, enabling python apps

[MrCreosote]

I have no name!@12bf5b3d903b:/opt/bitnami/spark$ ./bin/spark-submit
--master yarn
--conf spark.hadoop.yarn.resourcemanager.hostname=yarn-resourcemanager
--conf spark.hadoop.yarn.resourcemanager.address=yarn-resourcemanager:8032
--conf spark.hadoop.fs.s3a.endpoint=http://minio:9002
--conf spark.hadoop.fs.s3a.access.key=minio
--conf spark.hadoop.fs.s3a.secret.key=minio123
--conf spark.hadoop.fs.s3a.path.style.access=true
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.yarn.stagingDir=s3a://yarn
--deploy-mode client examples/src/main/python/pi.py 10 2>/dev/null
Pi is roughly 3.147040

[Tianhao-Gu]

Changes look fine. But I cannot verify the build locally.

(dev) tgu@cdm-prototype-yarn (dev-py_install)$docker-compose up --build
WARN[0000] /Users/tgu/Development/kb_apps/cdm-prototype-yarn/docker-compose.yaml: `version` is obsolete 
[+] Building 0.8s (12/26)                                                docker:rancher-desktop
 => [yarn-resourcemanager internal] load build definition from Dockerfile                  0.0s
 => => transferring dockerfile: 3.54kB                                                     0.0s
 => [yarn-nodemanager internal] load metadata for docker.io/apache/hadoop:3.3.6            0.1s
 => [yarn-nodemanager internal] load build definition from Dockerfile                      0.0s
 => => transferring dockerfile: 3.54kB                                                     0.0s
 => [yarn-resourcemanager internal] load .dockerignore                                     0.0s
 => => transferring context: 2B                                                            0.0s
 => [yarn-nodemanager internal] load .dockerignore                                         0.0s
 => => transferring context: 2B                                                            0.0s
 => [yarn-resourcemanager  1/10] FROM docker.io/apache/hadoop:3.3.6@sha256:62a22627f35a1b  0.0s
 => [yarn-resourcemanager internal] load build context                                     0.0s
 => => transferring context: 167B                                                          0.0s
 => [yarn-nodemanager internal] load build context                                         0.0s
 => => transferring context: 167B                                                          0.0s
 => CACHED [yarn-resourcemanager  2/10] COPY ./conf/yum/CentOS-Base.repo /etc/yum.repos.d  0.0s
 => CACHED [yarn-resourcemanager  3/10] RUN mkdir -p /opt/temp                             0.0s
 => CACHED [yarn-resourcemanager  4/10] WORKDIR /opt/temp                                  0.0s
 => ERROR [yarn-resourcemanager  5/10] RUN yum clean all     && yum makecache fast     &&  0.6s
------                                                                                          
 > [yarn-resourcemanager  5/10] RUN yum clean all     && yum makecache fast     && yum -y update     && yum -y install epel-release     && yum -y install wget make cmake gcc bzip2-devel libffi-devel zlib-devel perl-core pcre-devel     && yum -y groupinstall "Development Tools"     && wget https://openssl.org/source/openssl-3.3.1.tar.gz     && tar -xzvf openssl-3.3.1.tar.gz     && cd openssl-3.3.1     && ./config --prefix=/usr --openssldir=/etc/ssl --libdir=lib no-shared zlib-dynamic     && make     && make install     && cd ..     && wget https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tgz     && tar xvf Python-3.11.9.tgz     && cd Python-3.11.9     && LDFLAGS="${LDFLAGS} -Wl,-rpath=/usr/local/openssl/lib" ./configure --with-openssl=/usr/local/openssl     && make     && make altinstall     && cd ../..     && rm -R /opt/temp:
0.608 /bin/sh: line 1:     7 Segmentation fault      (core dumped) yum clean all
------
failed to solve: process "/bin/sh -c yum clean all     && yum makecache fast     && yum -y update     && yum -y install epel-release     && yum -y install wget make cmake gcc bzip2-devel libffi-devel zlib-devel perl-core pcre-devel     && yum -y groupinstall \"Development Tools\"     && wget https://openssl.org/source/openssl-3.3.1.tar.gz     && tar -xzvf openssl-3.3.1.tar.gz     && cd openssl-3.3.1     && ./config --prefix=/usr --openssldir=/etc/ssl --libdir=lib no-shared zlib-dynamic     && make     && make install     && cd ..     && wget https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tgz     && tar xvf Python-3.11.9.tgz     && cd Python-3.11.9     && LDFLAGS=\"${LDFLAGS} -Wl,-rpath=/usr/local/openssl/lib\" ./configure --with-openssl=/usr/local/openssl     && make     && make altinstall     && cd ../..     && rm -R /opt/temp" did not complete successfully: exit code: 139

[MrCreosote]

I'm guessing another mac issue, that's a bummer

[Tianhao-Gu]

I'm guessing another mac issue, that's a bummer

I am also worried about how we can test jupyterlab later with this setup. But it will come later.

[MrCreosote]

An ubuntu VM maybe?