The correct way to run other algorithms?

[alex-r-bigelow]

Using my branch from #1, I can get fibonacci data back without any trouble, but in an effort to collect larger datasets for @sayefsakin, I'm a little lost on the best way to run something else.

Here's my attempt to adapt the LRA code from the phylanx example as follows:

from remote_run import remote_run, to_string, viz
from jetlag import Universal, RemoteJobWatcher
from knownsystems import *
import sys

uv = Universal()
uv.load(
    backend=backend_tapis,
    email="[email protected]",
    jetlag_id='rostam-sbrandt',
)

def lra(file_name, xlo1, xhi1, ylo1, yhi1, xlo2, xhi2, ylo2, yhi2, alpha,
        iterations, enable_output):
    from phylanx.ast import file_read_csv, constant, shape, transpose, exp, dot
    data = file_read_csv(file_name)
    x = data[xlo1:xhi1, ylo1:yhi1]
    y = data[xlo2:xhi2, ylo2]
    weights = constant(0.0, shape(x, 1))
    transx = transpose(x)
    pred = constant(0.0, shape(x, 0))
    error = constant(0.0, shape(x, 0))
    gradient = constant(0.0, shape(x, 1))
    step = 0
    while step < iterations:
        if enable_output:
            print("step: ", step, ", ", weights)
        pred = 1.0 / (1.0 + exp(-dot(x, weights)))
        error = pred - y
        gradient = dot(transx, error)
        weights = weights - (alpha * gradient)
        step += 1
    return weights

job = remote_run(uv, lra, ("/phylanx-data/CSV/breast_cancer.csv", 0, 569, 0, 30, 0, 569, 30, 31, 1e-5, 750, 0), nodes=16, ppn=16)
job.wait()
print("result:",job.get_result())

try:
    viz(job)
except:
    print("Exception during viz step:",sys.exc_info()[0])

I get these errors in job-###/job.err:

+ tar xzvf input.tgz
+ echo 'export AGAVE_JOB_NODE_COUNT=16'
+ echo 'export AGAVE_JOB_PROCESSORS_PER_NODE=1'
+ echo 'export nx=0'
+ echo 'export ny=0'
+ echo 'export nz=0'
+ cd ./run_dir
+ source ./runapp.sh
++ source ../.env
+++ export AGAVE_JOB_NODE_COUNT=16
+++ AGAVE_JOB_NODE_COUNT=16
+++ export AGAVE_JOB_PROCESSORS_PER_NODE=1
+++ AGAVE_JOB_PROCESSORS_PER_NODE=1
+++ export nx=0
+++ nx=0
+++ export ny=0
+++ ny=0
+++ export nz=0
+++ nz=0
+++ lscpu
+++ grep '^CPU(s):'
+++ cut -d: -f2
++ export 'CPUS=                32'
++ CPUS='                32'
++ export APEX_OTF2=1
++ APEX_OTF2=1
++ export 'APEX_PAPI_METRICS=PAPI_TOT_CYC PAPI_BR_MSP PAPI_TOT_INS PAPI_BR_INS PAPI_LD_INS PAPI_SR_INS PAPI_L1_D
++ APEX_PAPI_METRICS='PAPI_TOT_CYC PAPI_BR_MSP PAPI_TOT_INS PAPI_BR_INS PAPI_LD_INS PAPI_SR_INS PAPI_L1_DCM PAPI
++ LD_LIBRARY_PATH=/home/jovyan/install/phylanx/lib64/phylanx:/home/jovyan/install/phylanx/lib64:/usr/local/lib6
++ export PYTHONPATH=/home/jovyan/.local/lib/python3.6/site-packages/phylanx-0.0.1-py3.6-linux-x86_64.egg
++ PYTHONPATH=/home/jovyan/.local/lib/python3.6/site-packages/phylanx-0.0.1-py3.6-linux-x86_64.egg
++ singularity exec /home/sbrandt/images/phylanx-test.simg python3 command.py
No RAPL events found.
Can't open fd for cpu0: Permission denied.
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.6/site-packages/phylanx-0.0.1-py3.6-linux-x86_64.egg/phylanx/ast/physl.p
    physl_db = db(_name_of_importing_file)
  File "/home/jovyan/.local/lib/python3.6/site-packages/phylanx-0.0.1-py3.6-linux-x86_64.egg/phylanx/ast/physl_d
    self.dbname, self.db, created = self.ensure_db(name, version)
  File "/home/jovyan/.local/lib/python3.6/site-packages/phylanx-0.0.1-py3.6-linux-x86_64.egg/phylanx/ast/physl_d
    os.makedirs(dbdir)
  File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/__physlcache__'

I'm guessing this might have something to do with trying to sneakily use the data already on rostam under the /phylanx-data/CSV/breast_cancer.csv directory?

Is there a better way to try to run something heavier through JetLag?

[stevenrbrandt]

You're tripping over a phylanx bug which I can very rarely reproduce... I do know a fix though. Hopefully, I can get it through soon (I've been out sick).