cloud sdk 123 causes PipelineSetupError in dev server

[snstanton]

I have a mapreduce pipeline that worked in SDK 122 that now fails in the dev server with the following error after updating to SDK 123:

Traceback (most recent call last):
File "/vl/local/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(_args, *_kwargs)
File "/Users/stanton/src/vlshare/appserver/src/vl/base.py", line 645, in wrapper
return handler(_args, *kwargs)
File "/Users/stanton/src/vlshare/appserver/src/vlops/handlers.py", line 319, in any
result = func(self.request)
File "/Users/stanton/src/vlshare/appserver/src/vlops/admin/files.py", line 301, in getZone
result = launchPipeline(GetZonePipeline(zone.key.urlsafe()))
File "/Users/stanton/src/vlshare/appserver/src/vl/pipe_utils.py", line 62, in launchPipeline
pipeline.start(base_path='/ops/pipeline')
File "/Users/stanton/src/vlshare/appserver/src/lib/pipeline/pipeline.py", line 673, in start
self, idempotence_key, str(e)))
PipelineSetupError: Error starting vlops.admin.files.GetZonePipeline(('aghkZXZ-Tm9uZXIXCxIKWm9uZUVudGl0eRiAgICAgMCvCgyiAQZnbG9iYWw',), *{})#342c4ceab7084adfb389b35b45960fdd:
As far as I can tell the pipeline is having problems submitting tasks to the task queue. Everything works as expected in the production environment, so this is specifically a problem with the dev server.

Any ideas? What additional logging would be helpful here?

[nbortolotti]

I believe I have a similar error:

ERROR 2016-11-15 18:25:09,247 webapp2.py:1552] Error starting views.MyPipeline((), *{})#6e0e7928be0e4ab9b11d93cddead70f2:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in call
rv = self.handle_exception(request, response, e)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in call
rv = self.router.dispatch(request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in call
return handler.dispatch()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(_args, *_kwargs)
File "/Users/nickbortolotti/CodeDevelopers/appengine/xworkflow/views.py", line 106, in post
pipeline.start()
File "/Users/nickbortolotti/CodeDevelopers/appengine/xworkflow/lib/pipeline/pipeline.py", line 673, in start
self, idempotence_key, str(e)))

Ideas?

[przemekpastuszka]

I was able to fix that by modifying util.py:53 to:
if pipeline._TEST_MODE or os.environ['SERVER_SOFTWARE'].startswith('Development'):

[blech]

@rtshadow could you please make that a pull request?

[cdman]

Hmm, might this be the same problem as #71? I think setting _TEST_MODE masks the issue and the proper fix would be to use the modules API as suggested in #71, rather than undocumented internal variables.