Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds usage report #222

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

se7entyse7en
Copy link

I've created this PR for #219. This is not supposed to be merged but in this way, you can comment and also answer some questions in this report. After the review, I think that we can then write the document needed for #219.

Signed-off-by: Lou Marvin Caraig <[email protected]>
@se7entyse7en
Copy link
Author

se7entyse7en commented Apr 22, 2019

Most probably I missed things that are needed before considering to Gemini a product, but @smacker you will surely be able to integrate them given you knowledge of the project 👍. I approached this task as somebody who doesn't know anything about Gemini.

@smacker
Copy link
Contributor

smacker commented Apr 22, 2019

I see ./sbt test failed for you. But were you able to run gemini in development mode?

@se7entyse7en
Copy link
Author

@smacker what do you mean by development mode exactly?

I've just tried running standalone and some exceptions occured (this doens't happen when inside the container):

✔  se7entyse7en in ~/Projects/src-d/gemini (usage-report)  $ SPARK_HOME=~/Downloads/spark-2.2.3-bin-hadoop2.7 ./hash --replace src/test/resources/siva
Using spark-submit from /Users/se7entyse7en/Downloads/spark-2.2.3-bin-hadoop2.7
Running Hashing as Apache Spark job, master: local[*]
Hashing 5 repositories in: 'src/test/resources/siva' ()
	file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/duplicate-funcs/27f7db976994baf808b205c1ca95ba961cebf59d.siva
	file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/duplicate-files/f281ab6f2e0e38dcc3af05360667d8f530c00103.siva
	file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/duplicate-files/9279be3cf07fb3cca4fc964b27acea57e0af461b.siva
	file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/unique-files/5fb38a5744b2496ff8484e57b46a754433ede457.siva
	file:/Users/se7entyse7en/Projects/src-d/gemini/src/test/resources/siva/2636f3c62f1a407b2996da6e3fe6fdc5d1ccd764.siva
 WARN 18:13:32 tech.sourced.gemini.Gemini (Gemini.scala:37) - Getting repositories at src/test/resources/siva in siva format
 WARN 18:13:35 tech.sourced.gemini.Gemini (Gemini.scala:46) - Hashing
 WARN 18:13:35 tech.sourced.gemini.Hash (Hash.scala:125) - Listing files
 WARN 18:13:36 tech.sourced.gemini.Hash (Hash.scala:141) - Extracting UASTs
 WARN 18:13:37 tech.sourced.gemini.Hash (Hash.scala:162) - Extracting features
[Stage 2:================>                                       (58 + 4) / 200]ERROR 18:13:46 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================>                                       (59 + 4) / 200]ERROR 18:13:46 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================>                                       (60 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================>                                      (61 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================>                                      (62 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================>                                      (64 + 4) / 200]ERROR 18:13:47 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================>                                     (66 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================>                                    (68 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================>                                    (71 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
 WARN 18:13:48 org.apache.spark.internal.Logging$class (Logging.scala:66) - FATAL hashes/chaos_machine.py: transform failed: received 9 errors:
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)
	check: key "col_offset": variable "pos_start_exists" redeclared (true vs false)

ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================>                                   (74 + 4) / 200]ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:48 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================>                                  (77 + 4) / 200]ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================>                                  (78 + 4) / 200]ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================>                                 (79 + 4) / 200]ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:49 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=======================>                                (83 + 4) / 200]ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=======================>                                (84 + 4) / 200]ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:========================>                               (87 + 4) / 200]ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:50 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=========================>                              (92 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================>                             (94 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================>                             (95 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================>                             (96 + 4) / 200]ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:51 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================>                            (98 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================>                           (100 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================>                          (102 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================>                          (105 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=============================>                         (106 + 4) / 200]ERROR 18:13:52 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=============================>                         (107 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=============================>                         (108 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================>                        (111 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================>                       (113 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================>                       (116 + 4) / 200]ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:53 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================================>                      (118 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================================>                      (119 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================>                     (120 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================>                     (122 + 4) / 200]ERROR 18:13:54 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================>                     (123 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================>                    (124 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================>                    (125 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================>                    (126 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================================>                   (130 + 4) / 200]ERROR 18:13:55 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================================>                  (132 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================================>                  (134 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================================>                 (137 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================================>                (139 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================================>                (141 + 4) / 200]ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:56 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:========================================>              (147 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=========================================>             (150 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=========================================>             (151 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==========================================>            (153 + 4) / 200]ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:57 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================================>           (158 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===========================================>           (159 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================================>          (160 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================================>          (162 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:============================================>          (163 + 4) / 200]ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:58 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================================>        (168 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================================>        (169 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==============================================>        (170 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================================>       (172 + 4) / 200]ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================================>       (173 + 4) / 200] WARN 18:13:59 org.apache.spark.internal.Logging$class (Logging.scala:66) - FATAL sorts/external-sort.py: syntax error: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python2.7/ast.py", line 37, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 161\n    main()\n       ^\nIndentationError: expected an indented block\n']
------ Python3 errors:
['Traceback (most recent call last):\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n    current_ast = ast2dict(code)\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/local/lib/python3.6/ast.py", line 35, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 161\n    main()\n       ^\nIndentationError: expected an indented block\n']
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:13:59 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===============================================>       (174 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:================================================>      (176 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================================>     (179 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================================>     (180 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=================================================>     (181 + 4) / 200]ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:00 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:==================================================>    (185 + 4) / 200]ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:===================================================>   (188 + 4) / 200]ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:====================================================>  (190 + 4) / 200]ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:=====================================================> (194 + 4) / 200] WARN 18:14:01 org.apache.spark.internal.Logging$class (Logging.scala:66) - FATAL File_Transfer_Protocol/ftp_send_receive.py: syntax error: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python2.7/ast.py", line 37, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 1\n    """\n    ^\nIndentationError: unexpected indent\n']
------ Python3 errors:
['Traceback (most recent call last):\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n    current_ast = ast2dict(code)\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/opt/driver/bin/.local/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/local/lib/python3.6/ast.py", line 35, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 1\n    """\n    ^\nIndentationError: unexpected indent\n']
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:01 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[Stage 2:======================================================>(199 + 1) / 200]ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
ERROR 18:14:02 tech.sourced.featurext.FEClient$ (Client.scala:120) - feature extractor error: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
 WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:52) - Feature Extraction exceptions
 WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:55) - Processed: 0, skipped: 169
 WARN 18:14:02 tech.sourced.gemini.Hash$$anonfun$report$2 (Hash.scala:56) - 	StatusRuntimeException -> 169
 WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:183) - creating document frequencies
 WARN 18:14:05 tech.sourced.gemini.Hash (Hash.scala:240) - hashing features
 WARN 18:14:05 tech.sourced.gemini.Gemini (Gemini.scala:49) - Saving hashes to DB
 WARN 18:14:05 tech.sourced.gemini.Hash (Hash.scala:290) - save meta to DB
 WARN 18:14:06 tech.sourced.gemini.Hash (Hash.scala:270) - save document frequencies to DB
 WARN 18:14:06 tech.sourced.gemini.Hash (Hash.scala:309) - save hashtables to DB
Done

@smacker
Copy link
Contributor

smacker commented Apr 22, 2019

what do you mean by development mode exactly

something like

./sbt assemblyPackageDependency
./sbt assembly
./hash -k local src/test/resources/siva
./report -k local
./query -k local <options>

according to your output most probably you did not.

./report & ./query most probably returned you nothing because ./hash failed.

@se7entyse7en
Copy link
Author

I actually ran gemini locally, but with all the deps on containers:

d481f61dd2d5        bblfsh/bblfshd:v2.12.0-drivers   "bblfshd -ctl-addres…"   About a minute ago   Up About a minute   0.0.0.0:9432->9432/tcp                                                 srcd-cli-bblfshd
8fb09de3a4b8        scylladb/scylla:2.0.0            "/docker-entrypoint.…"   4 days ago           Up About a minute   7000-7001/tcp, 9160/tcp, 9180/tcp, 10000/tcp, 0.0.0.0:9042->9042/tcp   gemini_scylla_1
5146ff34d557        gemini_featurext                 "python ./server.py"     4 days ago           Up About a minute                                                                          gemini_featurext_1
1b91ec55d391        bblfsh/bblfshd:v2.11.0-drivers   "bblfshd"                4 days ago           Up About a minute                                                                          gemini_bblfshd_1

Do you perhaps mean running everything on the host?

@smacker
Copy link
Contributor

smacker commented Apr 23, 2019

I mean according to your logs:

 WARN 18:14:02 tech.sourced.gemini.Hash (Hash.scala:55) - Processed: 0, skipped: 169

gemini didn't process anything and didn't find any similarities

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants