Patch 1 #12
Open
Patch 1 #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…reter ### What is this PR for? Livy will create SQLContext/HiveContext internally, (LIVY-94), so it is not necessary to create that in LivyInterpreter. Otherwise sqlContext in zeppelin will override that in livy. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1473 ### How should this be tested? Tested manually. HiveContext is created properly in livy (with proper livy configuration), and can access hive data. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#1450 from zjffdu/ZEPPELIN-1473 and squashes the following commits: bae46a1 [Jeff Zhang] ZEPPELIN-1473. It is not necessary to create SQLContext in LivyInterpreter
### What is this PR for? A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://github.com/apache/zeppelin/blob/master/CONTRIBUTING.md This is a rewrite of the "Quick Start" page, fixing syntax/grammar issues while trying to keep to the original content. I may look at other pages, but I wanted my first PR to be very small. ### What type of PR is it? Documentation ### Todos Verify the accuracy of the configuration settings section ### What is the Jira issue? (Does this apply to documentation PRs?) ### How should this be tested? Do documentation changes like this need to be tested? If so, is the github preview sufficient for these changes? How is Zeppelin's markdown interpreter different than Github's? If Github's MD preview isn't sufficient, is it possible to build the documentation locally? ### Screenshots (if appropriate) ### Questions: * Since this is my first PR on the project, I limited the scope of my changes. Is there any desire to change the content of these pages in any way? Do we have documentation conventions? Author: Roger Filmyer <[email protected]> Closes apache#1416 from rfilmyer/master and squashes the following commits: f2c126b [Roger Filmyer] Rewrite of "Quick Start"
### What is this PR for? apache#1395 changed alignment of html tag and it caused to make space between interpreter group and name. ### What type of PR is it? Bug Fix ### Screenshots (if appropriate) **Before** <img width="436" alt="screen shot 2016-09-22 at 11 31 35 am" src="https://cloud.githubusercontent.com/assets/8503346/18743269/7fa34b16-80b8-11e6-979a-194e7f6bef8f.png"> **After** <img width="411" alt="screen shot 2016-09-22 at 11 32 02 am" src="https://cloud.githubusercontent.com/assets/8503346/18743271/83807038-80b8-11e6-80b1-775c3369940e.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes apache#1448 from minahlee/showInterpreterList and squashes the following commits: 5e2315a [Mina Lee] Remove space in available interpreters list
### What is this PR for? After apache#1416 merged, the bullet list in https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/install.html#installation isn't properly rendered now. It needs an extra newline. ### What type of PR is it? Hot Fix ### Screenshots (if appropriate) - Before <img width="834" alt="screen shot 2016-09-25 at 4 13 25 pm" src="https://cloud.githubusercontent.com/assets/10060731/18813621/5e10c2dc-833b-11e6-8410-bb1d37d8228b.png"> - After <img width="834" alt="screen shot 2016-09-25 at 4 13 36 pm" src="https://cloud.githubusercontent.com/assets/10060731/18813622/6705b744-833b-11e6-8fab-3a69d8585c7a.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <[email protected]> Closes apache#1457 from AhyoungRyu/fix/install.md and squashes the following commits: 73c1a4f [AhyoungRyu] Add a newline for bullet list markdown rendering
### What is this PR for?
`ZeppelinContext.show` doesn't work for spark 1.6. The root cause is that `Dataset` is also available in spark 1.6, so the following line will be false when cls is `Dataset` while o is `Dataframe` in spark 1.6
```
if (cls.isInstance(o)) {
```
This PR create a list of supported class and make it a member of `ZeppelinContext `so that we don't need to create it every time.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1427
### How should this be tested?
Tested it manually on spark 1.6 using the following sample code
```
z.show(sqlContext.sql("show tables"))
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <[email protected]>
Closes apache#1440 from zjffdu/ZEPPELIN-1427 and squashes the following commits:
62dbcad [Jeff Zhang] add unit test
a7ba67d [Jeff Zhang] ZEPPELIN-1427. Scala z.show() doesn't work on v.0.6.1
…n is created ### What is this PR for? The issue is that we create 2 SparkSession in zeppelin_pyspark.py (Because we create SQLContext first which will create SparkSession underlying). This cause 2 instances of SparkSession in JVM side and this means we have 2 instances of Catalog as well. So udf registered in SQLContext can not be used in SparkSession. This PR will create SparkSession first and then assign its internal SQLContext to sqlContext in pyspark. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1442 ### How should this be tested? Integration test is added. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No … Author: Jeff Zhang <[email protected]> Closes apache#1452 from zjffdu/ZEPPELIN-1442 and squashes the following commits: a15e3c6 [Jeff Zhang] fix unit test 93060b6 [Jeff Zhang] ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created
### What is this PR for?
If user specifies library path in interpreter dependency setting, even when the file on this path is updated, new file doesn't take effect on interpreter _restart_ but does only when user _clicks Edit -> Save._
The mechanism of dependency loading is copying all dependency libraries under `local-repo/
{interpreterId}` and add these directory to classpath of interpreter process. Zeppelin copies these dependencies either on Zeppelin startup or dependency saving/editing.
This PR checks if the library on specified local path is updated, and copy them to `local-repo/
{interpreterId}` on restart if there is change.
### What type of PR is it?
Bug Fix & Improvement
### What is the Jira issue?
[ZEPPELIN-1482](https://issues.apache.org/jira/browse/ZEPPELIN-1482)
### How should this be tested?
1. Download commons-csv-1.1.jar and commons-csv-1.2.jar to /my/path
2. cp commons-csv-1.2.jar /my/path/commons-csv.jar
3. Set dependency artifact of spark interpreter to /my/path/commons-csv.jar
4. Run `%spark import org.apache.commons.csv.CSVFormat.Predefined` in paragraph and see if it runs without error
5. cp commons-csv-1.1.jar /my/path/commons-csv.jar
6. Restart spark interpreter
7. Run `%spark import org.apache.commons.csv.CSVFormat.Predefined` in paragraph and see if error occurs.
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Mina Lee <[email protected]>
Closes apache#1453 from minahlee/ZEPPELIN-1482 and squashes the following commits:
ea11664 [Mina Lee] Check if dependency library on specified path has changed and copy them under local-repo/{interpreterId} on interpreter restart
## What is this PR for? The PR is a interpreter for [Apache Beam](http://beam.incubator.apache.org) which is an open source unified platform for data processing pipelines. A pipeline can be build using one of the Beam SDKs. The execution of the pipeline is done by different Runners . Currently, Beam supports Apache Flink Runner, Apache Spark Runner, and Google Dataflow Runner. ### What type of PR is it? - Feature ### Todos * Test case * Review Comments * Documentation ### What is the Jira issue? * [ZEPPELIN-682] ### How should this be tested? - Start the Zeppelin server - The prefix of interpreter is `%beam` and then write your code with required imports and the runner ### Screenshots (if appropriate)   ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: mahmoudelgamal <[email protected]> Author: mfelgamal <[email protected]> Author: Fouad <[email protected]> Closes apache#1334 from mfelgamal/beam-interpreter-static-repl-7 and squashes the following commits: da66c27 [mahmoudelgamal] Modify condition of checking static modifier 55c1322 [mahmoudelgamal] set spark version to 1.6.2 and throw original exception 27d7690 [mahmoudelgamal] set spark version to 1.6.1 and some modifications 750041c [mahmoudelgamal] Add readme file and modify pom file and travis.yml ca88f94 [mahmoudelgamal] edit pom file and .travis.yml 3d65427 [mahmoudelgamal] update .travis.yml file f19f98d [mahmoudelgamal] Make easy example with imports ands some modifications 74c14ca [mahmoudelgamal] Update the licenses acc7afb [mahmoudelgamal] Change beam to version 0.2.0 e821614 [mahmoudelgamal] Removing hadoop-core and print stack trace to failure 5cb7c7b [mahmoudelgamal] Add some changes to doc and pom file 75fc4f7 [mahmoudelgamal] add interpreter to navigation.html and remove extra spaces and lines 9b1b385 [mahmoudelgamal] put beam in alphabetical order 9c1e25d [mahmoudelgamal] Adding changes like logging and conventions and license 2aa6d65 [mahmoudelgamal] changing class name to StaticRepl and adding some modifications 7cf25fb [mahmoudelgamal] Adding some tests 3c5038f [mahmoudelgamal] Modifying the documentation 5695077 [mahmoudelgamal] Modifying pom file and Making documentation 26fc59b [mahmoudelgamal] Refactoring of the code 3a2bd85 [mahmoudelgamal] Adding the beam to zeppelin 7 ab7ee2d [mahmoudelgamal] beam interpreter 85957ff [mfelgamal] Merge pull request #10 from apache/master 852c3d3 [mfelgamal] Merge pull request #9 from apache/master a4bcc0d [mfelgamal] Merge pull request #8 from apache/master 858f1e1 [mfelgamal] Merge pull request #7 from apache/master 03a1e80 [mfelgamal] Merge pull request #4 from apache/master 2586651 [Fouad] Merge pull request #2 from apache/master
…l error to user ### What is this PR for? This is about showing information to the user when there are errors on running paragraphs eg. there could be permission related issue with notebook. ### What type of PR is it? Improvement ### Todos NA ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-808 ### How should this be tested? * Create a notebook and change the file system permission for the notebook folder to have system level write permission to a different user.For eg. if you are running your local zeppelin server with id [USERNAME], then change the file system permission for one of your notebooks created with the former username to different one eg. ROOT user who will only have the write permission * Try to run all the paragraphs or any individual paragraph for the notebook * The information as shown in the screenshot should be displayed and the dialog could be closed by the 'Close' button. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Kavin <[email protected]> Closes apache#1408 from kavinkumarks/zeppelin-808-handle-notebook-permission-error and squashes the following commits: 21fd49a [Kavin] Exit the run paragraph execution when there are errors with the notebook file system. 950ebda [Kavin] Handle FileSystemException on running paragraph and show the relevant error message to the user. 9f7cf67 [Kavin] Show information to the user when there are errors with related to permission on running paragraphs.
### What is this PR for? Phoenix has two different connection types: thick and thin. This PR is about describing the difference between the two and including properties for both in docs/interpreter/jdbc.md ### What type of PR is it? Documentation ### What is the Jira issue? [ZEPPELIN-1452](https://issues.apache.org/jira/browse/ZEPPELIN-1452) ### How should this be tested? No tests necessary ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Randy Gelhausen <[email protected]> Closes apache#1436 from randerzander/master and squashes the following commits: 3ed029e [Randy Gelhausen] Update jdbc.md 6e2dff0 [Randy Gelhausen] Update jdbc.md 2964ba8 [Randy Gelhausen] Update jdbc.md d2720c3 [Randy Gelhausen] tweaked jdbc readme 38b1692 [Randy Gelhausen] Updated JDBC docs to include Phoenix thin client
### What is this PR for? Markdown interpreter's class name have changed from `Markdown` to `MarkdownInterpreter` in apache#1384 and this will bring some compatibility issue in case user have `Markdown` class specified in `conf/interpreter.json` file. This PR rollbacks markdown class name from `MarkdownInterpreter` to `Markdown` to avoid side effect ### What type of PR is it? Hotfix ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes apache#1449 from minahlee/update/markdownClassName and squashes the following commits: 7bdad44 [Mina Lee] Change classname of MarkdownInterpreter -> Markdown
### What is this PR for? This PR is for refactoring code for JDBCInterpreter. There is no putting 'Connection' to 'propertyKeyUnusedConnectionListMap' anywhere in the original code. ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1405 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <[email protected]> Closes apache#1396 from astroshim/ZEPPELIN-1405 and squashes the following commits: b07e162 [astroshim] add checking connection is null f6998c2 [astroshim] Merge branch 'master' into ZEPPELIN-1405 1862ae6 [astroshim] Merge branch 'master' into ZEPPELIN-1405 efc2bfc [astroshim] rebase 21217a7 [astroshim] fix indentation. 4d4f85c [astroshim] refactoring code of close() 9f1e368 [astroshim] replace ConnectionPool 4dabbcc [astroshim] wip) changing to use dbcp 12dd7cb [astroshim] remove propertyKeyUnusedConnectionListMap map
### What is this PR for? This PR is for the documentation of running zeppelin with CDH docker environment. and This PR is the part of https://issues.apache.org/jira/browse/ZEPPELIN-1198. Tested CDH5.7 on ubuntu. ### What type of PR is it? Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1281 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <[email protected]> Author: AhyoungRyu <[email protected]> Author: HyungSung <[email protected]> Closes apache#1451 from astroshim/ZEPPELIN-1281 and squashes the following commits: 5dcb8c1 [astroshim] move configurations to right path and add excluding rat-plugin 09408e3 [HyungSung] Merge pull request #11 from AhyoungRyu/ZEPPELIN-1281-ahyoung 850119c [AhyoungRyu] Generate TOC & change some sentences e687a53 [AhyoungRyu] Replace zeppelin_with_cdh.png to crop the url part cc9a023 [AhyoungRyu] Remove main title link anchor b525f68 [astroshim] separate cdh doc with spark_cluster_mode.md e66993f [astroshim] fix doc a7b5b2d [astroshim] cdh docker environment
Moon recommended I migrate a How To tutorial I wrote into the website. http://mail-archives.apache.org/mod_mbox/incubator-zeppelin-users/201511.mbox/browser I didn't create a JIRA issue, it doesn't look like that is nessicary for website updates? If it is, I'm sorry I'll go make one. Sorry, I'm new. Author: rawkintrevo <[email protected]> Closes apache#418 from rawkintrevo/zeppelin-flink-spark-tutorial and squashes the following commits: 0cd86b3 [rawkintrevo] [DOCS] Flink and Spark Cluster Tutorial
### What is this PR for? Fixing the image path of the tutorial made in apache#418 ### What type of PR is it? Hot Fix ### What is the Jira issue? None, just a Hotfix ### How should this be tested? Build, move into full website, you should see the images ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Damien CORNEAU <[email protected]> Closes apache#1474 from corneadoug/fix/FlinkTutorialImg and squashes the following commits: cabd8df [Damien CORNEAU] Change path to tutorial images
… count ### What is this PR for? While cloning a notebook create a new name using the current notebook name by prefixing "Copy of" and end with count e.g: name: test pre-filled clone name: Copy of test 1 ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1440 ### How should this be tested? Create a new book and clone on ui or see unit test in notename.js ### Screenshots (if appropriate) Before <img width="1221" alt="before" src="https://cloud.githubusercontent.com/assets/2031306/18517800/302309bc-7abb-11e6-8808-b98b9b75a0f4.png"> After <img width="1247" alt="after" src="https://cloud.githubusercontent.com/assets/2031306/18517805/349d27f2-7abb-11e6-8702-0c064db52697.png"> ### Questions: * Does the licenses files need update? na * Is there breaking changes for older versions? na * Does this needs documentation? na Author: Renjith Kamath <[email protected]> Closes apache#1429 from r-kamath/ZEPPELIN-1440 and squashes the following commits: b8b4f24 [Renjith Kamath] ZEPPELIN-1440 remove redundant beforeEach from test 282e912 [Renjith Kamath] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-1440 9770a20 [Renjith Kamath] ZEPPELIN-1440 remove prefix. fix folder bug b1f5b5c [Renjith Kamath] ZEPPELIN-1440 Notebook clone: prefix name with "Copy of" and endwith count
### What is this PR for? This handles replacing the registration of interpreter with static block by the interpreter-setting.json file ### What type of PR is it? Sub-ask ### Todos NA ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-908 ### How should this be tested? * There shouldn't be any warning like below on starting the server `INFO [2016-09-29 00:25:46,247] ({main} CassandraInterpreter.java[<clinit>]:155) - Bootstrapping Cassandra Interpreter WARN [2016-09-29 00:25:46,250] ({main} Interpreter.java[register]:347) - Static initialization is deprecated for interpreter cassandra, You should change it to use interpreter-setting.json in your jar or interpreter/{interpreter}/interpreter-setting.json INFO [2016-09-29 00:25:46,250] ({main} InterpreterFactory.java[init]:204) - Interpreter cassandra.cassandra found. class=org.apache.zeppelin.cassandra.CassandraInterpreter` * And ensure that the cassandra related paragraphs run without any error ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Kavin <[email protected]> Closes apache#1473 from kavinkumarks/zeppelin-908-register-cassandra-interpreter and squashes the following commits: d1fe586 [Kavin] Removed static initialization of interpreter registration with the interpreter-setting.json file.
### What is this PR for? A bug fix: Added validation in the note import dialog box to check for the uploaded json file size and throw error report if the file size exceeds 1MB, as the websocket frame is not able to send json file of size over 1MB. ### What type of PR is it? Bug Fix ### Todos NA ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-868 ### How should this be tested? 1. Deploy Zeppelin and click on 'Import Note' in the Welcome to Zeppelin page. 2. Click 'Choose a JSON here' and upload a json file whose file size is over 1MB ### Screenshots (if appropriate)     ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: vensant <[email protected]> Closes apache#1430 from vensant/ZEPPELIN-868 and squashes the following commits: 29ad4ce [vensant] committed the code which missed during rebase bd1acc0 [vensant] fixed the review comments on the pull request c65f2c7 [vensant] client level validation added for import file size check 5dcdcfe [vensant] made changes for reading the max limit from the configuration list and dynamically showing it in UI e48aac6 [vensant] Rectified the errors due to grunt build no color failure 77a0cc1 [vensant] Added validation in the note import dialog box to check for the uploaded json file size as the websocket frame is not able to send file of size over 1MB
### What is this PR for? When the data is downloaded as CSV/TSV, the comma/tab in the actual data has to be handled so that they come exactly as the same data when downloaded. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1001 ### How should this be tested? Modify the data to be loaded to have a comma. Create a paragraph to pull up that data and display. Now click on the download as CSV/TSV button in the tool bar. Once the data is downloaded verify whether the original data is unaltered and the comma is escaped. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: meenakshisekar <[email protected]> Closes apache#1445 from meenakshisekar/ZEPPELIN-1001 and squashes the following commits: a4a2009 [meenakshisekar] Formatting errors corrected for build 5e14da8 [meenakshisekar] review comments fixed ce27b82 [meenakshisekar] Changes committed as per review comments in PR 2f7d78a [meenakshisekar] Altered the code as per PR suggestion 1465 657dcbe [meenakshisekar] Zeppelin-1001 Modified the data with comma/tab to be surronded by double quotes so that they are escaped. 73b1868 [meenakshisekar] Zeppelin-1001 Modified the data with comma/tab to be surronded by double quotes so that they are escaped.
### What is this PR for? I excluded some docs pages from [search page](https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/search.html) in Zeppelin documentation site. This can be done by removing `title` value in [front matter](https://jekyllrb.com/docs/frontmatter/) as I did same for `pleasecontribute.md` in [here](apache@6e775f5). ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-1514](https://issues.apache.org/jira/browse/ZEPPELIN-1514) ### How should this be tested? 1) build gh-pages (website) branch ``` JEKYLL_ENV=production bundle exec jekyll build mkdir -p tmp/zeppelin_website/docs/ cp -r _site/ /tmp/zeppelin_website/ ``` 2) build this patch(docs) and copy it under docs/0.7.0-SNAPSHOT of website ``` cd ZEPPELIN_HOME/docs/ bundle exec jekyll build --safe cp -r _site/ /tmp/zeppelin_website/docs/0.7.0-SNAPSHOT/ ``` 3) start httpServer ``` cd /tmp/zeppelin_website python -m SimpleHTTPServer ``` 4) browse `http://localhost:8000` ### Screenshots (if appropriate) The below pages shouldn't be searched <img width="861" alt="screen shot 2016-10-01 at 4 01 37 pm" src="https://cloud.githubusercontent.com/assets/10060731/19012559/80339cbe-87f4-11e6-8ee5-59405c048171.png"> <img width="539" alt="screen shot 2016-10-01 at 4 08 03 pm" src="https://cloud.githubusercontent.com/assets/10060731/19012560/82f32f3c-87f4-11e6-9c25-0f22eb723245.png"> <img width="558" alt="screen shot 2016-10-01 at 4 08 11 pm" src="https://cloud.githubusercontent.com/assets/10060731/19012561/8780398c-87f4-11e6-8446-39f0f8f8c994.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: AhyoungRyu <[email protected]> Closes apache#1478 from AhyoungRyu/fix/excludeNonDocs and squashes the following commits: da1eb84 [AhyoungRyu] Make atom, rss, sitemap file not to be searched
### What is this PR for? When there are a variety of para graph interpreter present, When you refresh the page, the code highlights are incorrectly applied. ### What type of PR is it? Bug Fix ### Todos - [x] - Create orderId key in getEditorMode function. ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1502 ### How should this be tested? 1. Create a para-graph form below. ``` %spark println("spark syn 01"); ``` ``` %spark println("spark syn 02"); ``` ``` %spark println("spark syn 03"); ``` ``` %spark println("spark syn 04"); ``` ``` %spark println("spark syn 05"); ``` ``` %pyspark print ("pyspark syn); ``` ``` %sql SELECT * FROM SQL_HIGH WHERE ONMYCOM ``` 2. Check the highlights of each of the para graphs. 3. Refresh the page and check the highlights again. ### Screenshots (if appropriate) #### bug (focus on pyspark)  #### this pr (focus on pyspark)  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <[email protected]> Closes apache#1468 from cloverhearts/dev/codeeditor-bugfixed and squashes the following commits: ff43ce3 [CloverHearts] change payload key name for interpreter syntax property de03a06 [CloverHearts] remove whitespace - jslint eaa1b86 [CloverHearts] bug fixed for page refresh initialize editor mode
### What is this PR for? This is some refactoring of zeppelin-web to apply the IIFE as stated in the Good Practice Guide #1 (https://zeppelin.apache.org/contribution/zeppelinweb/goodPracticeGuide01.html) ### What type of PR is it? Refactoring ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1496 ### How should this be tested? Run The webapp, and go around the page, things should work like usual. But overall, just need to check that there is no type or parameter mismatch during the refactoring process (controller to controller, params in $inject etc...) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Damien CORNEAU <[email protected]> Closes apache#1469 from corneadoug/ZEPPELIN-1496 and squashes the following commits: 966cb27 [Damien CORNEAU] refactor components controllers 1807cf6 [Damien CORNEAU] Refactor components directives 0055191 [Damien CORNEAU] Refactor components service and factory f86adb4 [Damien CORNEAU] Finish refactoring of App 409c65c [Damien CORNEAU] IIFE refactoring of job page 4c45e08 [Damien CORNEAU] Refactor for IIFE credential, home and interpreter controller 7e88eef [Damien CORNEAU] Add congiguration.controller.js 6adb522 [Damien CORNEAU] Move use strict outside IFFE ffba0f3 [Damien CORNEAU] Add IIFE to app.controller.js
…te firewall use-cases ### What is this PR for? When using Zeppelin behind corporate firewall, sometimes the dependencies download just fails silently. This PR has 2 objectives: * add proxy credentials information for dependencies repo * raise clear error message in case of dependencies download failure There are 3 commits. The first one add extra inputs in the form for adding new repository  The second commit fixes some issues and display a clear and explicit error message when download of dependencies fail. Before that, when the download fails, we can see the below behaviour  * the error message is displayed twice because the call twice the method `checkDownloadingDependencies();`. One in the success callback of: ```javascript $scope.updateInterpreterSetting = function(form, settingId) { ... $http.put(baseUrlSrv.getRestApiBase() + '/interpreter/setting/' + settingId, request) .success(function(data, status, headers, config) { $scope.interpreterSettings[index] = data.body; removeTMPSettings(index); thisConfirm.close(); checkDownloadingDependencies(); $route.reload(); }) .error(function(data, status, headers, config) { ... }; ``` Another call is inside success callback of `getInterpreterSettings()` ```javascript var getInterpreterSettings = function() { $http.get(baseUrlSrv.getRestApiBase() + '/interpreter/setting') .success(function(data, status, headers, config) { $scope.interpreterSettings = data.body; checkDownloadingDependencies(); }).error(function(data, status, headers, config) { .... ``` The problem is that `$route.reload();` in the success callback of `updateInterpreterSetting()` will trigger `init()` then `getInterpreterSettings()` so `checkDownloadingDependencies()` is called twice. I remove the call to `checkDownloadingDependencies()` from success callback of `updateInterpreterSetting()` The second modification is on class `DependencyResolver`. In the screen capture above, we get a **cryptic** NullPointerException coming from `DefaultRepositorySystem`. I now catch this NPE to wrap it into a more sensible and clearer exception: ```java public List<ArtifactResult> getArtifactsWithDep(String dependency, Collection<String> excludes) throws RepositoryException { Artifact artifact = new DefaultArtifact(dependency); DependencyFilter classpathFilter = DependencyFilterUtils.classpathFilter(JavaScopes.COMPILE); PatternExclusionsDependencyFilter exclusionFilter = new PatternExclusionsDependencyFilter(excludes); CollectRequest collectRequest = new CollectRequest(); collectRequest.setRoot(new Dependency(artifact, JavaScopes.COMPILE)); synchronized (repos) { for (RemoteRepository repo : repos) { collectRequest.addRepository(repo); } } DependencyRequest dependencyRequest = new DependencyRequest(collectRequest, DependencyFilterUtils.andFilter(exclusionFilter, classpathFilter)); //Catch NPE thrown by aether and give a proper error message try { return system.resolveDependencies(session, dependencyRequest).getArtifactResults(); } catch (NullPointerException ex) { throw new RepositoryException(String.format("Cannot fetch dependencies for %s", dependency)); } } ``` The result is much more cleaner  The last commit is just doc update  ### What type of PR is it? [Improvement] ### Todos * [ ] - Code Review * [ ] - Simple test with no Internet connection * [ ] - Test within a corporate firewall env with a third-party dependency, requiring download ### What is the Jira issue? **[ZEPPELIN-1376]** ### How should this be tested? ##### Simple test * `git fetch origin pull/1369/head:WebProxy` * `git checkout WebProxy` * `mvn clean package -DskipTests` * `bin/zeppelin-daemon.sh restart` * disconnect from the Internet (pull out the cable, shutdown wifi ...) * add a random dependency to the Spark interpreter (take `info.archinnov:achilles-core:4.2.2` for example) * validate the change, you should see an error popup on the top-right corner saying that Zeppelin cannot download the dependency ##### Corporate firewall test * follow the steps above for simple test * create a new repository (see how to **[here]**) and set the proxy information * retry the steps above to ensure that the download is successful ### Screenshots (if appropriate) See above ### Questions: * Does the licenses files need update? --> **NO** * Is there breaking changes for older versions? --> **NO** * Does this needs documentation? --> **YES, DONE** [ZEPPELIN-1376]: https://issues.apache.org/jira/browse/ZEPPELIN-1376 [here]: http://localhost:4000/manual/dependencymanagement.html Author: DuyHai DOAN <[email protected]> Author: doanduyhai <[email protected]> Closes apache#1369 from doanduyhai/ZEPPELIN-1376 and squashes the following commits: b8d44e7 [doanduyhai] [ZEPPELIN-1376] Improve error popup display 177fbd3 [DuyHai DOAN] [ZEPPELIN-1376] Fixes JS bug to display error popup for other interpreters 9f76ef4 [DuyHai DOAN] [ZEPPELIN-1376] Do not repeat the same error popup multiple times b264193 [DuyHai DOAN] [ZEPPELIN-1376] Add unit test and fix impl for DependencyResolver to catch NPE 1913a0a [DuyHai DOAN] [ZEPPELIN-1376] Update documentation f01be9b [DuyHai DOAN] [ZEPPELIN-1376] Raise clear error message in case of dependencies download failure 6f2b6f8 [DuyHai DOAN] [ZEPPELIN-1376] Add proxy credentials information for dependencies repo
…mprove JDBC docs ### What is this PR for? Currently we can use `%jdbc(prefix)` for multi database connections(e.g. configuring both postgresql and hive in one JDBC interpreter). But after apache#1360 merged, Zeppelin doesn't support `%jdbc(prefix)` to `%prefix` anymore. So auto-prefix feature is not working for `%jdbc(prefix)`. The auto-prefix only works for `%prefix` now. Considering this status, it would be better we remove the JDBC connection examples in `jdbc/interpreter-setting.json` (this is come from apache#1096) so that users can create several JDBC interpreters instead of using multiple connections with one JDBC interpreter. I removed the related contents in `jdbc.md` as well. Also, the contents of [current JDBC documentation page](http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/jdbc.html) is quite confusing and unorganized. So I updated the contents with some screenshot images to guide "How to create JDBC interpreter", "How to edit the interpreter properties for the connection", "How to use `%prefix` with the interpreter", and so on. ### What type of PR is it? Improvement & Documentation ### What is the Jira issue? [ZEPPELIN-1489](https://issues.apache.org/jira/browse/ZEPPELIN-1489) ### How should this be tested? * Removing example properties in `interpreter-setting.json` 1. after applying this patch and build with `mvn clean package -DskipTests -Pspark-1.6 -pl 'jdbc,zeppelin-interpreter,zeppelin-web,zeppelin-server,zeppelin-zengine,zeppelin-display'` 2. create JDBC interpreter and check whether the example settings are gone or not * JDBC docs 1. Build only `docs/` dir as described in [here](https://github.com/apache/zeppelin/blob/master/docs/README.md#build-documentation) 2. Go to `interpreter -> JDBC` and just compare this locally builded page with [the original JDBC page](https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/jdbc.html) ### Screenshots (if appropriate) - Before <img width="1579" alt="before" src="https://cloud.githubusercontent.com/assets/10060731/19045323/1ff0c706-89d3-11e6-9b6f-dc75877f81f3.png"> - After <img width="944" alt="screen shot 2016-10-04 at 1 05 00 am" src="https://cloud.githubusercontent.com/assets/10060731/19045324/24a9187a-89d3-11e6-90d6-b80acbc6af7c.png"> - Some parts of updated JDBC docs Since many contents are changed, it would be better to build `docs/` locally to review all of the change. <img width="695" alt="screen shot 2016-10-04 at 12 36 39 am" src="https://cloud.githubusercontent.com/assets/10060731/19043794/9d9a32fc-89cc-11e6-9d15-f6036a1b738e.png"> <img width="704" alt="screen shot 2016-10-04 at 12 36 59 am" src="https://cloud.githubusercontent.com/assets/10060731/19043800/a62fc90e-89cc-11e6-976d-5c697729eca4.png"> <img width="677" alt="screen shot 2016-10-04 at 12 37 31 am" src="https://cloud.githubusercontent.com/assets/10060731/19043807/acbc9766-89cc-11e6-8c73-eab1cc18440b.png"> <img width="688" alt="screen shot 2016-10-04 at 12 37 43 am" src="https://cloud.githubusercontent.com/assets/10060731/19043816/b06b5690-89cc-11e6-9298-a20b49fea622.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: AhyoungRyu <[email protected]> Closes apache#1480 from AhyoungRyu/ZEPPELIN-1489 and squashes the following commits: 76bf55e [AhyoungRyu] Minor update d5aaa97 [AhyoungRyu] Remove useless screenshot images c6f9ed4 [AhyoungRyu] Add screenshot images 99a18e2 [AhyoungRyu] Remove jdbc setting examples in interpreter-setting.json 050ecc0 [AhyoungRyu] Update jdbc.md 20da102 [AhyoungRyu] Redshit -> Redshift in JDBCInterpreter.java
### What is this PR for? Last profile of CI test fails after apache#1461. This PR fixes it. ``` Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 110.883 sec <<< FAILURE! - in org.apache.zeppelin.integration.ParagraphActionsIT testWidth(org.apache.zeppelin.integration.ParagraphActionsIT) Time elapsed: 20.055 sec <<< FAILURE! java.lang.AssertionError: New Width is : 4 Expected: <true> but: was <false> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:865) at org.junit.rules.ErrorCollector$1.call(ErrorCollector.java:65) at org.junit.rules.ErrorCollector.checkSucceeds(ErrorCollector.java:78) at org.junit.rules.ErrorCollector.checkThat(ErrorCollector.java:63) at org.apache.zeppelin.integration.ParagraphActionsIT.testWidth(ParagraphActionsIT.java:314) testWidth(org.apache.zeppelin.integration.ParagraphActionsIT) Time elapsed: 20.056 sec <<< FAILURE! java.lang.AssertionError: New Width is : 8 Expected: <true> but: was <false> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:865) at org.junit.rules.ErrorCollector$1.call(ErrorCollector.java:65) at org.junit.rules.ErrorCollector.checkSucceeds(ErrorCollector.java:78) at org.junit.rules.ErrorCollector.checkThat(ErrorCollector.java:63) at org.apache.zeppelin.integration.ParagraphActionsIT.testWidth(ParagraphActionsIT.java:314) ``` ### What type of PR is it? Test fix ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes apache#1504 from minahlee/fix/testWidth and squashes the following commits: 410bb55 [Mina Lee] Fix testWidth selnium test
…he Spark ### What is this PR for? Simplify travis test to reduce usage of resources ### What type of PR is it? [Improvement] ### Todos * [x] - Remove start-up/stop SparkCluster ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1520 ### How should this be tested? Travis will pass without any error ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Jongyoul Lee <[email protected]> Closes apache#1487 from jongyoul/ZEPPELIN-1520 and squashes the following commits: 3bccf66 [Jongyoul Lee] Removed some unused scripts anymore 15a3711 [Jongyoul Lee] Cleaned up commented lines 1237658 [Jongyoul Lee] Removed checking mechanism f37dacf [Jongyoul Lee] Changed master to local[2] 2aac444 [Jongyoul Lee] Remove scripts of start/stop SparkCluster
### What is this PR for? This PR fixes path of spark testcase. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1509 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <[email protected]> Closes apache#1475 from astroshim/ZEPPELIN-1509 and squashes the following commits: a1576ea [astroshim] Merge branch 'master' into ZEPPELIN-1509 dead681 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into ZEPPELIN-1509 bc74738 [astroshim] fix testcase path
### What is this PR for? This PR sets default value for ZEPPELIN_INTP_MEM to avoid OOM Exception in SparkInterpreter when Zeppelin has zero configuration. This PR should be merged to both branch-0.6 and master. ### What type of PR is it? Bug Fix ### How should this be tested? 1. Build with: ``` mvn clean package -DskipTests -pl '!zeppelin-distribution,!file,!alluxio,!livy,!hbase,!bigquery,!python,!jdbc,!ignite,!lens,!postgresql,!cassandra,!kylin,!elasticsearch,!flink,!markdown,!shell,!angular' ``` 2. Unset SPARK_HOME in conf/zeppelin-env.sh if you have. 3. Run Zeppelin with java 1.7. 4. Run tutorial and see if it doesn't hang. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes apache#1505 from minahlee/hotfix/default_intp_jvm and squashes the following commits: 0dfda4f [Mina Lee] Set default ZEPPELIN_INTP_MEM
…n repository ### What is this PR for? While removing duplication of specifying build profile in apache#1321, build profile has been changed from `-Ppublish-distr` to `-Pbuild-distr`. We need to restore this change and use `-Ppublish-distr` profile to publish `*.javadoc.jar ` and `*.sources.jar`. ### What type of PR is it? Hot Fix ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Mina Lee <[email protected]> Closes apache#1506 from minahlee/build_profile and squashes the following commits: b84b371 [Mina Lee] Change build profile for distributing artifact to maven repository
### What is this PR for? For now, it is hard to figure out what the yarn application of the livy session represent, it would be better to display the appId and webui link in the output of LivyInterpreter for diagnosing purpose. It can also be applied to the native SparkInterpreter, but could be done in another ticket. ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1430 ### How should this be tested? Tested manually ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#1426 from zjffdu/ZEPPELIN-1430 and squashes the following commits: 88009cb [Jeff Zhang] update doc 459f75e [Jeff Zhang] fix typo eb7ec27 [Jeff Zhang] add flag to enable display app info in frontend a087a1d [Jeff Zhang] ZEPPELIN-1430. Display appId and webui link in LivyInterpreter's output
### What is this PR for? Based on apache#338 , I refactor most of pig interpreter. As I don't think the approach in apache#338 is the best approach. In apache#338, we use script `bin/pig` to launch pig script, it is different to control that job (hard to kill and get progress and stats info). In this PR, I use pig api to launch pig script. Besides that I implement another interpreter type `%pig.query` to leverage the display system of zeppelin. For the details you can check `pig.md` ### What type of PR is it? [Feature] ### Todos * Syntax Highlight * new interpreter type `%pig.udf`, so that user can write pig udf in zeppelin directly and don't need to build udf jar manually. ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-335 ### How should this be tested? Unit test is added and also manual test is done ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Author: Ali Bajwa <[email protected]> Author: AhyoungRyu <[email protected]> Author: Jeff Zhang <[email protected]> Closes apache#1476 from zjffdu/ZEPPELIN-335 and squashes the following commits: 73a07f0 [Jeff Zhang] minor update a1b742b [Jeff Zhang] minor update on doc e858301 [Jeff Zhang] address comments c85a090 [Jeff Zhang] add license 58b4b2f [Jeff Zhang] minor update of docs 1ae7db2 [Jeff Zhang] Merge pull request #2 from AhyoungRyu/ZEPPELIN-335/docs fe014a7 [AhyoungRyu] Fix docs title in front matter df7a6db [AhyoungRyu] Add pig.md to dropdown menu 5e2e222 [AhyoungRyu] Minor update for pig.md 39f161a [Jeff Zhang] address comments 05a3b9b [Jeff Zhang] add pig.md a09a7f7 [Jeff Zhang] refactor pig Interpreter c28beb5 [Ali Bajwa] Updated based on comments: 1. Documentation: added pig.md with interpreter documentation and added pig entry to index.md 2. Added test junit test based on passwd file parsing example here https://pig.apache.org/docs/r0.10.0/start.html#run 3. Removed author tag from comment (this was copied from shell interpreter https://github.com/apache/incubator-zeppelin/blob/master/shell/src/main/java/org/apache/zeppelin/shell/ShellInterpreter.java#L42) 4. Implemented cancel functionality 5. Display output stream in case of error 2586336 [Ali Bajwa] exposed timeout and pig executable via interpreter and added comments 7abad20 [Ali Bajwa] initial commit of pig interpreter
### What is this PR for? Fix input form placeholder for `note url` ### What type of PR is it? Improvement ### Todos * [x] - Task ### What is the Jira issue? [ZEPPELIN-1540](https://issues.apache.org/jira/browse/ZEPPELIN-1540) ### How should this be tested? go to import note menu -> add from url -> see input forms ### Screenshots (if appropriate) Before: <img width="593" alt="screen shot 2016-10-13 at 2 37 24 pm" src="https://cloud.githubusercontent.com/assets/1642088/19337731/9ea41248-9152-11e6-859b-757f44b2866a.png"> After: <img width="601" alt="screen shot 2016-10-13 at 2 36 06 pm" src="https://cloud.githubusercontent.com/assets/1642088/19337716/7f90522c-9152-11e6-8ad2-cbfcdc241e10.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Khalid Huseynov <[email protected]> Closes apache#1509 from khalidhuseynov/fix/note-import-placeholder and squashes the following commits: c01b982 [Khalid Huseynov] note name -> note url
### What is this PR for? fixed 'add from URL' button and disable backdrop function. ### What type of PR is it? Bug Fix ### Todos None ### What is the Jira issue? [ZEPPELIN-1550](https://issues.apache.org/jira/browse/ZEPPELIN-1550) ### How should this be tested? click 'Import note' -> click 'Add from URL' -> click 'x' button -> open the dialog again -> it shows the URL page not the initial page. ### Screenshots (if appropriate) - before  - after  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Yunho Maeng <[email protected]> Closes apache#1523 from yunho0130/ZEPPELIN-1550 and squashes the following commits: a045928 [Yunho Maeng] fixed 'add from URL' button and disable backdrop fuction when user use modal window
…shortens. ### What is this PR for? In document page(http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/), the search button goes to next line when display's width shortens. ### What type of PR is it? [Improvement] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1552 ### Screenshots (if appropriate) <img width="927" alt="2016-10-15 2 04 00" src="https://cloud.githubusercontent.com/assets/6567102/19398321/0d76a1c4-9287-11e6-86e0-9f120c00b143.png"> when (768px ≤ width < 992px) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyonzin <[email protected]> Author: AhyoungRyu <[email protected]> Author: 정현진 <[email protected]> Closes apache#1525 from hyonzin/ZEPPELIN-1552 and squashes the following commits: f1d97ef [AhyoungRyu] Fix search icon's placement in small browser 244992d [정현진] Straighten wrong term 97d1bdf [hyonzin] Fix search link button to not goes next line when display's width shortens
### What is this PR for? Applying for new registration mechanism for Apache Kylin ### What type of PR is it? [Feature] ### Todos * [x] - Added interpreter-setting.json ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-916 ### How should this be tested? ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jongyoul Lee <[email protected]> Closes apache#1515 from jongyoul/ZEPPELIN-916 and squashes the following commits: 5e23043 [Jongyoul Lee] Added interpreter-setting.json Fixed test environments
…erpreters ### What is this PR for? See apache#1439. This is a second attempt based on prior feedback, particularly from Leemoonsoo who advised that it would be better to contain all of the code inside of the Interpreter Process. Aside from this, the major changes since the previous PR are: - The terminology "hooks" is used instead of "callbacks". Since the information being passed to the interpreter isn't actually a "callback", strictly speaking, the more general term "hook" is probably a better choice. All variable / class names were updated correspondingly. - The registry is now keyed by class name rather than the `replName` specified by the magic on top of the paragraph. The interface provided by `ZeppelinContext` allows for a `replName` to be converted to `className` through an internal mapping. - Two new event codes, `PRE_EXEC_DEV` and `POST_EXEC_DEV` have been added. This is primarily intended to separate use cases for the interpreter maintainers (via subclasses of `Interpreter`) and the notebook users (via the `ZeppelinContext` class), as otherwise user attempts at registering a hook would overwrite that set by the interpreter maintainer. - Global scope for hook registration is supported for the developer use cases. ### What type of PR is it? New Feature ### What is the Jira issue? [ZEPPELIN-1423](https://issues.apache.org/jira/browse/ZEPPELIN-1423) ### How should this be tested? In a new note, add the following lines of code to a paragraph: ```python %pyspark z.registerHook("post_exec", "print 'This code should be executed before the paragraph code!'") z.registerHook("pre_exec", "print 'This code should be executed after the paragraph code!'") ``` Then run any other paragraph in the note containing some other code, eg ```python %pyspark print "This code should be entered into the paragraph by the user!" ``` The output should be: ``` This code should be executed before the paragraph code! This code should be entered into the paragraph by the user! This code should be executed after the paragraph code! ``` You should also test out the other two methods (`getCallback()` and `unregisterCallback()`) specified in `ZeppelinContext.java`. One final caveat that should be mentioned: If there are errors in the code you specify for a pre-execute event, it will render the interpreter useless since the current implementation prepends the the code specified in `pre_exec` directly to the paragraph entered code before calling `interpret()`. The current workaround for this would be to either restart the interpreter group or call `unregisterHook()` via a different REPL within the interpreter group (eg, `z.unregisterHook("pre_exec", "pyspark")` from the spark interpreter). I would appreciate if anyone here would be willing to share any better approaches here. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Alex Goodman <[email protected]> Closes apache#1470 from agoodm/ZEPPELIN-1423v2 and squashes the following commits: 56ede60 [Alex Goodman] Automatically detect default interpreter for registerHook() 044a99d [Alex Goodman] Ensure that registered hooks are applied after call to open() 1331fe1 [Alex Goodman] Update interpreters.md 07cac65 [Alex Goodman] Implemented user-defined hook registry system for spark/pyspark interpreters 8fad936 [Alex Goodman] Added Interpreter Hooks to Interpreter Process
… incompatible license ### What is this PR for? According to https://developers.arcgis.com/terms/ and and http://www.esri.com/legal/software-license, current map visualization depends on an online service that has incompatible restrictions to Apache 2 license. Please see apache#1210 (comment) Possible alternative way is explained [here](apache#1210 (comment)). I'm very sad to remove this very good contribution. So please anyone review the license and comment here if you have other opinions, or other way around. ### What type of PR is it? [Task] ### Todos * [x] - remove implementation from source tree ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1532 ### Questions: * Does the licenses files need update? yes * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <[email protected]> Closes apache#1501 from Leemoonsoo/ZEPPELIN-1532 and squashes the following commits: b5ad91a [Lee moon soo] Remove map visualization dad81c4 [Lee moon soo] Remove map visualization from source tree
### What is this PR for? Graceful handling of errors in storage sync layer ### What type of PR is it? Bug Fix | Improvement ### Todos * [x] - catch and continue ### What is the Jira issue? [ZEPPELIN-1557](https://issues.apache.org/jira/browse/ZEPPELIN-1557) ### How should this be tested? green CI ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Khalid Huseynov <[email protected]> Closes apache#1532 from khalidhuseynov/fix/storage-graceful-sync and squashes the following commits: d036d9e [Khalid Huseynov] catch exception, log, and continue
…broadcast ### What is this PR for? This PR addresses part of multi-user note management in Zeppelin. One of the tasks namely listing notes per user on Zeppelin start was addressed in apache#1330. However that PR didn't solve all problems, and reloading notes was incomplete as well as socket broadcast was not user aware [ZEPPELIN-1437](https://issues.apache.org/jira/browse/ZEPPELIN-1437), [ZEPPELIN-1438](https://issues.apache.org/jira/browse/ZEPPELIN-1438). This PR addresses those issue. ### What type of PR is it? Improvement ### Todos * [x] - list notes per user on reload * [x] - broadcast per user (multicast) * [x] - tests * [x] - use authorization module to filter notes on sync * [x] - broadcast on permissions change * [ ] - discussion and review ### What is the Jira issue? [Zeppelin-1437](https://issues.apache.org/jira/browse/ZEPPELIN-1437), [ZEPPELIN-1438](https://issues.apache.org/jira/browse/ZEPPELIN-1438) ### How should this be tested? 1. Start Zeppelin 2. Login as user1, and user2 on different windows 3. Each user should be able to see their own note workbench 4. If note changed to private (readers, writers not empty), that note should disappear from others note workbench. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: Khalid Huseynov <[email protected]> Closes apache#1392 from khalidhuseynov/feat/multi-user-notes and squashes the following commits: a2ce268 [Khalid Huseynov] broadcast note list on perm update - zeppelin-1438 9cf1d88 [Khalid Huseynov] fix init not to initialize every time 17eae84 [Khalid Huseynov] bugfix: add precondition for NP 781207e [Khalid Huseynov] bugfix: reload only once 537cc0e [Khalid Huseynov] apply filter from authorization in sync 09e6723 [Khalid Huseynov] notebookAuthorization as singleton 9427e62 [Khalid Huseynov] multicast fine grained note lists to users instead of broadcast 6614e2b [Khalid Huseynov] improve tests 1399407 [Khalid Huseynov] remove unused imports d9c3bc9 [Khalid Huseynov] filter reload using predicates 92f37f5 [Khalid Huseynov] substitute old getAllNotes(subject) with new implementation b7f19c9 [Khalid Huseynov] separate getAllNotes() and getAllNotes(subject) 17e2d4c [Khalid Huseynov] first draft
### What is this PR for? This is to resolve possible NPEs in storage mechanism related to empty subject ### What type of PR is it? Bug Fix ### Todos * [x] - pass `subject` instead of `null` ### What is the Jira issue? [ZEPPELIN-1487](https://issues.apache.org/jira/browse/ZEPPELIN-1487) ### How should this be tested? storage tests pass, CI green ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Khalid Huseynov <[email protected]> Closes apache#1459 from khalidhuseynov/fix/storage-npe and squashes the following commits: 1f2dcd3 [Khalid Huseynov] pass subject to get method in sync
…-server, zeppelin-zengine ### What is this PR for? While running jUnit tests in zeppelin-server & zeppelin-zengine projects, several note are created during the process and those notes are never purged after the execution of the tests. ### What type of PR is it? Bug Fix ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1542 ### How should this be tested? Prerequisites: 1. Get the list of note that already exist in server. Now build zeppelin project by running maven commands(mvn clean package)(mvn clean package -pl 'zeppelin-interpreter,zeppelin-zengine,zeppelin-server'). 1. Now no extra note should get created after running the maven command ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: rajarajan-g <[email protected]> Closes apache#1524 from rajarajan-g/ZEPPELIN-1542 and squashes the following commits: d97cb4a [rajarajan-g] code changes for ZEPPELIN-1542
…terpreter. ### What is this PR for? This PR fixes dependency library loading bug when creating new interpreter. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1534 ### How should this be tested? 1. Create new JDBC interpreter refer to http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/jdbc.html#mysql. 2. Create new paragraph for testing new JDBC interpreter. 3. Run paragraph. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <[email protected]> Closes apache#1507 from astroshim/ZEPPELIN-1534 and squashes the following commits: aebe352 [astroshim] add loadInterpreterDependencies function in createNewSetting()
### What is this PR for? The result of an aggregation query returned by the interpreter contains only "key" and "doc_count" in case of a multi-buckets aggregations. But the result returned by Elasticsearch can contain more data according to the query. This PR is an improvement of the result returned by the interpreter. ### What type of PR is it? [Improvement] ### Todos * [X] - Dev of the improvement in the interpreter * [X] - Add a test case ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1537 ### How should this be tested? In a paragraph, enter a query with multiple aggregations: search /logs { "aggs" : { "length" : { "terms": { "field": "status" }, "aggs" : { "sum_length" : { "sum" : { "field" : "content_length" } } } } } ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Bruno Bonnin <[email protected]> Closes apache#1508 from bbonnin/master and squashes the following commits: a0a7bb9 [Bruno Bonnin] Elasticsearch improvement for results of aggregations
This PR replaces apache#1309 Author: Chris Snow <[email protected]> Closes apache#1531 from snowch/snowch-patch-1 and squashes the following commits: f5f7a70 [Chris Snow] use toString() method of CommandLine for logging 09a6d23 [Chris Snow] added missing semi-colon for statement 25e8a29 [Chris Snow] [ZEPPELIN-1314] dump out the R command
### What is this PR for? A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html ### What type of PR is it? [Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: LantaoJin <[email protected]> Closes apache#1511 from LantaoJin/master and squashes the following commits: 671cd68 [LantaoJin] keep old py4j for backward compatibility 3bd84cc [LantaoJin] [ZEPPELIN-1544]upgrade spark version to 2.0.1 as profile spark-2.0
…tebooks ### What is this PR for? Currently, if you have a Notebook large number of, There is a sharp decrease in performance of the Code editor. The number and Paragraph creation of the Notebook does not have a relationship. We are should always use the Code editor of the same performance. I had to print a Notebook list only if there is a request. There was quite a lot of performance improvements. ### What type of PR is it? Improvement ### Todos - [x] - change Notebook list render logic ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1559 https://issues.apache.org/jira/browse/ZEPPELIN-1513 ### How should this be tested? 1. Create a notebook over 1000. (tested 10,000) I've used script on this. (important! please, backup for notebooks directory.) ``` #/bin/bash idx="0" while [ $idx -lt 1000 ] # notebooks count 1000 do mkdir $idx echo """ { \"paragraphs\": [ { \"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\", \"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\", \"config\": { \"colWidth\": 12.0, \"graph\": { \"mode\": \"table\", \"height\": 300.0, \"optionOpen\": false, \"keys\": [], \"values\": [], \"groups\": [], \"scatter\": {}, \"map\": { \"baseMapType\": \"Streets\", \"isOnline\": true, \"pinCols\": [] } }, \"enabled\": true, \"editorMode\": \"ace/mode/scala\" }, \"settings\": { \"params\": {}, \"forms\": {} }, \"apps\": [], \"jobName\": \"paragraph_1476345687682_-1459653828\", \"id\": \"20161013-170127_68316618\", \"dateCreated\": \"Oct 13, 2016 5:01:27 PM\", \"status\": \"READY\", \"progressUpdateIntervalMs\": 500 } ], \"name\": \""${idx}"\", \"id\": \""${idx}"\", \"angularObjects\": { \"2BZSC9D3G:shared_process\": [], \"2BXGWF5TF:shared_process\": [], \"2BYKTADMA:shared_process\": [], \"2BYFT9HTZ:shared_process\": [], \"2BXDTJMED:shared_process\": [], \"2BZHPZ6NS:shared_process\": [], \"2BYKRWYB3:shared_process\": [], \"2BZH2UAT4:shared_process\": [], \"2BZ38J35G:shared_process\": [], \"2BXJ2X464:shared_process\": [], \"2BYQ57ED9:shared_process\": [], \"2BXDSA2SN:shared_process\": [], \"2BWZCTD2B:shared_process\": [], \"2BWT81MNU:shared_process\": [], \"2BWA7ZTRD:shared_process\": [], \"2BXW6X5KN:shared_process\": [], \"2BVWDQPXH:shared_process\": [], \"2BZGAF8KG:shared_process\": [] }, \"config\": {}, \"info\": {} } """ > $idx/note.json idx=$[$idx+1] done ``` 2. zeppelin restart or refresh notebook. 3. Try coding in any notebook. (fast) ### Screenshots (if appropriate) #### before ( 5000 notebooks)  #### after ( 5000 notebooks)  ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <[email protected]> Closes apache#1533 from cloverhearts/ZEPPELIN-1559-Notebook-broadcast-opti and squashes the following commits: bd58102 [CloverHearts] id Delete unused for navbar 14e88bb [CloverHearts] selectively modified to render the Notebook list.
### What is this PR for? Enabling each user to run same interpreter. ### What type of PR is it? [Improvement] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1210 ### How should this be tested? 1. Enable shiro to use authentication mode 1. Check `per user` in your interpreter tab 1. Run different paragraphs with different users 1. Run `%spark sc.version`, you will see the two `res0: ...` in your paragraphs ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: CloverHearts <[email protected]> Author: Jongyoul Lee <[email protected]> Closes apache#1265 from jongyoul/ZEPPELIN-1210 and squashes the following commits: 48a0d8e [Jongyoul Lee] Fixed ZEPPELIN-1542 Fixed flaky test e84703d [Jongyoul Lee] Fixed ZEPPELIN-1542 ad80951 [Jongyoul Lee] Fixed some wrong logic of getInterpreterInstanceKey cee39f4 [Jongyoul Lee] Fixed to pass shiro information to InterpreterFactory from ZeppelinServer 5e7da34 [Jongyoul Lee] Changed instanceKey and processKey for dealing with new UI d201950 [CloverHearts] fix eqeqeq issue for frontweb b18bff4 [CloverHearts] implement frontend for interpreter per user mode and misc mode 1f64e52 [CloverHearts] change default value for pernote and peruser 787a366 [CloverHearts] change Back-end test cases and member type (perNote, perUser) 8586e1f [CloverHearts] change ui for interpreter running Per x mode 0b5d671 [Jongyoul Lee] Fixed the style 960bde1 [Jongyoul Lee] Removed SecurityUtils.getPrincipal Added fromMessage.principal to get right user 01c7cf1 [Jongyoul Lee] Fixed NPE while testing ZeppelinSparkClusterTest 1fb50ab [Jongyoul Lee] Fixed NPE while testing ZeppelinSparkClusterTest d1c4344 [Jongyoul Lee] Fixed getEditorSetting for having users' info 12a27db [Jongyoul Lee] Fixed test after rebase 510942b [Jongyoul Lee] Fixed test after rebase cb66946 [Jongyoul Lee] Fixed test after rebase 18b39bd [Jongyoul Lee] Fixed test after rebase daa634f [Jongyoul Lee] Fixed some tests after rebase ed558be [Jongyoul Lee] Fixed some tests after rebase fa7fccb [Jongyoul Lee] Fixed destroying process of remoteInterpreterProcess 0a73241 [Jongyoul Lee] Fixed conflict while rebasing. df423d3 [Jongyoul Lee] Fixed NotebookRestApiTest b151366 [Jongyoul Lee] Fixed some codes after rebase a32afd7 [Jongyoul Lee] Fixed some tests 7b7eb78 [Jongyoul Lee] Fixed some tests 47cc668 [Jongyoul Lee] Fixed tests to use AuthenticationInfo 012cf99 [Jongyoul Lee] Fixed some mismatch after rebase 9a03d40 [Jongyoul Lee] Reverted some value to default ones 8589545 [Jongyoul Lee] Added option in UI ccbedc1 [Jongyoul Lee] WIP 94dfed2 [Jongyoul Lee] WIP 6480d1d [Jongyoul Lee] resolved conflicts
…ok with group permission ### What is this PR for? Zeppelin home page list notebooks doesn't show notebook with group permission ### What type of PR is it? [Bug Fix] ### Todos * [x] - consume userAndRole instead of AuthenticationInfo ### What is the Jira issue? * [ZEPPELIN-1483](https://issues.apache.org/jira/browse/ZEPPELIN-1483) ### How should this be tested? In current scenario only those notebook lists that have direct user permission, those with group does not list up, but if user have link to those notebook, it can still be accessed. IMO the notebook with group permission should also be listed in the home screen. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? n/a * Is there breaking changes for older versions? n/a * Does this needs documentation? n/a Author: Prabhjyot Singh <[email protected]> Author: Prabhjyot Singh <[email protected]> Closes apache#1454 from prabhjyotsingh/ZEPPELIN-1483 and squashes the following commits: 2484833 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-1483 c8d810e [Prabhjyot Singh] organise imports d3261c4 [Prabhjyot Singh] consume userAndRole instead of AuthenticationInfo
spelling error responce => response seprated => seperated
correct separated
fix typo seperated -> separated
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
Todos
What is the Jira issue?
How should this be tested?
Outline the steps to test the PR here.
Screenshots (if appropriate)
Questions: