[GitHub] zeppelin issue #2630: [HOTFIX] fix build spark and R interpreters
Github user tinkoff-dwh commented on the issue: https://github.com/apache/zeppelin/pull/2630 @jongyoul @zjffdu after merge https://github.com/apache/zeppelin/pull/2596 https://github.com/apache/zeppelin/pull/2596/files#diff-dbda0c4083ad9c59ff05f0273b5e760fR935 `property.getProperty...` ---
[GitHub] zeppelin pull request #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
GitHub user zjffdu opened a pull request: https://github.com/apache/zeppelin/pull/2631 ZEPPELIN-2197. Interpreter Lifecycle Manager ### What is this PR for? This PR implement the lifecycle manager. There're 2 implementions. * NullLifecycleManager. Nothing is done as before. User has to start/stop interpreter explicitly in UI. * TimeoutLifecycleManager.Interpreter will be closed after idle for one threshold of time. By default it is 1 hour. ### What type of PR is it? [ Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2197 ### How should this be tested? Unit test is added. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2197 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2631.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2631 commit cd112f9abb8538e5684fbf6ddb61013137150389 Author: Jeff Zhang Date: 2017-09-28T05:02:23Z save commit 6376d7617617daf92f67168c1fc767054e5616b6 Author: Jeff Zhang Date: 2017-10-25T06:18:31Z ZEPPELIN-2197. Interpreter Lifecycle Manager ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2631 @Leemoonsoo @jongyoul @Tagar Please help review. Thanks ---
[GitHub] zeppelin pull request #2628: [ZEPPELIN-2999] Cannot create shell interpreter...
Github user prabhjyotsingh closed the pull request at: https://github.com/apache/zeppelin/pull/2628 ---
[GitHub] zeppelin issue #2628: [ZEPPELIN-2999] Cannot create shell interpreter withou...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/2628 Thanks for the review, will merge this if no more discussion. ---
[GitHub] zeppelin pull request #2628: [ZEPPELIN-2999] Cannot create shell interpreter...
GitHub user prabhjyotsingh reopened a pull request: https://github.com/apache/zeppelin/pull/2628 [ZEPPELIN-2999] Cannot create shell interpreter without timeout property ### What is this PR for? A user can not run any shell interpreter if "shell.command.timeout.millisecs" is not present in interpreter setting. ### What type of PR is it? [Improvement] ### What is the Jira issue? * [https://issues.apache.org/jira/browse/ZEPPELIN-2999](https://issues.apache.org/jira/browse/ZEPPELIN-2999) ### How should this be tested? * Remove "shell.command.timeout.millisecs" from sh interpreter settings and then try to run any sh paragraph, it should run without any error. ### Questions: * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/prabhjyotsingh/zeppelin ZEPPELIN-2999 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2628.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2628 commit e2a62f63d2a14ac0c13ca9774dbede3946736f8b Author: Prabhjyot Singh Date: 2017-10-23T06:43:24Z ZEPPELIN-2999: Cannot create shell interpreter without timeout property commit 4406157d2205f95e74815406f6f9e3ec20127150 Author: Prabhjyot Singh Date: 2017-10-23T07:31:19Z use "getProperty(String key, String defaultValue)" instead of "getProperty(String key)" ---
Re: Read Hbase table in pyspark
please help on this .
On Wed, Oct 25, 2017 at 11:35 AM, Indtiny S wrote:
> Hi,
> I am trying to read Hbase tables in pyspark data frame,
> I am using the below code
> but I am getting the ClassNotFoundException error
>
> df=sqlContext.read.format('jdbc').options(driver="org.
> apache.phoenix.jdbc.PhoenixDriver",url='jdbc:
> phoenix:localhost:2181:/hbase-unsecure',dbtable='table_name').load()
>
>
> java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
>
>
> I have the libraries phoenix-spark-4.7.0-HBase-1.1.jar and
> phoenix-4.7.0-HBase-1.1-client.jar but dont know where to place them .
>
>
> I am using zeppelin 0.7.0
>
>
> Rgds
>
> In
>
>
>
>
>
>
[jira] [Created] (ZEPPELIN-3007) Display a note name without any path in the tab title
Tinkoff DWH created ZEPPELIN-3007: - Summary: Display a note name without any path in the tab title Key: ZEPPELIN-3007 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3007 Project: Zeppelin Issue Type: Improvement Reporter: Tinkoff DWH The note may have a long path, for example: "Dir 1/SubDir 2/Note Name". Now, the full path and the name are displayed in the browser tab title. It will be much better if only a note name is displayed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Error while building Zeppelin master
This looks like a bug building in Window env. Can you do that on Linux or Mac? On Tue, Oct 24, 2017 at 11:56 PM, Andrea Santurbano wrote: > Hi guys, > when i try to build Zeppelin from apache master repo on Windows 10 i get > this error: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (copy-artifact) > on project zeppelin-interpreter: Error copying artifact from > C:\Users\Andrea\workspace_zeppelin\zeppelin-master\ > zeppelin-interpreter\target\zeppelin-interpreter-0.8.0-SNAPSHOT.jar > to > C:\Users\Andrea\workspace_zeppelin\zeppelin-master\zeppelin-interpreter\ > interpreter\Zeppelin: > Interpreter\zeppelin-interpreter-0.8.0-SNAPSHOT.jar > > Can someone help me to understand why? > Thanks > Andrea > -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net
[GitHub] zeppelin issue #2630: [HOTFIX] fix build spark and R interpreters
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/2630 Yes, what I wondered is how that PR passed our CI. `property` doesn't exist in the class and it shouldn't have passed CI. May we have a hole on our CI? ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/2631 I have a basic question. Does it work in `scoped` and `isolated`? ---
[GitHub] zeppelin issue #2630: [HOTFIX] fix build spark and R interpreters
Github user tinkoff-dwh commented on the issue: https://github.com/apache/zeppelin/pull/2630 @jongyoul this branch has no conflicts with branch master. In CI all ok, because branch of PR has field property. After merge jobs falls (CI, branch master) https://travis-ci.org/apache/zeppelin/builds/291922211?utm_source=github_status&utm_medium=notification ---
[GitHub] zeppelin issue #2630: [HOTFIX] fix build spark and R interpreters
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/2630 Yes, I know the current situation and this PR solves what kind of problem, but our merge script uses squash merge from PR branch and it means if that PR branch has `property` in that branch, it would have been merged by our merge script, otherwise if that PR branch doesn't have `property` in that branch, CI would fail. But AFAIK, that branch doesn't have `property` but CI passed. I just wanted to know how it happens. ---
Re: Read Hbase table in pyspark gives java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
Hi,
I'm not sure but you can try to locate them under interpreter/spark if you
can do it
JL
On Wed, Oct 25, 2017 at 3:05 PM, Indtiny S wrote:
> Hi,
> I am trying to read Hbase tables in pyspark data frame,
> I am using the below code
> but I am getting the ClassNotFoundException error
>
> df=sqlContext.read.format('jdbc').options(driver="org.
> apache.phoenix.jdbc.PhoenixDriver",url='jdbc:
> phoenix:localhost:2181:/hbase-unsecure',dbtable='table_name').load()
>
>
> java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
>
>
> I have the libraries phoenix-spark-4.7.0-HBase-1.1.jar and
> phoenix-4.7.0-HBase-1.1-client.jar but dont know where to place them .
>
>
> I am using zeppelin 0.7.0
>
>
> Rgds
>
> In
>
>
>
>
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user Tagar commented on the issue: https://github.com/apache/zeppelin/pull/2631 Thank you @zjffdu. I just thought about this scenario: a Spark job runs for 1.5 hours, would it be killed by the LifeCycleManager in this case? (assuming here default timeout of 1 hour) If it is, then might be nice also to have a grace period when an interpreter wouldn't be killed if it has a running job? In the above example, let's say timeout=1 hour and grace period=1 hour. So an interpreter would be killed if it is completely inactive for 1 hour, or in 2 hours if it had a Spark job that was still spinning. Thoughts? ---
[GitHub] zeppelin issue #2627: [ZEPPELIN-2368]. Option to run all paragraphs *sequent...
Github user namanmishra91 commented on the issue: https://github.com/apache/zeppelin/pull/2627 @zjffdu Yes, that's right. ---
[jira] [Created] (ZEPPELIN-3008) Please update Scala 2.11 version from 2.11.7 to 2.11.8
François Garillot created ZEPPELIN-3008: --- Summary: Please update Scala 2.11 version from 2.11.7 to 2.11.8 Key: ZEPPELIN-3008 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3008 Project: Zeppelin Issue Type: Wish Reporter: François Garillot Priority: Minor When using the Spark interpreter, we've managed to hit an issue with a 2.11.7/ 2.11.8 limited binary incompatibility, it seems. It involves a cell full of imports, some of which push a 2.11.8 scala-library on classpath. Full description on the deeplearning4j/Datavec project: https://github.com/deeplearning4j/DataVec/issues/444#issuecomment-339430722 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2631 @Tagar It won't be killed, because LifecycleManager will know client is polling job progress via api Interpreter.getProgress. https://github.com/apache/zeppelin/pull/2631/files#diff-f55969c5887615d2142806214747bba9R330 ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2631 @jongyoul For now, only Interpreter Process's lifecycle is controlled in TimeoutLifecycleManager. session level control will be done in future if necessary. https://github.com/apache/zeppelin/pull/2631/files#diff-53b72da47096c21dca5f395fe86761b3R16 ---
[GitHub] zeppelin pull request #2632: [HotFix] - Incorrect interpreter dir
GitHub user zjffdu opened a pull request: https://github.com/apache/zeppelin/pull/2632 [HotFix] - Incorrect interpreter dir ### What is this PR for? This is for the bug hotfix introduced in #2592 . The issue is that new interpreter created can not run properly because the incorrect interpreter dir. Thanks @tinkoff-dwh for reporting this issue. ### What type of PR is it? [ Hot Fix] ### Todos * [ ] - Task ### How should this be tested? * First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/zeppelin HotFix_Interpreter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2632.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2632 commit 2d7ab03c5b331152f3b3ed634a4bdd7c6350ffc2 Author: Jeff Zhang Date: 2017-10-25T13:53:37Z [HotFix] - Incorrect interpreter dir ---
[GitHub] zeppelin pull request #2632: [HotFix] - Incorrect interpreter dir
Github user zjffdu closed the pull request at: https://github.com/apache/zeppelin/pull/2632 ---
[GitHub] zeppelin issue #2627: [ZEPPELIN-2368]. Option to run all paragraphs *sequent...
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2627 @namanmishra91 In that case, the paragraph will be execute twice. One improvement thing I think we can do it in frontend is to freeze the frontend to disable user run paragraphes when user click the run all button. Thoughts ? ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user Gauravshah commented on the issue: https://github.com/apache/zeppelin/pull/2631 @zjffdu what if I am not on my desk and not polling the job ? ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2631 @Gauravshah It won't be killed. Because JobRunner in zeppelin server process will poll the job status periodically . Add one more test to verify it. ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user Tagar commented on the issue: https://github.com/apache/zeppelin/pull/2631 @zjffdu got it - thank you. ---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user Tagar commented on the issue: https://github.com/apache/zeppelin/pull/2631 One last thing - from user experience it would be convenient to know when their interpreters timed out. Something like a popup or just some sort of a graphical flag would do, I guess? Not sure how hard it'll be add this at this point. ---
[jira] [Created] (ZEPPELIN-3009) Don't iterate all the properties when getting property from ZeppelinConfiguration
Jeff Zhang created ZEPPELIN-3009: Summary: Don't iterate all the properties when getting property from ZeppelinConfiguration Key: ZEPPELIN-3009 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3009 Project: Zeppelin Issue Type: Improvement Reporter: Jeff Zhang For now, ZeppelinConfiguration would always iterate all the properties for getting any single property value, this is time wasteful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2631 @Tagar Good point, Currently there's no such things in frontend. For now, what user see in frontend is that a new interpreter group is created. (For spark interpreter, a new spark app is started) ---
[jira] [Created] (ZEPPELIN-3010) SparkInterpreterModeTest#testYarnClusterMode test failing consistently
Sonia Garudi created ZEPPELIN-3010:
--
Summary: SparkInterpreterModeTest#testYarnClusterMode test failing
consistently
Key: ZEPPELIN-3010
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3010
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-zengine
Affects Versions: 0.8.0
Environment: Ubuntu 14.04
x86, ppc64le
$ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
Reporter: Sonia Garudi
The testcase fails with the below error :
{code:java}
java.lang.RuntimeException: SPARK_HOME is not specified in interpreter-setting
for non-local mode, if you specify it in zeppelin-env.sh, please move that into
interpreter setting
at
org.apache.zeppelin.interpreter.launcher.SparkInterpreterLauncher.setupPropertiesForSparkR(SparkInterpreterLauncher.java:132)
at
org.apache.zeppelin.interpreter.launcher.SparkInterpreterLauncher.buildEnvFromProperties(SparkInterpreterLauncher.java:57)
at
org.apache.zeppelin.interpreter.launcher.ShellScriptLauncher.launch(ShellScriptLauncher.java:69)
at
org.apache.zeppelin.interpreter.InterpreterSetting.createInterpreterProcess(InterpreterSetting.java:649)
at
org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:59)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:100)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:159)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:129)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:287)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:204)
at
org.apache.zeppelin.interpreter.SparkInterpreterModeTest.testInterpreterBasics(SparkInterpreterModeTest.java:53)
at
org.apache.zeppelin.interpreter.SparkInterpreterModeTest.testYarnClusterMode(SparkInterpreterModeTest.java:130)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Re: Read Hbase table in pyspark gives java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
Hi,
I have those libraries but where to place those libraries so that zeppelin
can pick up.
or is there any way to set the library path using sparkcontext i.e using sc?
Regards
In
On Wed, Oct 25, 2017 at 9:22 PM, Jongyoul Lee wrote:
> Hi,
>
> I'm not sure but you can try to locate them under interpreter/spark if you
> can do it
>
> JL
>
> On Wed, Oct 25, 2017 at 3:05 PM, Indtiny S wrote:
>
> > Hi,
> > I am trying to read Hbase tables in pyspark data frame,
> > I am using the below code
> > but I am getting the ClassNotFoundException error
> >
> > df=sqlContext.read.format('jdbc').options(driver="org.
> > apache.phoenix.jdbc.PhoenixDriver",url='jdbc:
> > phoenix:localhost:2181:/hbase-unsecure',dbtable='table_name').load()
> >
> >
> > java.lang.ClassNotFoundException: org.apache.phoenix.jdbc.PhoenixDriver
> >
> >
> > I have the libraries phoenix-spark-4.7.0-HBase-1.1.jar and
> > phoenix-4.7.0-HBase-1.1-client.jar but dont know where to place them .
> >
> >
> > I am using zeppelin 0.7.0
> >
> >
> > Rgds
> >
> > In
> >
> >
> >
> >
> >
> >
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
