Hi Jongyoul,
Thanks for merging ZEPPELIN-440. I downloaded changes from
https://github.com/apache/incubator-zeppelin/pull/455 and here's what I
found.
When I ran two Vertica queries, I got the error below:
ERROR [2015-12-04 09:18:47,153] ({pool-2-thread-3} Job.java[run]:183) - Job
failed
java.lang.AbstractMethodError:
com.vertica.jdbc.VerticaConnectionImpl.isValid(I)Z
at
org.apache.zeppelin.hive.HiveInterpreter.getConnection(HiveInterpreter.java:163)
at
org.apache.zeppelin.hive.HiveInterpreter.getStatement(HiveInterpreter.java:196)
at
org.apache.zeppelin.hive.HiveInterpreter.executeSql(HiveInterpreter.java:208)
at
org.apache.zeppelin.hive.HiveInterpreter.interpret(HiveInterpreter.java:292)
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:300)
at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
at
org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:157)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
When I removed the connection.isValid check, the queries ran without any
errors.
if (connection.isClosed()) { // || !connection.isValid(10)) {
However, I noticed that even though the jobs for the two queries (select
sleep(60)) were started around the same time, they both finished 120
seconds after they were started. I also looked at the sessions in Vertica
and found that 'select sleep(60), 4' was started after 'select sleep(60),
3' completed. Is this behavior expected? Our goal is to prevent one long
Vertica query from delaying other Vertica queries.
MacBook-Pro logs $ cat
zeppelin-interpreter-hive-pwagle-MacBook-Pro.local.log
INFO [2015-12-04 09:28:44,545] ({Thread-0}
RemoteInterpreterServer.java[run]:97) - Starting remote interpreter server
on port 54257
INFO [2015-12-04 09:28:44,818] ({pool-1-thread-3}
RemoteInterpreterServer.java[createInterpreter]:168) - Instantiate
interpreter org.apache.zeppelin.hive.HiveInterpreter
INFO [2015-12-04 09:28:44,862] ({pool-2-thread-2}
SchedulerFactory.java[jobStarted]:132) - Job
remoteInterpretJob_1449250124861 started by scheduler
org.apache.zeppelin.hive.HiveInterpreter1131923628
ERROR [2015-12-04 09:28:44,865] ({pool-2-thread-2}
HiveInterpreter.java[open]:128) - hive will be ignored. driver.hive and
hive.url is mandatory.
INFO [2015-12-04 09:28:44,865] ({pool-2-thread-2}
HiveInterpreter.java[interpret]:290) - PropertyKey: vertica, SQL command:
'select sleep(60), 3'
INFO [2015-12-04 09:28:53,312] ({pool-2-thread-3}
SchedulerFactory.java[jobStarted]:132) - Job
remoteInterpretJob_1449250133312 started by scheduler
org.apache.zeppelin.hive.HiveInterpreter1131923628
INFO [2015-12-04 09:28:53,312] ({pool-2-thread-3}
HiveInterpreter.java[interpret]:290) - PropertyKey: vertica, SQL command:
'select sleep(60), 4'
INFO [2015-12-04 09:30:48,970] ({pool-2-thread-3}
SchedulerFactory.java[jobFinished]:138) - Job
remoteInterpretJob_1449250133312 finished by scheduler
org.apache.zeppelin.hive.HiveInterpreter1131923628
INFO [2015-12-04 09:30:48,970] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:138) - Job
remoteInterpretJob_1449250124861 finished by scheduler
org.apache.zeppelin.hive.HiveInterpreter1131923628
Thanks for your help!
Prasad
On Thu, Dec 3, 2015 at 9:12 PM, Jongyoul Lee <jongy...@gmail.com> wrote:
> Hi Prasad,
>
> PostgresSqlInterpreter doesn't seems to support parallel execution. I
> merged ZEPPELIN-440 for supporting it today. Could you please use this
> interpreter? You can find docs and changes in
> https://github.com/apache/incubator-zeppelin/pull/455
>
> Regards,
> Jongyoul
>
> On Fri, Dec 4, 2015 at 10:32 AM, Prasad Wagle <prasadwa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are evaluating Zeppelin and are excited about its potential. Right
>> now, we are using it to access Presto and Vertica. We downloaded
>> https://github.com/apache/incubator-zeppelin/pull/68 for the Presto
>> interpreter and are using PostgreSqlInterpreter to access Vertica.
>>
>> The main issue we are facing is related to serial execution of queries.
>> Since the default scheduler is FIFOScheduler, when a Vertica query is
>> running in one notebook, all other Vertica queries issued from the same or
>> different notes are pending.
>>
>> We changed the getScheduler method in PostgreSqlInterpreter to return
>> ParallelScheduler.
>> return SchedulerFactory.singleton().createOrGetParallelScheduler(
>> PostgreSqlInterpreter.class.getName() + this.hashCode(), 2);
>>
>> But this resulted in the exception below. Is this expected behavior? Does
>> the PostgreSqlInterpreter have to be redesigned to support parallel
>> execution of queries?
>>
>> In zeppelin-interpreter-psql-pwagle-MacBook-Pro.local.log:
>>
>> INFO [2015-12-03 10:33:35,891] ({pool-1-thread-2}
>> RemoteInterpreterServer.java[createInterpreter]:168) - Instantiate
>> interpreter org.apache.zeppelin.postgresql.PostgreSqlInterpreter
>> INFO [2015-12-03 10:33:35,940] ({pool-2-thread-2}
>> SchedulerFactory.java[jobStarted]:132) - Job
>> remoteInterpretJob_1449167615939 started by scheduler
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter761108485
>> INFO [2015-12-03 10:33:35,942] ({pool-2-thread-2}
>> PostgreSqlInterpreter.java[open]:132) - Open psql connection!
>> INFO [2015-12-03 10:33:35,942] ({pool-2-thread-2}
>> PostgreSqlInterpreter.java[close]:182) - Close psql connection!
>> INFO [2015-12-03 10:33:36,511] ({pool-2-thread-2}
>> SqlCompleter.java[getSqlKeywordsCompletions]:144) - JDBC
>> DriverName:/vertica.jar-sql.keywords
>> INFO [2015-12-03 10:33:37,779] ({pool-2-thread-3}
>> SchedulerFactory.java[jobStarted]:132) - Job
>> remoteInterpretJob_1449167617779 started by scheduler
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter761108485
>> INFO [2015-12-03 10:33:49,352] ({pool-2-thread-2}
>> PostgreSqlInterpreter.java[open]:153) - Successfully created psql connection
>> INFO [2015-12-03 10:33:49,352] ({pool-2-thread-2}
>> PostgreSqlInterpreter.java[interpret]:288) - Run SQL command '
>> select sleep(30), 1;'
>> INFO [2015-12-03 10:33:49,352] ({pool-2-thread-3}
>> PostgreSqlInterpreter.java[interpret]:288) - Run SQL command '
>> select sleep(30), 2;'
>> ERROR [2015-12-03 10:34:49,482] ({pool-2-thread-3}
>> PostgreSqlInterpreter.java[executeSql]:271) - Cannot run
>> select sleep(30), 2;
>> java.sql.SQLException: [Vertica][JDBC](12060) ResultSet object has been
>> closed.
>> at com.vertica.jdbc.SForwardResultSet.checkIfOpen(Unknown Source)
>> at com.vertica.jdbc.SForwardResultSet.next(Unknown Source)
>> at
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter.executeSql(PostgreSqlInterpreter.java:235)
>> at
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter.interpret(PostgreSqlInterpreter.java:289)
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:299)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
>> at
>> org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:157)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> ERROR [2015-12-03 10:34:49,482] ({pool-2-thread-2} Job.java[run]:183) -
>> Job failed
>> org.apache.zeppelin.interpreter.InterpreterException:
>> java.lang.NullPointerException
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:61)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:299)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
>> at
>> org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:157)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: java.lang.NullPointerException
>> at
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter.executeSql(PostgreSqlInterpreter.java:262)
>> at
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter.interpret(PostgreSqlInterpreter.java:289)
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
>> ... 11 more
>> INFO [2015-12-03 10:34:49,489] ({pool-2-thread-3}
>> SchedulerFactory.java[jobFinished]:138) - Job
>> remoteInterpretJob_1449167617779 finished by scheduler
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter761108485
>> INFO [2015-12-03 10:34:49,490] ({pool-2-thread-2}
>> SchedulerFactory.java[jobFinished]:138) - Job
>> remoteInterpretJob_1449167615939 finished by scheduler
>> org.apache.zeppelin.postgresql.PostgreSqlInterpreter761108485
>>
>>
>> Thanks,
>> Prasad
>>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
