shota suzuki created ZEPPELIN-3109:
--------------------------------------
Summary: pyspark can't access mariadb via jdbc
Key: ZEPPELIN-3109
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3109
Project: Zeppelin
Issue Type: Bug
Components: Interpreters, pySpark, python-interpreter
Affects Versions: 0.7.3
Environment: host OS: centos7
docker: ver17.09-ce
container: apache/zeppelin:0.7.3
Reporter: shota suzuki
I want to access mariadb using pyspark but error raised.
my notebook as follows.
```bash
%spark.dep
z.load("/zeppelin/interpreter/jdbc/mariadb-java-client-2.2.0.jar")
->
res0: org.apache.zeppelin.dep.Dependency =
org.apache.zeppelin.dep.Dependency@6ed2e421
```
```bash
%spark.pyspark
import pyspark
print(pyspark.__version__)
dataframe = sqlContext.read.format("jdbc").options(
url ="jdbc:mariadb://mariadb:3306/db",
driver="org.mariadb.jdbc.Driver",
dbtable="table",
user="user",
password="pass"
).load().take(10)
->
2.1.0
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-400087723623135418.py", line 367, in <module>
raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-400087723623135418.py", line 360, in <module>
exec(code, _zcUserQueryNameSpace)
File "<stdin>", line 8, in <module>
File
"/zeppelin/interpreter/spark/pyspark/pyspark.zip/pyspark/sql/readwriter.py",
line 155, in load
return self._df(self._jreader.load())
File
"/zeppelin/interpreter/spark/pyspark/py4j-0.10.4-src.zip/py4j/java_gateway.py",
line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/zeppelin/interpreter/spark/pyspark/pyspark.zip/pyspark/sql/utils.py",
line 63, in deco
return f(*a, **kw)
File
"/zeppelin/interpreter/spark/pyspark/py4j-0.10.4-src.zip/py4j/protocol.py",
line 319, in get_return_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o47.load.
: java.lang.UnsatisfiedLinkError: Unable to load library 'c':
/usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:145)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:188)
at com.sun.jna.Library$Handler.<init>(Library.java:123)
at com.sun.jna.Native.loadLibrary(Native.java:255)
at com.sun.jna.Native.loadLibrary(Native.java:241)
at
org.mariadb.jdbc.internal.util.PidFactory$CLibrary.<clinit>(PidFactory.java:112)
at
org.mariadb.jdbc.internal.util.PidFactory.lambda$getInstance$0(PidFactory.java:78)
at
org.mariadb.jdbc.internal.com.send.SendHandshakeResponsePacket.writeConnectAttributes(SendHandshakeResponsePacket.java:211)
at
org.mariadb.jdbc.internal.com.send.SendHandshakeResponsePacket.send(SendHandshakeResponsePacket.java:184)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:796)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:775)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:453)
at
org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1084)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:493)
at
org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:156)
at org.mariadb.jdbc.Driver.connect(Driver.java:90)
at
org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:59)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:50)
at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:113)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:45)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
```
mariadb and zeppelin container are reachable. I confirmed it by jdbc
interpreter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
