If I understand correctly, you'll just want to package your implementation
with your preference of project manager (maven, sbt, etc.) which registers
your dialect implementation into JdbcDialects, and pass the jar and let end
users load the jar. That will automatically do everything and they can us
Not sure, too.
Can't you use Spark Packages for your scenario?
https://spark-packages.org/
On Thu, Dec 12, 2019 at 9:46 AM Hyukjin Kwon wrote:
> I am not so sure about it too. I think it is enough to expose JDBCDialect
> as an API (which seems already is).
> It brings some overhead to dev (e.g.
I am not so sure about it too. I think it is enough to expose JDBCDialect
as an API (which seems already is).
It brings some overhead to dev (e.g., to test and review PRs related to
another third party).
Such third party integration might better exist as a third party library
without a strong reaso
> Is this something that would be exposed/relevant to the Python API? Or is
this just for people implementing their own Spark data source?
It's latter, and it also helps simplifying built-in data sources as well
(as I found the needs while working on
https://github.com/apache/spark/pull/26845)
On
Nice, thanks for the answer! I'll craft a PR soon. Thanks again.
On Thu, Dec 12, 2019 at 3:32 AM Ryan Blue wrote:
> Sounds good to me, too.
>
> On Wed, Dec 11, 2019 at 1:18 AM Jungtaek Lim
> wrote:
>
>> Thanks for the quick response, Wenchen!
>>
>> I'll leave this thread for early tomorrow so t
Thank you all. I'll make a PR to Apache Spark website.
Bests,
Dongjoon.
On Tue, Dec 10, 2019 at 11:43 PM Wenchen Fan wrote:
> Sounds good. Thanks for bringing this up!
>
> On Wed, Dec 11, 2019 at 3:18 PM Takeshi Yamamuro
> wrote:
>
>> That looks nice, thanks!
>> I checked the previous v2.4.4 r
Is this something that would be exposed/relevant to the Python API? Or is
this just for people implementing their own Spark data source?
On Wed, Dec 11, 2019 at 12:35 AM Jungtaek Lim
wrote:
> Hi devs,
>
> I'd like to propose to add close() on DataWriter explicitly, which is the
> place for resou
Sounds good to me, too.
On Wed, Dec 11, 2019 at 1:18 AM Jungtaek Lim
wrote:
> Thanks for the quick response, Wenchen!
>
> I'll leave this thread for early tomorrow so that someone in US timezone
> can chime in, and craft a patch if no one objects.
>
> On Wed, Dec 11, 2019 at 4:41 PM Wenchen Fan
Recapping today's sync on the wider dev list for visibility:
The original proposals here can be refactored into 3 distinct changes which
could be integrated iteratively. In order of decreasing priority:
1. Allow MapStatus to take an arbitrary/opaque payload and rip out hard
references to ex
My moderately informed take is that the "Hadoop 2.7" build is really a
"Hadoop 2.x" build and AFAIK should work with 2.8, 2.9, but, I
certainly haven't tested it nor have the PR builders. Just use the
"Hadoop provided" build on your env. Of course, you might well want to
use Hadoop 3.x (3.2.x speci
Hi,
We've been considering using the download package Spark 2.4.4 that's
pre-built for Hadoop 2.7 with Hadoop 2.7.7.
When used with Spark, Hadoop 2.7 is often quoted as the most stable.
However, Hadoop 2.7.7 is End Of Life. The most recent Hadoop vulnerabilities
have only been fixed in versions
It kind of already is. I was able to build the VerticaDialect as a sort of
plugin as follows:
Check out apache/spark tree
Copy in VerticaDialect.scala
Build with “mvn -DskipTests compile”
package the compiled class plus companion object into a JAR
Copy JAR to jars folder in Spark binary installa
Can we make the JDBCDialect a public API that users can plugin? It looks
like an end-less job to make sure Spark JDBC source supports all databases.
On Wed, Dec 11, 2019 at 11:41 PM Xiao Li wrote:
> You can follow how we test the other JDBC dialects. All JDBC dialects
> require the docker integr
You can follow how we test the other JDBC dialects. All JDBC dialects
require the docker integration tests.
https://github.com/apache/spark/tree/master/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc
On Wed, Dec 11, 2019 at 7:33 AM Bryan Herger
wrote:
> Hi, to answer
Hi, to answer both questions raised:
Though Vertica is derived from Postgres, Vertica does not recognize type names
TEXT, NVARCHAR, BYTEA, ARRAY, and also handles DATETIME differently enough to
cause issues. The major changes are to use type names and date format
supported by Vertica.
For tes
How can the dev community test it?
Xiao
On Wed, Dec 11, 2019 at 6:52 AM Sean Owen wrote:
> It's probably OK, IMHO. The overhead of another dialect is small. Are
> there differences that require a new dialect? I assume so and might
> just be useful to summarize them if you open a PR.
>
> On Tue,
It's probably OK, IMHO. The overhead of another dialect is small. Are
there differences that require a new dialect? I assume so and might
just be useful to summarize them if you open a PR.
On Tue, Dec 10, 2019 at 7:14 AM Bryan Herger
wrote:
>
> Hi, I am a Vertica support engineer, and we have ope
Thanks for the quick response, Wenchen!
I'll leave this thread for early tomorrow so that someone in US timezone
can chime in, and craft a patch if no one objects.
On Wed, Dec 11, 2019 at 4:41 PM Wenchen Fan wrote:
> PartitionReader extends Closable, seems reasonable to me to do the same
> for
18 matches
Mail list logo
