Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[Antoine Pitrou]

Hello Alva,

This is a reasonable request, but it might come with its own drawbacks 
as well.


One significant drawback is that adding the Swift implementation to the 
cross-implementation integration tests will be slightly more complicated.
It is very important that all Arrow implementations are 
integration-tested against each other, otherwise we only have a 
theoretical guarantee that they are compatible. See how this is done here:

https://arrow.apache.org/docs/dev/format/Integration.html

Unless I'm mistaken, neither Swift nor Julia are running the integration 
tests.


Regards

Antoine.



Le 09/10/2023 à 22:26, Alva Bandy a écrit :

Hi,

I would like to request a repo for Arrow Swift (similar to arrow-rs).  Swift 
arrow is currently fully Swift and doesn't leverage the C++ libraries. One of 
the goals of Arrow Swift was to provide a fully Swift impl and splitting them 
now would help ensure that Swift Arrow stays on this path.

Also, the Swift Package Manager uses a git repo url to pull down a package.  
This can lead to a large download since the entire arrow repo will be pulled 
down just to include Arrow Swift.  It would be great to make this change before 
registering Swift Arrow with a Swift registry (such as Swift Package Registry).

Please let me know if this is possible and if so, what would be the process 
going forward.

Thank you,
Alva Bandy

Re: [Java][Discuss]: consensus for JDK 8 deprecation

[Dane Pitkin]

To summarize the discussion so far:

* Some Arrow Java users are still on JDK 8
* Arrow v14 is proposed as the final version with JDK 8 support
* Arrow v14 can support patch releases if necessary for JDK 8 users
* There is an open question to decide if JDK 11 should be dropped
simultaneously

Gang Wu, I'm curious what are your thoughts given your initial concerns?

-Dane

On Sat, Oct 7, 2023 at 12:00 AM Jacob Wujciak-Jens
 wrote:

> From a release engineer perspective (without java knowledge) I agree with
> Micah, I'd rather make a patch release for an older version if needed but
> modernize the codebase and simplify CI!
>
>
> On Sat, Oct 7, 2023 at 5:27 AM Micah Kornfield 
> wrote:
>
> > I think given the stability of Arrow Java, dropping support probably
> makes
> > sense.  If a bug comes up or consumers really need to new features we can
> > always make a patch release of an older version.
> >
> > On Thu, Oct 5, 2023 at 3:13 PM Dane Pitkin  >
> > wrote:
> >
> > > I also learned today that Apache Spark has dropped support for Java 8
> and
> > > 11 for their next release (v4.0)[1]. Should we consider dropping Java
> 11
> > as
> > > well?
> > >
> > > [1]https://github.com/apache/spark/pull/43005
> > >
> > > -Dane
> > >
> > > On Thu, Oct 5, 2023 at 3:30 PM Dane Pitkin 
> wrote:
> > >
> > > > I created a GH issue[1] proposing the removal of Java 8 support. It
> > > > would target the Arrow v15 release (~Jan 2024).
> > > >
> > > > IMO it would be in the best interest of the project for two major
> > > reasons:
> > > > 1. Unblock the Java Platform Module System (JPMS)[2] implementation.
> > > > 2. Unblock Arrow from upgrading dependencies that no longer support
> > Java
> > > > 8. (See [1] for examples)
> > > >
> > > > Since Arrow Java has been quite stable, will Java 8 users be okay
> with
> > > > pinning Arrow to the last supported release (v14) if the Arrow
> project
> > > > ultimately decides to remove Java 8 support?
> > > >
> > > >
> > > > [1]https://github.com/apache/arrow/issues/38051
> > > > [2]https://en.wikipedia.org/wiki/Java_Platform_Module_System
> > > >
> > > > -Dane
> > > >
> > > > On Fri, Sep 15, 2023 at 12:26 PM Dane Pitkin 
> > > wrote:
> > > >
> > > >> - As a low level library, users have to add specific flags to use
> > > >>>  Java 9 and up with Arrow to resolve issues with java.nio. This has
> > > >>>  been annoying for our customers constantly. If this is not
> resolved,
> > > >>>  I would say we may see a lot of complaints in the future.
> > > >>>
> > > >> I filed issue 37739[1] to track this, but it sounds like this can't
> be
> > > >> changed until Java 21 or 24.
> > > >>
> > > >> - It seems that the EOL of Java 8 from Oracle is Dec 2030 [2]. A lot
> > > >>>  users will still stay on it for a long time. At least this is true
> > for
> > > >>> our
> > > >>>  customers. So I am afraid we may not upgrade to newer versions
> > > >>>  of Arrow if it no longer supports Java 8.
> > > >>>
> > > >> Java 8 does have a long Extended Support timeline, but a recent
> > > >> report shows Java 11 increasing in adoption vs Java 8. "More than
> 56%
> > of
> > > >> applications are now using Java 11 in production (up from 48% in
> 2022
> > > and
> > > >> 11% in 2020). Java 8 is a close second with nearly 33% of
> applications
> > > >> using it in production (down from 46% in 2022)."[2]
> > > >> I expect the Java ecosystem will find a way to move on from Java 8
> > much
> > > >> sooner than 2030, meaning many of Arrow's dependencies could drop
> > > support
> > > >> for Java 8 before then. At this point, Arrow may be forced to
> support
> > a
> > > >> higher minimum Java version.
> > > >>
> > > >> That being said, it's hard to argue against real use cases. I'd be
> > > >> curious to hear what Java version other users of Arrow are using
> (and
> > if
> > > >> there is a timeline to upgrade if on Java 8).
> > > >>
> > > >>
> > > >> [1]https://github.com/apache/arrow/issues/37739
> > > >> [2]
> > > >>
> > >
> >
> https://newrelic.com/sites/default/files/2023-04/new-relic-2023-state-of-the-java-ecosystem-2023-04-20.pdf
> > > >>
> > > >>
> > > >> -Dane
> > > >>
> > > >>
> > > >> On Thu, Sep 14, 2023 at 11:45 AM Gang Wu  wrote:
> > > >>
> > > >>> Thanks for bringing this up!
> > > >>>
> > > >>> I have two concerns of dropping Java 8 support:
> > > >>> - As a low level library, users have to add specific flags [1] to
> use
> > > >>>  Java 9 and up with Arrow to resolve issues with java.nio. This has
> > > >>>  been annoying for our customers constantly. If this is not
> resolved,
> > > >>>  I would say we may see a lot of complaints in the future.
> > > >>> - It seems that the EOL of Java 8 from Oracle is Dec 2030 [2]. A
> lot
> > > >>>  users will still stay on it for a long time. At least this is true
> > for
> > > >>> our
> > > >>>  customers. So I am afraid we may not upgrade to newer versions
> > > >>>  of Arrow if it no longer supports Java 8.
> > > >>>
> > > >>> [1]
> > https://arrow.apache.org/docs/jav

RE: Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[Alva Bandy]

Hi Antoine,

Thanks for the reply.  

It would be great to get the Swift implementation added to the integration 
test.  I have a task for adding the C Data Interface and I will work on getting 
the integration test running for Swift after that task.  Can we move forward 
with setting up the repo as long as there is a task/issue to ensure the 
integration test will be run against Swift soon or would this be a blocker?

Also, I am not sure about Julia, I have not looked into Julia’s implementation.

Thank you,
Alva Bandy

On 2023/10/10 08:54:30 Antoine Pitrou wrote:
> 
> Hello Alva,
> 
> This is a reasonable request, but it might come with its own drawbacks 
> as well.
> 
> One significant drawback is that adding the Swift implementation to the 
> cross-implementation integration tests will be slightly more complicated.
> It is very important that all Arrow implementations are 
> integration-tested against each other, otherwise we only have a 
> theoretical guarantee that they are compatible. See how this is done here:
> https://arrow.apache.org/docs/dev/format/Integration.html
> 
> Unless I'm mistaken, neither Swift nor Julia are running the integration 
> tests.
> 
> Regards
> 
> Antoine.
> 
> 
> 
> Le 09/10/2023 à 22:26, Alva Bandy a écrit :
> > Hi,
> > 
> > I would like to request a repo for Arrow Swift (similar to arrow-rs).  
> > Swift arrow is currently fully Swift and doesn't leverage the C++ 
> > libraries. One of the goals of Arrow Swift was to provide a fully Swift 
> > impl and splitting them now would help ensure that Swift Arrow stays on 
> > this path.
> > 
> > Also, the Swift Package Manager uses a git repo url to pull down a package. 
> >  This can lead to a large download since the entire arrow repo will be 
> > pulled down just to include Arrow Swift.  It would be great to make this 
> > change before registering Swift Arrow with a Swift registry (such as Swift 
> > Package Registry).
> > 
> > Please let me know if this is possible and if so, what would be the process 
> > going forward.
> > 
> > Thank you,
> > Alva Bandy
> > 
> 

Re: [DISCUSS] Arrow PyCapsule Protocol

[Will Jones]

Hello Arrow devs,

We are winding down discussion and review. I have created a rendered
version of the proposed protocol: [1]

Feel free to add feedback in the PR [2] or on this thread.

Best,
Will Jones

[1]
http://crossbow.voltrondata.com/pr_docs/37797/format/CDataInterface/PyCapsuleInterface.html
[2] https://github.com/apache/arrow/pull/37797

On Fri, Sep 22, 2023 at 8:11 PM Will Jones  wrote:

> Hello Arrow devs,
>
> Based on Joris' idea in [1], I've drafted up a protocol specification for
> PyCapsules holding Arrow C Data / Stream Interface structs. PR: [2]
>
> This has two goals:
>
> 1. Provide a library-independent representation of these structs in Python
> 2. Standardize methods to export objects to these PyCapsules.
>
> This will help projects like nanoarrow be able to import Arrow data safely
> from more than just PyArrow [3]. It would also allow libraries to easily
> interchange Arrow data without requiring going through PyArrow or writing a
> bespoke export function.
>
> I would welcome feedback in the PR [2].
>
> Thanks for your attention,
>
> Will Jones
>
> [1] https://github.com/apache/arrow/issues/34031
> [2] https://github.com/apache/arrow/pull/37797
> [3]
> https://github.com/apache/arrow-nanoarrow/blob/c4816261dc34f5f898b1658359c25b867b1079cd/python/src/nanoarrow/lib.py#L21-L35
>
>

Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[Antoine Pitrou]

Hi Alva,

I'll let others give their opinions on the repo.

Regards

Antoine.


Le 10/10/2023 à 19:25, Alva Bandy a écrit :

Hi Antoine,

Thanks for the reply.

It would be great to get the Swift implementation added to the integration 
test.  I have a task for adding the C Data Interface and I will work on getting 
the integration test running for Swift after that task.  Can we move forward 
with setting up the repo as long as there is a task/issue to ensure the 
integration test will be run against Swift soon or would this be a blocker?

Also, I am not sure about Julia, I have not looked into Julia’s implementation.

Thank you,
Alva Bandy

On 2023/10/10 08:54:30 Antoine Pitrou wrote:


Hello Alva,

This is a reasonable request, but it might come with its own drawbacks
as well.

One significant drawback is that adding the Swift implementation to the
cross-implementation integration tests will be slightly more complicated.
It is very important that all Arrow implementations are
integration-tested against each other, otherwise we only have a
theoretical guarantee that they are compatible. See how this is done here:
https://arrow.apache.org/docs/dev/format/Integration.html

Unless I'm mistaken, neither Swift nor Julia are running the integration
tests.

Regards

Antoine.



Le 09/10/2023 à 22:26, Alva Bandy a écrit :

Hi,

I would like to request a repo for Arrow Swift (similar to arrow-rs).  Swift 
arrow is currently fully Swift and doesn't leverage the C++ libraries. One of 
the goals of Arrow Swift was to provide a fully Swift impl and splitting them 
now would help ensure that Swift Arrow stays on this path.

Also, the Swift Package Manager uses a git repo url to pull down a package.  
This can lead to a large download since the entire arrow repo will be pulled 
down just to include Arrow Swift.  It would be great to make this change before 
registering Swift Arrow with a Swift registry (such as Swift Package Registry).

Please let me know if this is possible and if so, what would be the process 
going forward.

Thank you,
Alva Bandy

Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[Dewey Dunnington]

Hi Alva,

I would encourage you to do whatever will make life more pleasant for
you and other contributors to the Swift Arrow implementation. I have
found development of an Arrow subproject (nanoarrow) in a separate
repository very pleasant. While I don't run integration tests there,
it's not because of any technical limitation (instead of pulling one
repo in your CI job, just pull two).

For the R bindings to Arrow, which do depend on the C++ bindings, we
do have some benefit because Arrow C++ changes that break R tend to
get fixed by the C++ contributor in their PR, rather than that
responsibility always falling on us. That said, it doesn't happen very
often, and we have informally toyed with the idea of moving out of the
monorepo to make it less intimidating for outside contributors.

Cheers,

-dewey

On Tue, Oct 10, 2023 at 2:33 PM Antoine Pitrou  wrote:
>
>
> Hi Alva,
>
> I'll let others give their opinions on the repo.
>
> Regards
>
> Antoine.
>
>
> Le 10/10/2023 à 19:25, Alva Bandy a écrit :
> > Hi Antoine,
> >
> > Thanks for the reply.
> >
> > It would be great to get the Swift implementation added to the integration 
> > test.  I have a task for adding the C Data Interface and I will work on 
> > getting the integration test running for Swift after that task.  Can we 
> > move forward with setting up the repo as long as there is a task/issue to 
> > ensure the integration test will be run against Swift soon or would this be 
> > a blocker?
> >
> > Also, I am not sure about Julia, I have not looked into Julia’s 
> > implementation.
> >
> > Thank you,
> > Alva Bandy
> >
> > On 2023/10/10 08:54:30 Antoine Pitrou wrote:
> >>
> >> Hello Alva,
> >>
> >> This is a reasonable request, but it might come with its own drawbacks
> >> as well.
> >>
> >> One significant drawback is that adding the Swift implementation to the
> >> cross-implementation integration tests will be slightly more complicated.
> >> It is very important that all Arrow implementations are
> >> integration-tested against each other, otherwise we only have a
> >> theoretical guarantee that they are compatible. See how this is done here:
> >> https://arrow.apache.org/docs/dev/format/Integration.html
> >>
> >> Unless I'm mistaken, neither Swift nor Julia are running the integration
> >> tests.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >>
> >> Le 09/10/2023 à 22:26, Alva Bandy a écrit :
> >>> Hi,
> >>>
> >>> I would like to request a repo for Arrow Swift (similar to arrow-rs).  
> >>> Swift arrow is currently fully Swift and doesn't leverage the C++ 
> >>> libraries. One of the goals of Arrow Swift was to provide a fully Swift 
> >>> impl and splitting them now would help ensure that Swift Arrow stays on 
> >>> this path.
> >>>
> >>> Also, the Swift Package Manager uses a git repo url to pull down a 
> >>> package.  This can lead to a large download since the entire arrow repo 
> >>> will be pulled down just to include Arrow Swift.  It would be great to 
> >>> make this change before registering Swift Arrow with a Swift registry 
> >>> (such as Swift Package Registry).
> >>>
> >>> Please let me know if this is possible and if so, what would be the 
> >>> process going forward.
> >>>
> >>> Thank you,
> >>> Alva Bandy
> >>>

Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[Jacob Wujciak-Jens]

+1 on Dewey's sentiment.

With regards to technicalities:
- a PMC member can create the repo via ASF's gitbox (I assume
'arrow-swift'?)
- the repo then needs to be configured using the '.asf.yaml'
  - which merge styles are allowed
  - branch protection rules
  - to which ml should notifications be sent
  - see [1] for more features
- CI
- PR/Issue template
- ...

What is the usual versioning scheme for swift projects and what release
cadence are you planning?

Best
Jacob


On Tue, Oct 10, 2023 at 10:25 PM Dewey Dunnington
 wrote:

> Hi Alva,
>
> I would encourage you to do whatever will make life more pleasant for
> you and other contributors to the Swift Arrow implementation. I have
> found development of an Arrow subproject (nanoarrow) in a separate
> repository very pleasant. While I don't run integration tests there,
> it's not because of any technical limitation (instead of pulling one
> repo in your CI job, just pull two).
>
> For the R bindings to Arrow, which do depend on the C++ bindings, we
> do have some benefit because Arrow C++ changes that break R tend to
> get fixed by the C++ contributor in their PR, rather than that
> responsibility always falling on us. That said, it doesn't happen very
> often, and we have informally toyed with the idea of moving out of the
> monorepo to make it less intimidating for outside contributors.
>
> Cheers,
>
> -dewey
>
> On Tue, Oct 10, 2023 at 2:33 PM Antoine Pitrou  wrote:
> >
> >
> > Hi Alva,
> >
> > I'll let others give their opinions on the repo.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 10/10/2023 à 19:25, Alva Bandy a écrit :
> > > Hi Antoine,
> > >
> > > Thanks for the reply.
> > >
> > > It would be great to get the Swift implementation added to the
> integration test.  I have a task for adding the C Data Interface and I will
> work on getting the integration test running for Swift after that task.
> Can we move forward with setting up the repo as long as there is a
> task/issue to ensure the integration test will be run against Swift soon or
> would this be a blocker?
> > >
> > > Also, I am not sure about Julia, I have not looked into Julia’s
> implementation.
> > >
> > > Thank you,
> > > Alva Bandy
> > >
> > > On 2023/10/10 08:54:30 Antoine Pitrou wrote:
> > >>
> > >> Hello Alva,
> > >>
> > >> This is a reasonable request, but it might come with its own drawbacks
> > >> as well.
> > >>
> > >> One significant drawback is that adding the Swift implementation to
> the
> > >> cross-implementation integration tests will be slightly more
> complicated.
> > >> It is very important that all Arrow implementations are
> > >> integration-tested against each other, otherwise we only have a
> > >> theoretical guarantee that they are compatible. See how this is done
> here:
> > >> https://arrow.apache.org/docs/dev/format/Integration.html
> > >>
> > >> Unless I'm mistaken, neither Swift nor Julia are running the
> integration
> > >> tests.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> > >> Le 09/10/2023 à 22:26, Alva Bandy a écrit :
> > >>> Hi,
> > >>>
> > >>> I would like to request a repo for Arrow Swift (similar to
> arrow-rs).  Swift arrow is currently fully Swift and doesn't leverage the
> C++ libraries. One of the goals of Arrow Swift was to provide a fully Swift
> impl and splitting them now would help ensure that Swift Arrow stays on
> this path.
> > >>>
> > >>> Also, the Swift Package Manager uses a git repo url to pull down a
> package.  This can lead to a large download since the entire arrow repo
> will be pulled down just to include Arrow Swift.  It would be great to make
> this change before registering Swift Arrow with a Swift registry (such as
> Swift Package Registry).
> > >>>
> > >>> Please let me know if this is possible and if so, what would be the
> process going forward.
> > >>>
> > >>> Thank you,
> > >>> Alva Bandy
> > >>>
>

Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[David Li]

I'm -0 on this without more reasoning. I don't think a large download is a 
compelling reason to split the repo, and being in the same repo doesn't mean 
you have to take a dependency on the C++ implementation. (Plus, unless there is 
enough of a community to replicate all the work done for C++ I suspect you will 
want access to Parquet, Dataset, Acero, etc.) 

On Tue, Oct 10, 2023, at 17:24, Jacob Wujciak-Jens wrote:
> +1 on Dewey's sentiment.
>
> With regards to technicalities:
> - a PMC member can create the repo via ASF's gitbox (I assume
> 'arrow-swift'?)
> - the repo then needs to be configured using the '.asf.yaml'
>   - which merge styles are allowed
>   - branch protection rules
>   - to which ml should notifications be sent
>   - see [1] for more features
> - CI
> - PR/Issue template
> - ...
>
> What is the usual versioning scheme for swift projects and what release
> cadence are you planning?
>
> Best
> Jacob
>
>
> On Tue, Oct 10, 2023 at 10:25 PM Dewey Dunnington
>  wrote:
>
>> Hi Alva,
>>
>> I would encourage you to do whatever will make life more pleasant for
>> you and other contributors to the Swift Arrow implementation. I have
>> found development of an Arrow subproject (nanoarrow) in a separate
>> repository very pleasant. While I don't run integration tests there,
>> it's not because of any technical limitation (instead of pulling one
>> repo in your CI job, just pull two).
>>
>> For the R bindings to Arrow, which do depend on the C++ bindings, we
>> do have some benefit because Arrow C++ changes that break R tend to
>> get fixed by the C++ contributor in their PR, rather than that
>> responsibility always falling on us. That said, it doesn't happen very
>> often, and we have informally toyed with the idea of moving out of the
>> monorepo to make it less intimidating for outside contributors.
>>
>> Cheers,
>>
>> -dewey
>>
>> On Tue, Oct 10, 2023 at 2:33 PM Antoine Pitrou  wrote:
>> >
>> >
>> > Hi Alva,
>> >
>> > I'll let others give their opinions on the repo.
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 10/10/2023 à 19:25, Alva Bandy a écrit :
>> > > Hi Antoine,
>> > >
>> > > Thanks for the reply.
>> > >
>> > > It would be great to get the Swift implementation added to the
>> integration test.  I have a task for adding the C Data Interface and I will
>> work on getting the integration test running for Swift after that task.
>> Can we move forward with setting up the repo as long as there is a
>> task/issue to ensure the integration test will be run against Swift soon or
>> would this be a blocker?
>> > >
>> > > Also, I am not sure about Julia, I have not looked into Julia’s
>> implementation.
>> > >
>> > > Thank you,
>> > > Alva Bandy
>> > >
>> > > On 2023/10/10 08:54:30 Antoine Pitrou wrote:
>> > >>
>> > >> Hello Alva,
>> > >>
>> > >> This is a reasonable request, but it might come with its own drawbacks
>> > >> as well.
>> > >>
>> > >> One significant drawback is that adding the Swift implementation to
>> the
>> > >> cross-implementation integration tests will be slightly more
>> complicated.
>> > >> It is very important that all Arrow implementations are
>> > >> integration-tested against each other, otherwise we only have a
>> > >> theoretical guarantee that they are compatible. See how this is done
>> here:
>> > >> https://arrow.apache.org/docs/dev/format/Integration.html
>> > >>
>> > >> Unless I'm mistaken, neither Swift nor Julia are running the
>> integration
>> > >> tests.
>> > >>
>> > >> Regards
>> > >>
>> > >> Antoine.
>> > >>
>> > >>
>> > >>
>> > >> Le 09/10/2023 à 22:26, Alva Bandy a écrit :
>> > >>> Hi,
>> > >>>
>> > >>> I would like to request a repo for Arrow Swift (similar to
>> arrow-rs).  Swift arrow is currently fully Swift and doesn't leverage the
>> C++ libraries. One of the goals of Arrow Swift was to provide a fully Swift
>> impl and splitting them now would help ensure that Swift Arrow stays on
>> this path.
>> > >>>
>> > >>> Also, the Swift Package Manager uses a git repo url to pull down a
>> package.  This can lead to a large download since the entire arrow repo
>> will be pulled down just to include Arrow Swift.  It would be great to make
>> this change before registering Swift Arrow with a Swift registry (such as
>> Swift Package Registry).
>> > >>>
>> > >>> Please let me know if this is possible and if so, what would be the
>> process going forward.
>> > >>>
>> > >>> Thank you,
>> > >>> Alva Bandy
>> > >>>
>>

Re: [Java][Discuss]: consensus for JDK 8 deprecation

[Gang Wu]

I agree that we have to move on. It seems that patch release to
Arrow v14 is a good idea, though I cannot estimate the effort to
backport large features like the new layouts that are currently
being added (e.g. RunEndEncoding, ListView, etc.).

As an Arrow developer, I am always happy to drop JDK 8. My
employer has leveraged Apache Arrow in the internal engine
and depends on Arrow Java in the Java SDK. For end users
who cannot get away with JDK 8, we might need to prepare
different Java SDKs and use features that are available in the
Arrow v14, or let the server side chooses which subset of
features based on the SDK version.

Thanks,
Gang



On Wed, Oct 11, 2023 at 12:40 AM Dane Pitkin 
wrote:

> To summarize the discussion so far:
>
> * Some Arrow Java users are still on JDK 8
> * Arrow v14 is proposed as the final version with JDK 8 support
> * Arrow v14 can support patch releases if necessary for JDK 8 users
> * There is an open question to decide if JDK 11 should be dropped
> simultaneously
>
> Gang Wu, I'm curious what are your thoughts given your initial concerns?
>
> -Dane
>
> On Sat, Oct 7, 2023 at 12:00 AM Jacob Wujciak-Jens
>  wrote:
>
> > From a release engineer perspective (without java knowledge) I agree with
> > Micah, I'd rather make a patch release for an older version if needed but
> > modernize the codebase and simplify CI!
> >
> >
> > On Sat, Oct 7, 2023 at 5:27 AM Micah Kornfield 
> > wrote:
> >
> > > I think given the stability of Arrow Java, dropping support probably
> > makes
> > > sense.  If a bug comes up or consumers really need to new features we
> can
> > > always make a patch release of an older version.
> > >
> > > On Thu, Oct 5, 2023 at 3:13 PM Dane Pitkin
>  > >
> > > wrote:
> > >
> > > > I also learned today that Apache Spark has dropped support for Java 8
> > and
> > > > 11 for their next release (v4.0)[1]. Should we consider dropping Java
> > 11
> > > as
> > > > well?
> > > >
> > > > [1]https://github.com/apache/spark/pull/43005
> > > >
> > > > -Dane
> > > >
> > > > On Thu, Oct 5, 2023 at 3:30 PM Dane Pitkin 
> > wrote:
> > > >
> > > > > I created a GH issue[1] proposing the removal of Java 8 support. It
> > > > > would target the Arrow v15 release (~Jan 2024).
> > > > >
> > > > > IMO it would be in the best interest of the project for two major
> > > > reasons:
> > > > > 1. Unblock the Java Platform Module System (JPMS)[2]
> implementation.
> > > > > 2. Unblock Arrow from upgrading dependencies that no longer support
> > > Java
> > > > > 8. (See [1] for examples)
> > > > >
> > > > > Since Arrow Java has been quite stable, will Java 8 users be okay
> > with
> > > > > pinning Arrow to the last supported release (v14) if the Arrow
> > project
> > > > > ultimately decides to remove Java 8 support?
> > > > >
> > > > >
> > > > > [1]https://github.com/apache/arrow/issues/38051
> > > > > [2]https://en.wikipedia.org/wiki/Java_Platform_Module_System
> > > > >
> > > > > -Dane
> > > > >
> > > > > On Fri, Sep 15, 2023 at 12:26 PM Dane Pitkin  >
> > > > wrote:
> > > > >
> > > > >> - As a low level library, users have to add specific flags to use
> > > > >>>  Java 9 and up with Arrow to resolve issues with java.nio. This
> has
> > > > >>>  been annoying for our customers constantly. If this is not
> > resolved,
> > > > >>>  I would say we may see a lot of complaints in the future.
> > > > >>>
> > > > >> I filed issue 37739[1] to track this, but it sounds like this
> can't
> > be
> > > > >> changed until Java 21 or 24.
> > > > >>
> > > > >> - It seems that the EOL of Java 8 from Oracle is Dec 2030 [2]. A
> lot
> > > > >>>  users will still stay on it for a long time. At least this is
> true
> > > for
> > > > >>> our
> > > > >>>  customers. So I am afraid we may not upgrade to newer versions
> > > > >>>  of Arrow if it no longer supports Java 8.
> > > > >>>
> > > > >> Java 8 does have a long Extended Support timeline, but a recent
> > > > >> report shows Java 11 increasing in adoption vs Java 8. "More than
> > 56%
> > > of
> > > > >> applications are now using Java 11 in production (up from 48% in
> > 2022
> > > > and
> > > > >> 11% in 2020). Java 8 is a close second with nearly 33% of
> > applications
> > > > >> using it in production (down from 46% in 2022)."[2]
> > > > >> I expect the Java ecosystem will find a way to move on from Java 8
> > > much
> > > > >> sooner than 2030, meaning many of Arrow's dependencies could drop
> > > > support
> > > > >> for Java 8 before then. At this point, Arrow may be forced to
> > support
> > > a
> > > > >> higher minimum Java version.
> > > > >>
> > > > >> That being said, it's hard to argue against real use cases. I'd be
> > > > >> curious to hear what Java version other users of Arrow are using
> > (and
> > > if
> > > > >> there is a timeline to upgrade if on Java 8).
> > > > >>
> > > > >>
> > > > >> [1]https://github.com/apache/arrow/issues/37739
> > > > >> [2]
> > > > >>
> > > >
> > >
> >
> https://newrelic.com/sites/default/files/2023-04/new-re

Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

[Sutou Kouhei]

Hi,

I'm OK with this if we can keep following the Apache Way:

* https://www.apache.org/theapacheway/
* https://theapacheway.com/


Can we discuss how to develop with separated arrow-swift
repository after we re-read the above documentations?

FYI: Here are discussions for arrow-rs and arrow-julia:

* [DISCUSS] [Rust] Move Rust components to new repos and process
  https://lists.apache.org/thread/w3dr1vlt9115r3x9m7bprmo4zpnog483
* [DISCUSS][Julia] How to restart at apache/arrow-julia?
  https://lists.apache.org/thread/9hy387r8hvfrtnqdvgyszj3mf53mdr96
* Future of the Julia arrow implementation
  https://github.com/apache/arrow-julia/issues/284


Thanks,
-- 
kou

In 
 

  "[DISCUSS][Swift] repo for swift similar to arrow-rs" on Mon, 9 Oct 2023 
20:26:53 +,
  Alva Bandy  wrote:

> Hi,
> 
> I would like to request a repo for Arrow Swift (similar to arrow-rs).  Swift 
> arrow is currently fully Swift and doesn't leverage the C++ libraries. One of 
> the goals of Arrow Swift was to provide a fully Swift impl and splitting them 
> now would help ensure that Swift Arrow stays on this path.
> 
> Also, the Swift Package Manager uses a git repo url to pull down a package.  
> This can lead to a large download since the entire arrow repo will be pulled 
> down just to include Arrow Swift.  It would be great to make this change 
> before registering Swift Arrow with a Swift registry (such as Swift Package 
> Registry).
> 
> Please let me know if this is possible and if so, what would be the process 
> going forward.
> 
> Thank you,
> Alva Bandy

Re: [Java][Discuss]: consensus for JDK 8 deprecation

[Jacob Wujciak-Jens]

> I cannot estimate the effort to backport large features like the new
layouts that are currently being added (e.g. RunEndEncoding, ListView,
etc.).

In my mind we are only talking about patch releases for security fixes or
similarly critical issues as otherwise the effort to maintain 'v14' (but
actually arrow-latest) would surely overshadow any gains made by
deprecating jdk 8?

On Wed, Oct 11, 2023 at 3:31 AM Gang Wu  wrote:

> I agree that we have to move on. It seems that patch release to
> Arrow v14 is a good idea, though I cannot estimate the effort to
> backport large features like the new layouts that are currently
> being added (e.g. RunEndEncoding, ListView, etc.).
>
> As an Arrow developer, I am always happy to drop JDK 8. My
> employer has leveraged Apache Arrow in the internal engine
> and depends on Arrow Java in the Java SDK. For end users
> who cannot get away with JDK 8, we might need to prepare
> different Java SDKs and use features that are available in the
> Arrow v14, or let the server side chooses which subset of
> features based on the SDK version.
>
> Thanks,
> Gang
>
>
>
> On Wed, Oct 11, 2023 at 12:40 AM Dane Pitkin  >
> wrote:
>
> > To summarize the discussion so far:
> >
> > * Some Arrow Java users are still on JDK 8
> > * Arrow v14 is proposed as the final version with JDK 8 support
> > * Arrow v14 can support patch releases if necessary for JDK 8 users
> > * There is an open question to decide if JDK 11 should be dropped
> > simultaneously
> >
> > Gang Wu, I'm curious what are your thoughts given your initial concerns?
> >
> > -Dane
> >
> > On Sat, Oct 7, 2023 at 12:00 AM Jacob Wujciak-Jens
> >  wrote:
> >
> > > From a release engineer perspective (without java knowledge) I agree
> with
> > > Micah, I'd rather make a patch release for an older version if needed
> but
> > > modernize the codebase and simplify CI!
> > >
> > >
> > > On Sat, Oct 7, 2023 at 5:27 AM Micah Kornfield 
> > > wrote:
> > >
> > > > I think given the stability of Arrow Java, dropping support probably
> > > makes
> > > > sense.  If a bug comes up or consumers really need to new features we
> > can
> > > > always make a patch release of an older version.
> > > >
> > > > On Thu, Oct 5, 2023 at 3:13 PM Dane Pitkin
> >  > > >
> > > > wrote:
> > > >
> > > > > I also learned today that Apache Spark has dropped support for
> Java 8
> > > and
> > > > > 11 for their next release (v4.0)[1]. Should we consider dropping
> Java
> > > 11
> > > > as
> > > > > well?
> > > > >
> > > > > [1]https://github.com/apache/spark/pull/43005
> > > > >
> > > > > -Dane
> > > > >
> > > > > On Thu, Oct 5, 2023 at 3:30 PM Dane Pitkin 
> > > wrote:
> > > > >
> > > > > > I created a GH issue[1] proposing the removal of Java 8 support.
> It
> > > > > > would target the Arrow v15 release (~Jan 2024).
> > > > > >
> > > > > > IMO it would be in the best interest of the project for two major
> > > > > reasons:
> > > > > > 1. Unblock the Java Platform Module System (JPMS)[2]
> > implementation.
> > > > > > 2. Unblock Arrow from upgrading dependencies that no longer
> support
> > > > Java
> > > > > > 8. (See [1] for examples)
> > > > > >
> > > > > > Since Arrow Java has been quite stable, will Java 8 users be okay
> > > with
> > > > > > pinning Arrow to the last supported release (v14) if the Arrow
> > > project
> > > > > > ultimately decides to remove Java 8 support?
> > > > > >
> > > > > >
> > > > > > [1]https://github.com/apache/arrow/issues/38051
> > > > > > [2]https://en.wikipedia.org/wiki/Java_Platform_Module_System
> > > > > >
> > > > > > -Dane
> > > > > >
> > > > > > On Fri, Sep 15, 2023 at 12:26 PM Dane Pitkin <
> d...@voltrondata.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> - As a low level library, users have to add specific flags to
> use
> > > > > >>>  Java 9 and up with Arrow to resolve issues with java.nio. This
> > has
> > > > > >>>  been annoying for our customers constantly. If this is not
> > > resolved,
> > > > > >>>  I would say we may see a lot of complaints in the future.
> > > > > >>>
> > > > > >> I filed issue 37739[1] to track this, but it sounds like this
> > can't
> > > be
> > > > > >> changed until Java 21 or 24.
> > > > > >>
> > > > > >> - It seems that the EOL of Java 8 from Oracle is Dec 2030 [2]. A
> > lot
> > > > > >>>  users will still stay on it for a long time. At least this is
> > true
> > > > for
> > > > > >>> our
> > > > > >>>  customers. So I am afraid we may not upgrade to newer versions
> > > > > >>>  of Arrow if it no longer supports Java 8.
> > > > > >>>
> > > > > >> Java 8 does have a long Extended Support timeline, but a recent
> > > > > >> report shows Java 11 increasing in adoption vs Java 8. "More
> than
> > > 56%
> > > > of
> > > > > >> applications are now using Java 11 in production (up from 48% in
> > > 2022
> > > > > and
> > > > > >> 11% in 2020). Java 8 is a close second with nearly 33% of
> > > applications
> > > > > >> using it in production (down from 46% in 2022)."[2]
> > > > > >> I

Re: [Java][Discuss]: consensus for JDK 8 deprecation

[Fokko Driesprong]

Hey everyone,

Great to bring this up. I can speak for the Iceberg community, and we
expect to support Java 8 for a long time there (unfortunately). Let me go
over some of the arguments here.

Java 8 does have a long Extended Support timeline, but a recent
> report shows Java 11 increasing in adoption vs Java 8. "More than 56% of
> applications are now using Java 11 in production (up from 48% in 2022 and
> 11% in 2020). Java 8 is a close second with nearly 33% of applications
> using it in production (down from 46% in 2022)."[2]


I think this is skewed, mostly because it is easy to upgrade a Spring
application to the latest version from Java, but if you're tied to the
Hadoop ecosystems, things are moving slowly.

2. Unblock Arrow from upgrading dependencies that no longer support Java 8.


Following the links, I noticed the dependencies are around tests (Mockito)
and static error checking (error-prone). Those sound like nice to have to
me.

An example; Apache Thrift dropped Java 8 a while ago but added the support
again since it was breaking the ecosystem
. Thrift is used in Parquet,
and in Parquet we cannot yet drop Java 8 support. I think of Arrow as a
low-level library, like Thirft and Iceberg, and I think it makes sense to
serve an as wide audience as possible (within reasonable bounds of course).

I would at least wait until the release of Spark 4. My experience is that
nobody is really eager to do backporting to older versions, and for me, I
still think the gains of dropping Java 8 support are not that big.

Kind regards,
Fokko Driesprong

Op wo 11 okt 2023 om 06:05 schreef Jacob Wujciak-Jens
:

> > I cannot estimate the effort to backport large features like the new
> layouts that are currently being added (e.g. RunEndEncoding, ListView,
> etc.).
>
> In my mind we are only talking about patch releases for security fixes or
> similarly critical issues as otherwise the effort to maintain 'v14' (but
> actually arrow-latest) would surely overshadow any gains made by
> deprecating jdk 8?
>
> On Wed, Oct 11, 2023 at 3:31 AM Gang Wu  wrote:
>
> > I agree that we have to move on. It seems that patch release to
> > Arrow v14 is a good idea, though I cannot estimate the effort to
> > backport large features like the new layouts that are currently
> > being added (e.g. RunEndEncoding, ListView, etc.).
> >
> > As an Arrow developer, I am always happy to drop JDK 8. My
> > employer has leveraged Apache Arrow in the internal engine
> > and depends on Arrow Java in the Java SDK. For end users
> > who cannot get away with JDK 8, we might need to prepare
> > different Java SDKs and use features that are available in the
> > Arrow v14, or let the server side chooses which subset of
> > features based on the SDK version.
> >
> > Thanks,
> > Gang
> >
> >
> >
> > On Wed, Oct 11, 2023 at 12:40 AM Dane Pitkin
>  > >
> > wrote:
> >
> > > To summarize the discussion so far:
> > >
> > > * Some Arrow Java users are still on JDK 8
> > > * Arrow v14 is proposed as the final version with JDK 8 support
> > > * Arrow v14 can support patch releases if necessary for JDK 8 users
> > > * There is an open question to decide if JDK 11 should be dropped
> > > simultaneously
> > >
> > > Gang Wu, I'm curious what are your thoughts given your initial
> concerns?
> > >
> > > -Dane
> > >
> > > On Sat, Oct 7, 2023 at 12:00 AM Jacob Wujciak-Jens
> > >  wrote:
> > >
> > > > From a release engineer perspective (without java knowledge) I agree
> > with
> > > > Micah, I'd rather make a patch release for an older version if needed
> > but
> > > > modernize the codebase and simplify CI!
> > > >
> > > >
> > > > On Sat, Oct 7, 2023 at 5:27 AM Micah Kornfield <
> emkornfi...@gmail.com>
> > > > wrote:
> > > >
> > > > > I think given the stability of Arrow Java, dropping support
> probably
> > > > makes
> > > > > sense.  If a bug comes up or consumers really need to new features
> we
> > > can
> > > > > always make a patch release of an older version.
> > > > >
> > > > > On Thu, Oct 5, 2023 at 3:13 PM Dane Pitkin
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > I also learned today that Apache Spark has dropped support for
> > Java 8
> > > > and
> > > > > > 11 for their next release (v4.0)[1]. Should we consider dropping
> > Java
> > > > 11
> > > > > as
> > > > > > well?
> > > > > >
> > > > > > [1]https://github.com/apache/spark/pull/43005
> > > > > >
> > > > > > -Dane
> > > > > >
> > > > > > On Thu, Oct 5, 2023 at 3:30 PM Dane Pitkin  >
> > > > wrote:
> > > > > >
> > > > > > > I created a GH issue[1] proposing the removal of Java 8
> support.
> > It
> > > > > > > would target the Arrow v15 release (~Jan 2024).
> > > > > > >
> > > > > > > IMO it would be in the best interest of the project for two
> major
> > > > > > reasons:
> > > > > > > 1. Unblock the Java Platform Module System (JPMS)[2]
> > > implementation.
> > > > > > > 2. Unblock Arrow from upgrading dependencies that no longer
> > support
> > > >