Hi, I think that we can say the Rust migration is complete once we merge https://github.com/apache/arrow/pull/10096. But it's a good time to think about the Julia migration.
Jacob, here is the Rust's new development process: https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit# (It seems that an anonymous user deleted a part of it accidentally.) Do you want to use the same development process as the Rust's one? Do you have any item you want to change? Thanks, -- kou In <cak7z5t96rapgltnasg6v+hxd7qqreuhxa5xzt7wcdv_gelh...@mail.gmail.com> "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04 -0700, Micah Kornfield <emkornfi...@gmail.com> wrote: > It seems the Rust migration is now complete. Do we want to wait to iron > out the other potential issues? > > I think the outstanding ones might be: > 1. Issue management > 2. Integration testing > > -Micah > > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei <k...@clear-code.com> wrote: > >> Hi Jacob, >> >> OK. Here is my plan: >> >> 1. We wait for the Rust's move to complete >> 2. We use a process similar to the Rust's move >> >> >> Thanks, >> -- >> kou >> >> In <cakyxbqrt6yahesuqg8beuh6u58smc95jtgejej2kuy0zrgy...@mail.gmail.com> >> "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021 08:37:41 >> -0600, >> Jacob Quinn <quinn.jac...@gmail.com> wrote: >> >> > Thank you kou! I appreciate the help. I'm happy to do whatever is >> required >> > to facilitate the moving/donating process from JuliaData/Arrow.jl to >> > apache/arrow-julia. >> > >> > -Jacob >> > >> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei <k...@clear-code.com> wrote: >> > >> >> Hi Jacob, >> >> >> >> I, a PMC member, talked to Kenta Murata, a commiter and a >> >> Julia user, about this. >> >> >> >> We support that you and Julia folks work on >> >> arrow/arrow-julia until we have enough PMC members from >> >> Julia folks. For example, we'll help IP clearance process to >> >> import the latest JuliaData/Arrow.js changes to apache/ and >> >> we'll start voting on Julia package release. >> >> >> >> >> >> Thanks, >> >> -- >> >> kou >> >> >> >> In <CAKyXBQo2=+NfS=hX6nEDPmMno-bSuXMR+=d-heskgn2mm57...@mail.gmail.com> >> >> "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021 >> 23:06:27 >> >> -0600, >> >> Jacob Quinn <quinn.jac...@gmail.com> wrote: >> >> >> >> > Micah/Wes, >> >> > >> >> > Yes, I've been following the rust proposal thread with great >> interest. I >> >> do >> >> > think that provides a great path forward: transferring the >> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the >> >> > "package history" technical challenges that in part led to the current >> >> > setup and concerns. I think being able to utilize github issues would >> >> also >> >> > be great; as I've mentioned elsewhere, it's much more >> >> traditional/expected >> >> > in the Julia ecosystem. >> >> > >> >> > I think the package could retain an independent versioning scheme. The >> >> >> additional process would be voting on release candidates. If the >> Julia >> >> >> folks want to try again and move development to a new, Julia-specific >> >> >> apache/* repository and apply the ASF governance to the project, the >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In some >> >> >> code donations / IP clearance, the contributors for the donated code >> >> >> become committers as part of the transaction. >> >> >> >> >> > >> >> > These all sound great and would greatly facilitate a better >> integration >> >> > under ASF governance. These points definitely resolve my main >> concerns. >> >> > >> >> > As I commented on the rust thread, I'm mostly interested in the >> future of >> >> > integration testing for rust/julia if they are split out into separate >> >> > repos. In the current Julia implementation, we have all the code to >> read >> >> > arrow json, and I just hand-generated the integration test data and >> >> > committed them in the repo itself, but it doesn't interface with other >> >> > languages (just reads arrow json, produces arrow file, reads arrow >> file, >> >> > compares w/ original arrow json). I'm happy to help work on the >> details >> >> of >> >> > what that looks like and pilot some solutions. I think with a solid >> >> > inter-repo integration testing framework, we can keep a strong sync >> >> between >> >> > projects. >> >> > >> >> > -Jacob >> >> > >> >> > >> >> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney <wesmck...@gmail.com> >> >> wrote: >> >> > >> >> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield < >> emkornfi...@gmail.com> >> >> >> wrote: >> >> >> > >> >> >> > > >> >> >> > > Ok, I've had a chance to discuss with a few other Julia >> developers >> >> and >> >> >> > > review various options. I think it's best to drop the Julia code >> >> from >> >> >> the >> >> >> > > physical apache/arrow repo. The extra overhead on development, >> >> release >> >> >> > > process, and user issue reporting and PR contributing are too >> much >> >> in >> >> >> > > addition to the technical challenges that we never resolved >> >> involving >> >> >> > > including the past Arrow.jl release version git trees in the >> >> >> apache/arrow >> >> >> > > repo. >> >> >> > >> >> >> > >> >> >> > Hi Jacob, >> >> >> > It seems you are on the new thread discussing a proposal for >> changing >> >> >> > Rust's development model. Would the proposal [1] address most of >> >> these >> >> >> > concerns if Julia was set up in the same way? >> >> >> > >> >> >> > It seems in the short term the stickiest point would be committer >> >> access >> >> >> > to the new repos, and I suppose the release mechanics still might >> be >> >> >> > challenging? >> >> >> >> >> >> I think the package could retain an independent versioning scheme. >> The >> >> >> additional process would be voting on release candidates. If the >> Julia >> >> >> folks want to try again and move development to a new, Julia-specific >> >> >> apache/* repository and apply the ASF governance to the project, the >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In some >> >> >> code donations / IP clearance, the contributors for the donated code >> >> >> become committers as part of the transaction. >> >> >> >> >> >> > >> >> >> > Thanks, >> >> >> > Micah >> >> >> > >> >> >> > [1] >> >> >> > >> >> >> >> >> >> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit >> >> >> > >> >> >> > On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney <wesmck...@gmail.com> >> >> wrote: >> >> >> > >> >> >> > > I went back and read the mailing list discussions from September >> >> about >> >> >> > > the donation and I would say there was not a clear enough >> statement >> >> >> > > from us about what the donation and IP clearance meant as far as >> the >> >> >> > > future of the Julia codebase. This is partly our fault — we have >> >> taken >> >> >> > > in 9 other code donations over the last 5 years, and in all cases >> >> the >> >> >> > > developers understood that they were to move their process to the >> >> >> > > Arrow repositories and communications channels. >> >> >> > > >> >> >> > > It did not occur to me at all that the code that you were >> putting in >> >> >> > > the Arrow repository would get treated like a read-only fork that >> >> you >> >> >> > > update periodically. If I had realized that, we wouldn't be in >> this >> >> >> > > situation. >> >> >> > > >> >> >> > > As a reminder about what Arrow and the ASF are all about: >> Community >> >> >> > > over Code. We think that building a collaborative, open community >> >> that >> >> >> > > works and plans together in public, makes decisions based on >> >> consensus >> >> >> > > with clear meritocratic ("doers decide") governance is the best >> way >> >> to >> >> >> > > build this project. The concerns that you have around the timing >> and >> >> >> > > frequency of releases for the Julia codebase are in my mind easy >> to >> >> >> > > resolve, and if you had indicated that having a customized >> process >> >> for >> >> >> > > Julia releases was a condition for your joining the community >> >> >> > > wholeheartedly, we would have been happy to help. I think that >> the >> >> >> > > benefits of common CI/CD infrastructure and opportunities to >> build >> >> >> > > deeper integrations between the Julia implementation and the >> other >> >> >> > > implementations (imagine... Julia kernels running in DataFusion?) >> >> >> > > would outweigh the sense of "loss of control" from developing >> >> within a >> >> >> > > larger project. >> >> >> > > >> >> >> > > On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn < >> quinn.jac...@gmail.com >> >> > >> >> >> > > wrote: >> >> >> > > > >> >> >> > > > Responses inline below: >> >> >> > > > >> >> >> > > > On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão < >> >> >> > > > jorgecarlei...@gmail.com> wrote: >> >> >> > > > >> >> >> > > > > Hi, >> >> >> > > > > >> >> >> > > > > > you all did not attempt to work in the community for any >> >> >> meaningful >> >> >> > > > > amount of time and >> >> >> > > > > are choosing not to try based on the perception that it will >> >> create >> >> >> > > > > unacceptable overhead for you >> >> >> > > > > >> >> >> > > > > It is not self-evident to me that Julia's community was >> >> >> sufficiently >> >> >> > > > > informed about what they >> >> >> > > > > had to give in in terms of process and release management >> when >> >> >> merging >> >> >> > > / >> >> >> > > > > donating. >> >> >> > > > > >> >> >> > > > >> >> >> > > > Yes, it was pretty unclear what the process was if we needed >> to do >> >> >> any >> >> >> > > kind >> >> >> > > > of patch release. I know that has been sorted out better >> recently, >> >> >> but >> >> >> > > back >> >> >> > > > in November, it didn't really seem like an option (i.e. >> >> independent >> >> >> > > > language patch releases). >> >> >> > > > >> >> >> > > > >> >> >> > > > > IMO this is a plausible explanation as to why the donation >> was >> >> >> made and >> >> >> > > > > then later abandoned. >> >> >> > > > > >> >> >> > > > > >> >> >> > > > I'll just note that the "abandonment" can only be a perception >> >> from >> >> >> the >> >> >> > > > apache/arrow side of things, but as I mentioned above, I also >> >> tried >> >> >> to >> >> >> > > > clearly state in the julia/Arrow/README that the development >> >> process >> >> >> > > would >> >> >> > > > continue with the JuliaData/Arrow.jl repo as the main "dev" >> >> branch, >> >> >> with >> >> >> > > > changes being upstreamed to the apache/arrow repo, which was >> >> followed >> >> >> > > > through, having an upstream of commits right before the 3.0.0 >> >> >> release, >> >> >> > > and >> >> >> > > > I was planning on doing the same soon for the 4.0.0 release. >> That >> >> is >> >> >> to >> >> >> > > > say, the Julia implementation has continued progressing forward >> >> quite >> >> >> > > > rapidly, IMO, but I can see that perhaps apache/arrow repo >> members >> >> >> may >> >> >> > > have >> >> >> > > > viewed it as "abandoned". >> >> >> > > > >> >> >> > > > >> >> >> > > > > I do not fully understand why the pain points Jacob mentioned >> >> were >> >> >> not >> >> >> > > > > brought up to the mailing list sooner, though. >> >> >> > > > > >> >> >> > > > >> >> >> > > > To be honest and frank, I didn't have pain points with the >> >> >> development >> >> >> > > > process I outlined when the code was donated and as stated in >> the >> >> >> README. >> >> >> > > > That was the process that made the donation possible and I >> >> imagined >> >> >> would >> >> >> > > > work well going forward, and has, until this thread started >> and it >> >> >> was >> >> >> > > > pointed out that this process isn't viable. The pain points >> were >> >> >> > > discussed >> >> >> > > > with the initial code donation, but in my mind were resolved >> with >> >> the >> >> >> > > > development process that was decided upon. >> >> >> > > > >> >> >> > > > >> >> >> > > > > This made us unable to potentially take corrective measures. >> I >> >> >> think >> >> >> > > that >> >> >> > > > > this is why everyone was taken a bit by surprise with this. >> >> >> > > > > >> >> >> > > > > Best, >> >> >> > > > > Jorge >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney < >> >> wesmck...@gmail.com> >> >> >> > > wrote: >> >> >> > > > > >> >> >> > > > > > hi Jacob — sorry to hear that. It's a bummer that you all >> did >> >> not >> >> >> > > > > > attempt to work in the community for any meaningful amount >> of >> >> >> time >> >> >> > > and >> >> >> > > > > > are choosing not to try based on the perception that it >> will >> >> >> create >> >> >> > > > > > unacceptable overhead for you. I believe the benefits would >> >> >> outweigh >> >> >> > > > > > the costs, but I suppose we will have to agree to disagree. >> >> >> > > > > > >> >> >> > > > > > Can you prepare a pull request to do the requisite >> repository >> >> >> > > surgery? >> >> >> > > > > > I hope the development goes well in the future and look >> >> forward >> >> >> to >> >> >> > > > > > seeing folks from the Julia ecosystem engaged here on >> growing >> >> the >> >> >> > > > > > Arrow ecosystem. >> >> >> > > > > > >> >> >> > > > > > Thanks, >> >> >> > > > > > Wes >> >> >> > > > > > >> >> >> > > > > > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn < >> >> >> quinn.jac...@gmail.com> >> >> >> > > > > wrote: >> >> >> > > > > > > >> >> >> > > > > > > Ok, I've had a chance to discuss with a few other Julia >> >> >> developers >> >> >> > > and >> >> >> > > > > > > review various options. I think it's best to drop the >> Julia >> >> >> code >> >> >> > > from >> >> >> > > > > the >> >> >> > > > > > > physical apache/arrow repo. The extra overhead on >> >> development, >> >> >> > > release >> >> >> > > > > > > process, and user issue reporting and PR contributing are >> >> too >> >> >> much >> >> >> > > in >> >> >> > > > > > > addition to the technical challenges that we never >> resolved >> >> >> > > involving >> >> >> > > > > > > including the past Arrow.jl release version git trees in >> the >> >> >> > > > > apache/arrow >> >> >> > > > > > > repo. >> >> >> > > > > > > >> >> >> > > > > > > We're still very much committed to working on the Julia >> >> >> > > implementation >> >> >> > > > > > and >> >> >> > > > > > > participating in the broader arrow community. I've >> enjoyed >> >> >> > > following >> >> >> > > > > the >> >> >> > > > > > > user/dev mailing lists and will continue to do so. We >> >> monitor >> >> >> > > format >> >> >> > > > > > > proposals and try to implement new functionality as >> quickly >> >> as >> >> >> > > > > possible. >> >> >> > > > > > We >> >> >> > > > > > > got the initial arrow flight proto code generated just >> last >> >> >> night >> >> >> > > in >> >> >> > > > > > fact. >> >> >> > > > > > > I'd still like to explore official integration with the >> >> archery >> >> >> > > test >> >> >> > > > > > suite >> >> >> > > > > > > to solidify the Julia implementation with integration >> >> tests; I >> >> >> > > think >> >> >> > > > > that >> >> >> > > > > > > would be very valuable for long-term confidence in the >> >> >> > > cross-language >> >> >> > > > > > > support of the Julia implementation. >> >> >> > > > > > > >> >> >> > > > > > > We realize one of the main implications will probably be >> >> >> dropping >> >> >> > > Julia >> >> >> > > > > > > from the list of "official implementations". We're >> >> encouraged >> >> >> by >> >> >> > > the >> >> >> > > > > many >> >> >> > > > > > > users who have already started using the Julia >> >> implementation >> >> >> and >> >> >> > > will >> >> >> > > > > > > strive to maintain a high rate of issue responsiveness >> and >> >> >> feature >> >> >> > > > > > > development to maintain project confidence. If there's a >> >> >> > > possibility of >> >> >> > > > > > > being included somewhere as an "unofficial" or >> >> "semi-official" >> >> >> > > > > > > implementation, we'd love to still be bundled with the >> >> broader >> >> >> > > arrow >> >> >> > > > > > > project somehow, like, for example, showing how Julia >> >> >> integrates >> >> >> > > with >> >> >> > > > > the >> >> >> > > > > > > archery test suite, once the work there is done. >> >> >> > > > > > > >> >> >> > > > > > > Best, >> >> >> > > > > > > >> >> >> > > > > > > -Jacob >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > > On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney < >> >> >> wesmck...@gmail.com> >> >> >> > > > > > wrote: >> >> >> > > > > > > >> >> >> > > > > > > > Also, on the issue that there are no Julia-focused PMC >> >> >> members — >> >> >> > > note >> >> >> > > > > > > > that I helped the JavaScript folks make their own >> >> independent >> >> >> > > > > releases >> >> >> > > > > > > > for quite a while: called the votes (e.g. [1]), helped >> get >> >> >> > > people to >> >> >> > > > > > > > verify and vote on the releases. After a time, it was >> >> >> decided to >> >> >> > > stop >> >> >> > > > > > > > releasing independently because there wasn't enough >> >> >> development >> >> >> > > > > > > > activity to justify it. >> >> >> > > > > > > > >> >> >> > > > > > > > [1]: >> >> >> > > https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html >> >> >> > > > > > > > >> >> >> > > > > > > > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney < >> >> >> > > wesmck...@gmail.com> >> >> >> > > > > > wrote: >> >> >> > > > > > > > > >> >> >> > > > > > > > > hi Jacob, >> >> >> > > > > > > > > >> >> >> > > > > > > > > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn < >> >> >> > > > > quinn.jac...@gmail.com> >> >> >> > > > > > > > wrote: >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > I can comment as the primary apache arrow liaison >> for >> >> the >> >> >> > > > > Arrow.jl >> >> >> > > > > > > > > > repository and original code donator. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > I apologize for the "surprise", but I commented a >> few >> >> >> times >> >> >> > > in >> >> >> > > > > > various >> >> >> > > > > > > > > > places and put a snippet in the README >> >> >> > > > > > > > > > < >> >> >> > > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > >> >> >> >> >> >> https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository >> >> >> > > > > > > > > >> >> >> > > > > > > > > > about >> >> >> > > > > > > > > > the approach I wanted to take w/ the Julia >> >> >> implementation in >> >> >> > > > > terms >> >> >> > > > > > of >> >> >> > > > > > > > > > keeping the JuliaData/Arrow.jl repository as a "dev >> >> >> branch" >> >> >> > > of >> >> >> > > > > > sorts >> >> >> > > > > > > > of the >> >> >> > > > > > > > > > apache/arrow code, upstreaming changes >> periodically. >> >> >> There's >> >> >> > > > > even a >> >> >> > > > > > > > script >> >> >> > > > > > > > > > < >> >> >> > > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > >> >> >> >> >> >> https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl >> >> >> > > > > > > > > >> >> >> > > > > > > > > > I wrote to mostly automate this upstreaming. I >> realize >> >> >> now >> >> >> > > that I >> >> >> > > > > > > > didn't >> >> >> > > > > > > > > > consider the "Arrow PMC" position on this kind of >> >> setup >> >> >> or >> >> >> > > seek >> >> >> > > > > to >> >> >> > > > > > > > affirm >> >> >> > > > > > > > > > that it would be ok to approach things like this. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > The reality is that Julia users are very engrained >> to >> >> >> expect >> >> >> > > > > Julia >> >> >> > > > > > > > packages >> >> >> > > > > > > > > > to live in a single stand-alone github repo, where >> >> issues >> >> >> > > can be >> >> >> > > > > > > > opened, >> >> >> > > > > > > > > > and pull requests are welcome. It was hard and >> still >> >> is >> >> >> hard >> >> >> > > to >> >> >> > > > > > imagine >> >> >> > > > > > > > > > "turning that off", since I believe we would lose a >> >> lot >> >> >> of >> >> >> > > > > > valuable bug >> >> >> > > > > > > > > > reports and first-time contributions. This isn't >> >> >> necessarily >> >> >> > > any >> >> >> > > > > > fault >> >> >> > > > > > > > of >> >> >> > > > > > > > > > how the bug report/contribution process is handled >> for >> >> >> the >> >> >> > > arrow >> >> >> > > > > > > > project >> >> >> > > > > > > > > > overall, though I'm also aware that there's a >> desire >> >> to >> >> >> make >> >> >> > > it >> >> >> > > > > > easier >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > < >> >> >> > > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > >> >> >> >> >> >> https://lists.apache.org/x/thread.html/r8817dfba08ef8daa210956db69d513fd27b7a751d28fb8f27e39cc7e@%3Cdev.arrow.apache.org%3E >> >> >> > > > > > > > > >> >> >> > > > > > > > > > and >> >> >> > > > > > > > > > it currently requires more and different effort >> than >> >> >> Julia >> >> >> > > users >> >> >> > > > > > are >> >> >> > > > > > > > used >> >> >> > > > > > > > > > to. I think it's more from how open, welcoming, and >> >> how >> >> >> > > strong >> >> >> > > > > the >> >> >> > > > > > > > culture >> >> >> > > > > > > > > > is in Julia around encouraging community >> contributions >> >> >> and >> >> >> > > the >> >> >> > > > > > tight >> >> >> > > > > > > > > > integration with github and its open-source project >> >> >> > > management >> >> >> > > > > > tools. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > Well, we are on track to having 1000 different people >> >> >> > > contribute to >> >> >> > > > > > > > > the project and have over 12,000 issues, so I don't >> >> think >> >> >> > > there is >> >> >> > > > > > > > > evidence that we are failing to attract new >> >> contributors or >> >> >> > > that >> >> >> > > > > > > > > feature requests / bugs aren't being reported. The >> way >> >> >> that we >> >> >> > > work >> >> >> > > > > > is >> >> >> > > > > > > > > _different_, so adapting to the Apache process will >> >> require >> >> >> > > change. >> >> >> > > > > > > > > >> >> >> > > > > > > > > > Additionally, I was and still am concerned about >> the >> >> >> overall >> >> >> > > > > > release >> >> >> > > > > > > > > > process of the apache/arrow project. I know there >> have >> >> >> been >> >> >> > > > > efforts >> >> >> > > > > > > > there >> >> >> > > > > > > > > > as well to make it easier for individual languages >> to >> >> >> > > release on >> >> >> > > > > > their >> >> >> > > > > > > > own >> >> >> > > > > > > > > > cadence, but just anecdotally, the >> JuliaData/Arrow.jl >> >> has >> >> >> > > > > > > > had/needed/wanted >> >> >> > > > > > > > > > 10 patch and minor releases since the original code >> >> >> donation, >> >> >> > > > > > whereas >> >> >> > > > > > > > the >> >> >> > > > > > > > > > apache/arrow project has had one (3.0.0). This >> leads >> >> to >> >> >> some >> >> >> > > of >> >> >> > > > > the >> >> >> > > > > > > > > > concerns I have with restricting development to >> just >> >> the >> >> >> > > > > > apache/arrow >> >> >> > > > > > > > > > repository: how exactly does the release process >> work >> >> for >> >> >> > > > > > individual >> >> >> > > > > > > > > > languages who may desire independent releases apart >> >> from >> >> >> the >> >> >> > > > > > quarterly >> >> >> > > > > > > > > > overall project releases? I think from the Rust >> >> thread I >> >> >> > > remember >> >> >> > > > > > that >> >> >> > > > > > > > you >> >> >> > > > > > > > > > just need a group of language contributors to all >> >> agree, >> >> >> but >> >> >> > > what >> >> >> > > > > > if >> >> >> > > > > > > > I'm >> >> >> > > > > > > > > > the only "active" Julia contributor? It's also >> unclear >> >> >> what >> >> >> > > the >> >> >> > > > > > > > > > expectations are for actual development: with the >> >> >> original >> >> >> > > code >> >> >> > > > > > > > donation >> >> >> > > > > > > > > > PRs, I know Neal "reviewed" the PRs, but perhaps >> >> missed >> >> >> the >> >> >> > > > > details >> >> >> > > > > > > > around >> >> >> > > > > > > > > > how I proposed development continue going forward. >> Is >> >> it >> >> >> > > required >> >> >> > > > > > to >> >> >> > > > > > > > have a >> >> >> > > > > > > > > > certain number of reviews before merging? On the >> Julia >> >> >> side, >> >> >> > > I >> >> >> > > > > can >> >> >> > > > > > try >> >> >> > > > > > > > to >> >> >> > > > > > > > > > encourage/push for those who have contributed to >> the >> >> >> > > > > > JuliaData/Arrow.jl >> >> >> > > > > > > > > > repository to help review PRs to apache/arrow, but >> I >> >> also >> >> >> > > can't >> >> >> > > > > > > > guarantee >> >> >> > > > > > > > > > we would always have someone to review. It just >> feels >> >> >> pretty >> >> >> > > > > > awkward >> >> >> > > > > > > > if I >> >> >> > > > > > > > > > keep needing to ping non-Julia people to "review" a >> >> PR to >> >> >> > > merge >> >> >> > > > > it. >> >> >> > > > > > > > Perhaps >> >> >> > > > > > > > > > this is just a problem of the overall Julia >> >> >> implementation >> >> >> > > > > > "smallness" >> >> >> > > > > > > > in >> >> >> > > > > > > > > > terms of contributors, but I'm not sure on the best >> >> >> answer >> >> >> > > here. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > Several things here: >> >> >> > > > > > > > > >> >> >> > > > > > > > > * If you want to do separate Julia releases, you are >> >> free >> >> >> to do >> >> >> > > > > that, >> >> >> > > > > > > > > but you have to follow the process (voting on the >> >> mailing >> >> >> list, >> >> >> > > > > > > > > publishing GPG-signed source artifacts) >> >> >> > > > > > > > > * If you had been working "in the community" since >> >> >> November, >> >> >> > > you >> >> >> > > > > > would >> >> >> > > > > > > > > probably already be a committer, so there is a >> >> >> bootstrapping >> >> >> > > here >> >> >> > > > > > that >> >> >> > > > > > > > > has failed to take place. In the meantime, we are >> more >> >> than >> >> >> > > happy >> >> >> > > > > to >> >> >> > > > > > > > > help you "earn your wings" (as a committer) as >> quickly >> >> as >> >> >> > > possible. >> >> >> > > > > > > > > But from my perspective, I see a code donation and >> two >> >> >> other >> >> >> > > > > commits, >> >> >> > > > > > > > > which isn't enough to make a case for committership. >> >> >> > > > > > > > > >> >> >> > > > > > > > > > So in short, I'm not sure on the best path >> forward. I >> >> >> think >> >> >> > > > > > strictly >> >> >> > > > > > > > > > restricting development to the apache/arrow >> physical >> >> >> > > repository >> >> >> > > > > > would >> >> >> > > > > > > > > > actively hurt the progress of the Julia >> >> implementation, >> >> >> > > whereas >> >> >> > > > > it >> >> >> > > > > > > > *has* >> >> >> > > > > > > > > > been progressing with increasing momentum since >> first >> >> >> > > released. >> >> >> > > > > > There >> >> >> > > > > > > > are >> >> >> > > > > > > > > > posts on the Julia discourse forum, in the Julia >> slack >> >> >> and >> >> >> > > zulip >> >> >> > > > > > > > > > communities, and quite a few issues/PRs being >> opened >> >> at >> >> >> the >> >> >> > > > > > > > > > JuliaData/Arrow.jl repository. There have been >> several >> >> >> calls >> >> >> > > for >> >> >> > > > > > arrow >> >> >> > > > > > > > > > flight support, with a member from Julia Computing >> >> >> actually >> >> >> > > close >> >> >> > > > > > to >> >> >> > > > > > > > > > releasing a gRPC client >> >> >> > > > > > > > > > <https://github.com/JuliaComputing/gRPCClient.jl> >> >> >> > > specifically >> >> >> > > > > > > > > > to help with flight support. But in terms of actual >> >> >> > > committers, >> >> >> > > > > > it's >> >> >> > > > > > > > been >> >> >> > > > > > > > > > primarily just myself, with a few minor >> contributions >> >> by >> >> >> > > others. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > I guess the big question that comes to mind is what >> >> are >> >> >> the >> >> >> > > hard >> >> >> > > > > > > > > > requirements to be considered an "official >> >> >> implementation"? >> >> >> > > Does >> >> >> > > > > > the >> >> >> > > > > > > > code >> >> >> > > > > > > > > > *have* to live in the same physical repo? Or if it >> >> >> passed the >> >> >> > > > > > series of >> >> >> > > > > > > > > > archery integration tests, would that be enough? I >> >> >> apologize >> >> >> > > for >> >> >> > > > > my >> >> >> > > > > > > > > > naivete/inexperience on all things "apache", but I >> >> >> imagine >> >> >> > > that's >> >> >> > > > > > a big >> >> >> > > > > > > > > > part of it: having official development/releases >> >> through >> >> >> the >> >> >> > > > > > > > apache/arrow >> >> >> > > > > > > > > > community, though again I'm not exactly sure on the >> >> >> formal >> >> >> > > > > > processes >> >> >> > > > > > > > here? >> >> >> > > > > > > > > > I would like to keep Julia as an official >> >> implementation, >> >> >> > > but I'm >> >> >> > > > > > also >> >> >> > > > > > > > > > mostly carrying the maintainership alone at the >> moment >> >> >> and >> >> >> > > want >> >> >> > > > > to >> >> >> > > > > > be >> >> >> > > > > > > > > > realistic with the future of the project. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > The critical matter is whether the >> >> development/maintenance >> >> >> > > work is >> >> >> > > > > > > > > conducted by the "Arrow community" in accordance with >> >> the >> >> >> > > Apache >> >> >> > > > > Way, >> >> >> > > > > > > > > which is to say individuals collaborating with each >> >> other >> >> >> on >> >> >> > > Apache >> >> >> > > > > > > > > channels (for communication and development) and >> >> avoiding >> >> >> the >> >> >> > > bad >> >> >> > > > > > > > > patterns you see sometimes in other communities (e.g. >> >> >> > > inconsistent >> >> >> > > > > > > > > openness). >> >> >> > > > > > > > > >> >> >> > > > > > > > > It's fine — really, no pressure — if you want to be >> >> >> > > independent and >> >> >> > > > > > do >> >> >> > > > > > > > > things your own way, you just have to be clear that >> you >> >> are >> >> >> > > > > > > > > independent and not operating as part of the Apache >> >> Arrow >> >> >> > > > > community. >> >> >> > > > > > > > > You can't have it both ways, though. No hard feelings >> >> >> whatever >> >> >> > > you >> >> >> > > > > > > > > decide, but the current "dump code over the wall >> >> >> occasionally" >> >> >> > > > > > > > > approach but work on independent channels is not >> >> >> compatible. >> >> >> > > > > Building >> >> >> > > > > > > > > healthy open source communities is hard, but this way >> >> has >> >> >> been >> >> >> > > > > shown >> >> >> > > > > > > > > to work well, which is why I've spent the last 6 >> years >> >> >> working >> >> >> > > hard >> >> >> > > > > > to >> >> >> > > > > > > > > bring people together to build this project and >> >> ecosystem! >> >> >> > > > > > > > > >> >> >> > > > > > > > > If you want to maintain a test harness here to >> verify an >> >> >> > > > > independent >> >> >> > > > > > > > > Julia implementation, that's fine, too. I'm >> disappointed >> >> >> that >> >> >> > > > > things >> >> >> > > > > > > > > failed to bootstrap after the code donation, so I >> want >> >> to >> >> >> see >> >> >> > > if we >> >> >> > > > > > > > > can course correct quickly or if not decide to go our >> >> >> separate >> >> >> > > > > ways. >> >> >> > > > > > > > > >> >> >> > > > > > > > > Thanks, >> >> >> > > > > > > > > Wes >> >> >> > > > > > > > > >> >> >> > > > > > > > > > I'm open to discussion and ideas on the best way >> >> forward. >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > -Jacob >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > On Tue, Mar 30, 2021 at 2:03 PM Wes McKinney < >> >> >> > > > > wesmck...@gmail.com> >> >> >> > > > > > > > wrote: >> >> >> > > > > > > > > > >> >> >> > > > > > > > > > > hi folks, >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > I was very surprised today to learn that the >> Julia >> >> >> Arrow >> >> >> > > > > > > > > > > implementation has continued operating more or >> less >> >> >> like an >> >> >> > > > > > > > > > > independent open source project since the code >> >> donation >> >> >> > > last >> >> >> > > > > > > > November: >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > >> https://github.com/JuliaData/Arrow.jl/commits/main >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > There may have been a misunderstanding about what >> >> was >> >> >> > > expected >> >> >> > > > > to >> >> >> > > > > > > > > > > occur after the code donation, but it's >> problematic >> >> >> for a >> >> >> > > bunch >> >> >> > > > > > of >> >> >> > > > > > > > > > > reasons (IP lineage / governance / community >> >> >> development) >> >> >> > > to >> >> >> > > > > have >> >> >> > > > > > > > work >> >> >> > > > > > > > > > > happening on the implementation "outside the >> >> >> community". >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > In any case, what is done is done, so the Arrow >> >> PMC's >> >> >> > > position >> >> >> > > > > on >> >> >> > > > > > > > this >> >> >> > > > > > > > > > > would be roughly to regard the work as a hard >> fork >> >> of >> >> >> > > what's in >> >> >> > > > > > > > Apache >> >> >> > > > > > > > > > > Arrow, which given its development activity is >> more >> >> or >> >> >> less >> >> >> > > > > > inactive >> >> >> > > > > > > > > > > [1]. (I had actually thought the project was >> simply >> >> >> > > inactive >> >> >> > > > > > after >> >> >> > > > > > > > the >> >> >> > > > > > > > > > > code donation) >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > The critical question now is, is there interest >> from >> >> >> Julia >> >> >> > > > > > developers >> >> >> > > > > > > > > > > in working "in the community", which is to say: >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > * Having development discussions on ASF channels >> >> >> (mailing >> >> >> > > list, >> >> >> > > > > > > > > > > GitHub, JIRA), planning and communicating in the >> >> open >> >> >> > > > > > > > > > > * Doing all development in ASF GitHub >> repositories >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > The answer to the question may be "no" (which is >> >> okay), >> >> >> > > but if >> >> >> > > > > > that's >> >> >> > > > > > > > > > > the case, I don't think we should be giving the >> >> >> impression >> >> >> > > that >> >> >> > > > > > we >> >> >> > > > > > > > > > > have an official Julia implementation that is >> >> >> developed and >> >> >> > > > > > > > maintained >> >> >> > > > > > > > > > > by the community (and so my argument would be >> >> >> > > unfortunately to >> >> >> > > > > > drop >> >> >> > > > > > > > > > > the donated code from the project). >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > If the answer is "yes", there needs to be a hard >> >> >> > > commitment to >> >> >> > > > > > move >> >> >> > > > > > > > > > > development to Apache channels and not look >> back. We >> >> >> would >> >> >> > > also >> >> >> > > > > > need >> >> >> > > > > > > > > > > to figure out what to do to document and >> synchronize >> >> >> the >> >> >> > > new IP >> >> >> > > > > > > > that's >> >> >> > > > > > > > > > > been created since the code donation. >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > Thanks, >> >> >> > > > > > > > > > > Wes >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > > [1]: >> >> >> > > > > https://github.com/apache/arrow/commits/master/julia/Arrow >> >> >> > > > > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > >> >> >> >> >> >>
