I'll add, the process for packaging CQLSH similar if not identical to how the Python driver will be packaged by Apache for pypl.org once the driver donation in CEP-8 <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation> is completed. The python driver already has an official PYPL distribution at https://pypi.org/project/cassandra-driver/.
On Wed, Dec 6, 2023 at 3:03 PM Jeff Widman <j...@jeffwidman.com> wrote: > 👋 I'm the other current maintainer of https://github.com/jeffwidman/cqlsh > . > > *> Knowing nothing about the pypi release/publish process, I'm curious how > you would stage and then publish the signed convenience package. > Background: what we publish post-release-vote needs to be signed and > identical to what is staged when the release vote starts. See the two > scripts prepare_release.sh and finish_release.sh > in https://github.com/apache/cassandra-builds/tree/trunk/cassandra-release > <https://github.com/apache/cassandra-builds/tree/trunk/cassandra-release> , > where all the packaging is done in prepare_ and finish_ is just about > pushing what's in staging to the correct public locations. I am assuming > that the CEP would be patching these two files, as well as adding files > in-tree to the pylib/ directory* > > From a tactical implementation standpoint, there's a few ingredients to a > release: > > 1. Packaging code used by PyPI such as `pyproject.toml`: > 2. Code freeze of the functional code. > 3. Versioning. > 4. Secret management of the PyPI credentials > 5. The actual publishing to PyPI. > 6. Maintaining the above, ie, fixing breakage and keeping it up to > date. > > Looking at them in more detail: > > 1. Packaging code used by PyPI such as `pyproject.toml` - This should > be easy to add into the tree. Brad / myself would be happy to contribute > and we should be able to pull most of it directly from > https://github.com/jeffwidman/cqlsh. > 2. Code freeze of the functional code - This already happens today > upon every Cassandra release. > 3. Versioning - Versioning is a pain since currently CQLSH versions > are not aligned with Cassandra. Furthermore the internal CQLSH version > number doesn't always increment when a new version of Cassandra / CQLSH is > released. However, PyPI requires every release artifact to have a unique > version number. So we work around this currently by saying "Here's pypi > version X, which contains the cqlsh version from Y extracted from Cassandra > version Z". > 1. If you want to keep CQLSH releases in-lockstep with Cassandra, > then life would be _much_ simpler if the CQLSH version directly pulled > from > the Cassandra version. > 2. However, there's still the problem that sometimes the CQLSH > python packaging may have a bug, which forces a new release of CQLSH. > Seems > a bit heavyweight to require a new release of Cassandra just to fix a > bug > in the python packaging. > 3. Another option is to have CQLSH release *not* tied at the hip to > Cassandra releases. Extract it to a separate project/repo and then pull > in > specific releases of CQLSH into the Cassandra final release. Probably > too > heavyweight right now given we are trying to simplify life, but wanted > to > mention it. > 4. I don't feel strongly on the above, other than the current state > of affairs of requiring three different versions is worse than either of > the above options. > 4. Secret management of the PyPI credentials > 1. I'm not sure if Apache projects have a special "apache account" > that they typically use, or if they add multiple maintainers as admins > on > PyPI, and then add/remove them as they join/drop the core team. Either > works for me. > 2. We'd probably want to keep Brad / myself as admins on PyPI as > we'll be more attentive to breakage / fixing things but that's really > up to > the discretion of the core team... I'm fine if you folks prefer to > remove > our access. > 3. Making the secrets available to the publishing tool can be > managed using PyPI's trusted publishing: > > https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#configuring-trusted-publishing > . > 5. The actual publishing to PyPI. > 1. The "staged" releases could be pushed to > https://test.pypi.org/project/cqlsh/ and then the final released > pushed to the normal https://pypi.org/project/cqlsh/ > 2. The commands to publish to PyPI could be added to the > prepare_release.sh and finish_release.sh. > 3. Alternatively, could add a CI/CD action such as a github action > directly to the Cassandra repo that watches for new git tags on the repo > and pushes those versions out to PyPI. > 4. I'm not particularly wedded to a solution, but if I had to > choose, I'd go with the GitHub action. The reason being that it's a > simple, > commonly followed way of doing things within the Python world, so if it > breaks, it's easy for a drive-by contributor who's familiar with > Python/PyPI to contribute a fix. > 6. Maintaining the above, ie, fixing breakage and keeping it up to > date. > 1. Brad and I are both happy to help with this on an ongoing basis. > 2. I suspect the core team won't want to give us commit rights > (understandably) so we'll still need someone on the core team to approve > any PR's that we contribute. That does worry me a little given that this > low-risk PR (https://github.com/apache/cassandra/pull/2389) has sat > open for 6 months despite multiple nudges, but that's a bigger > conversation > than CQLSH. > 3. There are some things that aren't code-based, such as repo > settings or admin rights, and those either require the core team to > own, or > give myself/Brad the ability to change. Probably easiest / most secure > if > the core team owns, but that's up to you folks. > > Let me know if I'm forgetting anything from an implementation perspective. > > -- > > Apart from the implementation, I did want to briefly get philosophical > about whether CQLSH should stay in python and support pip as an official > distribution target. > > Many of you approach Cassandra from the perspective of either a Java > developer or a server administrator deeply familiar with Cassandra. It's > easy to forget what the developer experience is like when you're brand new > to Cassandra. For example, when I started using Cassandra, I was a > python/go developer, hitting some confusing responses from Cassandra. No > one on my team was very familiar with the Java toolchain no did we have > access to the underlying database cluster, so it was a relief when we > discovered `pip install cqlsh` didn't require futzing with Debian packages > or Java. Debugging a weird response from Cassandra was nothing more than > `import pdb; pdb.set_trace()` and then run the script to poke around the > raw responses. No need to tcpdump, export off the server locally so we > could inspect in wireshark. > > Beyond application developers, there's also many DevOps teams tasked with > setting up / maintaining Cassandra clusters for application teams. I don't > think that's a good way to organize engineering teams, but regardless, tis > a fact that many companies go this route. Python is super common in DevOps > scripting world, more so than Java, so exposing `cqlsh` in Python keeps > Cassandra approachable to a wider audience. > > Since the tool is written in Python, we assumed the PyPI artifact would be > an official distribution target. It was only after we'd been using it for a > few months that we realized that the `cqlsh` available through pip was > quite outdated. That's what originally motivated me to volunteer to help > maintain the project. > > Obviously there's an increased maintenance cost, but my point is let's not > forget the benefits of making Cassandra accessible to new end users who > aren't Java-centric. > > Just my $0.02. > > > > On Mon, Aug 14, 2023 at 7:14 AM Mick Semb Wever <m...@apache.org> wrote: > >> >> I'm supportive of this CEP. And thanks for all the maintenance on this >> over the years! >> >> Knowing nothing about the pypi release/publish process, I'm curious how >> you would stage and then publish the signed convenience package. >> Background: what we publish post-release-vote needs to be signed and >> identical to what is staged when the release vote starts. See the two >> scripts prepare_release.sh and finish_release.sh in >> https://github.com/apache/cassandra-builds/tree/trunk/cassandra-release >> , where all the packaging is done in prepare_ and finish_ is just about >> pushing what's in staging to the correct public locations. I am assuming >> that the CEP would be patching these two files, as well as adding files >> in-tree to the pylib/ directory >> >> >> On Wed, 9 Aug 2023 at 18:18, Brad <bscho...@gmail.com> wrote: >> >>> >>> As per the CEP process guidelines, I'm starting a formal DISCUSS thread >>> to resume the conversation started here[1]. >>> >>> The developers who maintain the Python CQLSH client on the official >>> Python PYPI repository would like to integrate and donate their open source >>> work to the Apache Cassandra project so it can be more tightly and >>> seamlessly integrated. >>> >>> The Apache Cassandra project pre-dates the adoption in Python 3.4 of >>> PyPI as the default package manager. As a result, an unofficial >>> distribution has been provided by a group of developers who have maintained >>> the repository there since October 2013. >>> >>> The installable version of CQLSH on PyPI.org allows end users to install >>> a cqlsh client with PIP - no tarball or path setup required. I.e., >>> >>> $ pip install cqlsh >>> >>> This popular package has 50K downloads per month and is today maintained >>> by Jeff Wideman and Brad Schoening. The PYPI package is updated upon every >>> major release by simply repackaging the CQLSH that ships with every >>> Cassandra release. >>> >>> CQLSH PyPI Repository: https://pypi.org/project/cqlsh/ >>> <https://urldefense.com/v3/__https://pypi.org/project/cqlsh/__;!!PbtH5S7Ebw!Zlneg-wAg-i69cUOjOGZ2sppmSq0Yofe9AaPf2BZgtNQ-gfBVWCT8Wem23iAhpBrf4fXNyehbeJCLwQ$> >>> >>> >>> This CEP Proposal suggests incorporating PYPI as a regular part of the >>> Cassandra release process and making the CQLSH project on PYPI an official >>> distribution point. >>> >>> The full CEP can be reviewed at: >>> >>> Wiki: CEP-35: Add PIP support for CQLSH >>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425995> >>> . >>> >>> Jira: CASSANDRA-18654 >>> <https://issues.apache.org/jira/browse/CASSANDRA-18654> >>> >>> >>> But in brief, the proposal will: >>> >>> - Add PyPI.org as an official distribution point for CQLSH >>> - Allow end users to install CQLSH with simply 'pip install cqlsh' >>> on MacOS, Windows and Linux platforms. >>> - Donate the modest amount of existing configuration files by the >>> authors to Apache Cassandra >>> - This only involves the Python CQLSH client, no changes to >>> distribution of Java server side code and tools are involved. >>> >>> We welcome further discussion and suggestions regarding this proposal on >>> the mailing list here. >>> >>> Regards, >>> >>> Jeff Widman & >>> Brad Schoening >>> >>> [1] https://lists.apache.org/thread/sy3p2b2tncg1bk6x3r0r60y10dm6l18d >>> <https://lists.apache.org/thread.html/ra7caa1dd42ccaa04bcabfbc33233995c125c655f9a3cdb2c7bd8e9f7%40%3Cdev.cassandra.apache.org%3E> >>> >> > > -- > > *Jeff Widman* > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) > <>< >
