Works for me
> On Nov 15, 2021, at 4:21 AM, Jacek Lewandowski <lewandowski.ja...@gmail.com>
> wrote:
>
> I'd put it another way - the scope is to make it possible to provide a new
> implementation of sstable format without the necessity to refactor
> Cassandra code. It implies a contract about the responsibilities of sstable
> format implementation so that the rest of the code can rely on that, and
> only on that, and do not make assumptions beyond that. But it does not
> claim that the created interfaces will not change even with a minor version
> release. When those interfaces are around for sometime, we can start a
> separate discussion about whether we want to put some guarantees on them.
>
> - - -- --- ----- -------- -------------
> Jacek Lewandowski
>
>
> On Wed, Nov 10, 2021 at 9:01 PM David Capwell <dcapw...@apple.com.invalid>
> wrote:
>
>> If this gets descoped to test only (can break all interfaces in a minor)
>> then my support concerns are no longer valid; I am cool with the CEP scoped
>> only to improving testing
>>
>>> On Nov 10, 2021, at 11:20 AM, Jacek Lewandowski <
>> lewandowski.ja...@gmail.com> wrote:
>>>
>>> For the other ticket (schema update handler interface) I was also
>> proposing
>>> a kind of @DeveloperApi annotation as seen in other projects but
>> similarly
>>> to this thread there were different opinions and no conclusion. After
>>> reading the comments I must agree that perhaps it is way too early to
>> mark
>>> this interface as stable. Perhaps it was too far-fetched to say it would
>> be
>>> for people who wished to replace the SSTable format. My focus is
>>> primarily on cleaning up the code (modularization and clean contracts)
>> and
>>> making it possible to introduce a new format in the future while allowing
>>> us to maintain the old format (no "if then else" approach)
>>>
>>> - - -- --- ----- -------- -------------
>>> Jacek Lewandowski
>>>
>>>
>>> On Wed, Nov 10, 2021 at 12:53 AM bened...@apache.org <
>> bened...@apache.org>
>>> wrote:
>>>
>>>>> I may be wrong here, but the CEP directly calls out making this api
>>>> public for people who wish to replace the SSTable format
>>>>
>>>> I don’t think this implies API stability. For starters, it doesn’t
>>>> stipulate that these implementations will be supported out of tree (the
>>>> only one I’m aware of, so far as I understand, is intended to be
>> incubated
>>>> in tree), nor does an API for external usage have to be stable. It’s
>> fine
>>>> to create an API and tell users it’s unstable, and that they should
>> closely
>>>> monitor patch version changes if they use it.
>>>>
>>>> That said, norms may be changing around what can go into patch releases
>>>> anyhow, so this may be a lot of noise about nothing. If all new
>> development
>>>> goes into trunk, then it’s all moot. But I don’t think we can make hard
>>>> assumptions about that today, as historically these sorts of intentions
>>>> haven’t lasted.
>>>>
>>>> I’m fairly against the idea of introducing hard restrictions on this,
>> and
>>>> potentially ossifying the codebase. I’m not keen to even consider out of
>>>> tree consumers of these APIs in any way, for compatibility,
>> upgradeability
>>>> or anything. There’s a lot that needs to be done over the coming years
>> to
>>>> improve the internal structure of the project, and unduly entrenching
>> the
>>>> current state of affairs would be a huge potential harm of these
>> efforts to
>>>> modularise the codebase.
>>>>
>>>> From: David Capwell <dcapw...@apple.com.INVALID>
>>>> Date: Tuesday, 9 November 2021 at 23:38
>>>> To: dev@cassandra.apache.org <dev@cassandra.apache.org>
>>>> Subject: Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
>>>>> My understanding is that the only interface that is expected to be
>>>> stable for external consumers is the secondary index API
>>>>
>>>> I may be wrong here, but the CEP directly calls out making this api
>> public
>>>> for people who wish to replace the SSTable format ("Cassandra developers
>>>> who want to develop and publish different file format
>> implementations."),
>>>> so if we need to support 2i API, why would we not support SSTable API as
>>>> well?
>>>>
>>>>> All of the other mentioned APIs are in my opinion for internal usage
>> only
>>>>
>>>> This gets back to my point; it is currently tribal knowledge what needs
>> to
>>>> work and what doesn’t, and without the broader set of committers knowing
>>>> this then the likely hood any new API will break in a minor is high.
>>>>
>>>>> On Nov 9, 2021, at 12:13 PM, bened...@apache.org wrote:
>>>>>
>>>>> I agree that we don’t need to block the CEP on this, and that we should
>>>> have that discussion. But it’s worth noting that the CEP should not
>>>> anticipate or depend on any specific outcome of that discussion.
>>>>>
>>>>> Since it is somewhat relevant for this discussion, my view is that no
>>>> interface should be assumed to be stable without the prior explicit
>>>> agreement of the community.
>>>>>
>>>>> My understanding is that the only interface that is expected to be
>>>> stable for external consumers is the secondary index API. Perhaps also
>>>> snitches? But also perhaps not, as the difficulty of upgrading these at
>> the
>>>> same time is pretty low for custom snitches. All of the other mentioned
>>>> APIs are in my opinion for internal usage only, so users should not
>> assume
>>>> compile time compatibility across any release, and I am certain we have
>>>> never tried to maintained this. This still facilitates forks of course,
>> by
>>>> localising the compatibility work.
>>>>>
>>>>>
>>>>> From: Jeremiah D Jordan <jeremiah.jor...@gmail.com>
>>>>> Date: Tuesday, 9 November 2021 at 19:43
>>>>> To: Cassandra DEV <dev@cassandra.apache.org>
>>>>> Subject: Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
>>>>> I would love to have this discussion and setup annotations or similar
>> to
>>>> formalize things. I just do not think we need to hold any up CEPs to do
>>>> so. That discussion should possibly be a CEP of its own proposing how
>> we
>>>> want to formalize interfaces? I would be happy to go through and try to
>>>> put together something for that or since you feel so strongly about it
>>>> maybe you want to David? At the very least it should get its own
>> DISCUSS
>>>> thread and then be written up in the wiki.
>>>>>
>>>>> -Jeremiah
>>>>>
>>>>>> On Nov 9, 2021, at 1:06 PM, Joshua McKenzie <jmcken...@apache.org>
>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> trunk -> anything goes, not trunk -> try not to change these
>> interfaces
>>>>>>
>>>>>> Have we ever clarified what "these interfaces" are? Was just talking
>> to
>>>>>> David and I realized I didn't even JavaDoc CommitLogReadHandler as
>>>> _being
>>>>>> designed_ for external usage. /sigh
>>>>>>
>>>>>> I think it'd be valuable for us to go through the codebase and
>> annotate
>>>>>> interfaces as intended to be exposed to 3rd parties; this has bothered
>>>> me
>>>>>> for years. Especially as we come up on a large number of new cleanups,
>>>>>> refactorings, and potentially genericizing some subsystems into API's
>>>>>> (CEP-18 descendents).
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 9, 2021 at 2:01 PM David Capwell
>> <dcapw...@apple.com.invalid
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>>> We already have many interfaces similar to these for Compaction
>>>>>>> Strategy, Indexing, Query Handler.
>>>>>>>
>>>>>>> Today-I-Learned QueryHandler is not allowed to be touched in a minor…
>>>> good
>>>>>>> to know…
>>>>>>>
>>>>>>>> not trunk -> try not to change these interfaces
>>>>>>>
>>>>>>> Outside of MBeans, I honestly do not know what interfaces fall into
>>>> this
>>>>>>> group; and for MBeans we have tests which block breaking changes.
>> The
>>>>>>> point I am making is that not everyone is aware of the rules, so
>> having
>>>>>>> something in place to help enforce such rules should be thought
>> about;
>>>> if
>>>>>>> we want to add pluggable hooks with the intent that external parties
>>>> can
>>>>>>> leverage such hooks, we should also add to the scope the maintenance
>> of
>>>>>>> these interfaces (we should not assume “tribal knowledge” will work).
>>>>>>>
>>>>>>> I am not trying to ask for something large or something requiring a
>>>> ton of
>>>>>>> work, I am just asking that this gets thought about during the
>> project
>>>> so
>>>>>>> it doesn’t get neglected. This could be as simple as an annotation
>>>> like
>>>>>>> @ExposedTo3rdParties (Hadoop does this to show an interface is
>> exposed
>>>> and
>>>>>>> must be maintained), or it could be something like split directories
>>>>>>> (src/java = private, src/java-exposed = public); I am trying not to
>>>> dictate
>>>>>>> an implementation, only trying to make sure we are setup to support
>>>> the CEP
>>>>>>> after the work is done.
>>>>>>>
>>>>>>>
>>>>>>>> On Nov 9, 2021, at 9:52 AM, Jeremiah D Jordan <
>>>> jeremiah.jor...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> We already have many interfaces similar to these for Compaction
>>>>>>> Strategy, Indexing, Query Handler. I would hope that commiters are
>>>> already
>>>>>>> following a policy along the lines of trunk -> anything goes, not
>>>> trunk ->
>>>>>>> try not to change these interfaces. I would expect that to be the
>> same
>>>>>>> policy for any new internal interfaces that are added. But given we
>>>>>>> already have many such interfaces, I see no reason to block adding
>>>> more of
>>>>>>> them while change policies are discussed.
>>>>>>>>
>>>>>>>> -Jeremiah
>>>>>>>>
>>>>>>>>> On Nov 9, 2021, at 10:44 AM, David Capwell
>>>> <dcapw...@apple.com.INVALID>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I still have one outstanding comment, but this is a comment for
>>>> several
>>>>>>> of the CEPs being worked on
>>>>>>>>>
>>>>>>>>>> And last comment, which I have also done in the other modularity
>>>>>>> thread… backwards compatibility and maintenance. It is not clear
>> right
>>>> now
>>>>>>> what java interfaces may not break and how we can maintain and extend
>>>> such
>>>>>>> interfaces in the future. If the goal is to allow 3rd parties to
>>>> plugin
>>>>>>> and offer new SSTable formats, are we as a project ok with having a
>>>> minor
>>>>>>> release do a binary or source non-compatible change? If not how do
>> we
>>>>>>> detect this? Until this problem is solved, I do not think we should
>>>> add
>>>>>>> any such interfaces.
>>>>>>>>>
>>>>>>>>> I would love some clarity on this. Specifically, if we assume a
>>>> patch
>>>>>>> author/reviewers are not familiar with the impact of changes these
>>>>>>> interfaces, what happens? Do we have tools to block this? Do we
>>>> require
>>>>>>> 3rd party authors to create massive shims to deal with every patch
>>>> level
>>>>>>> version out there? I would love more clarity on how we maintain
>> these
>>>> new
>>>>>>> pluggable interfaces.
>>>>>>>>>
>>>>>>>>>> On Nov 9, 2021, at 4:45 AM, Branimir Lambov <blam...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Does anyone have any further comments or questions on the
>> proposal,
>>>> or
>>>>>>> are
>>>>>>>>>> we ready to move forward to a vote?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Branimir
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 2, 2021 at 7:15 PM David Capwell
>>>>>>> <dcapw...@apple.com.invalid>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> I apologize I did not mention those things explicitly. All the
>>>> places
>>>>>>>>>>> where
>>>>>>>>>>>> sstable files are accessed directly would have to be refactored.
>>>>>>>>>>>
>>>>>>>>>>> Works for me
>>>>>>>>>>>
>>>>>>>>>>>> Speaking about the implementation, one idea I was thinking about
>>>> was
>>>>>>> that
>>>>>>>>>>>> the factories for formats are registered using Java's native
>>>> service
>>>>>>>>>>>> loader.
>>>>>>>>>>>
>>>>>>>>>>> I am a fan of ServiceLoader as a means of plugging in.
>>>>>>>>>>>
>>>>>>>>>>>> I hope this explains a bit
>>>>>>>>>>>
>>>>>>>>>>> Yep; thanks!
>>>>>>>>>>>
>>>>>>>>>>>> On Nov 2, 2021, at 1:46 AM, Jacek Lewandowski <
>>>>>>>>>>> lewandowski.ja...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> David,
>>>>>>>>>>>>
>>>>>>>>>>>> I apologize I did not mention those things explicitly. All the
>>>> places
>>>>>>>>>>> where
>>>>>>>>>>>> sstable files are accessed directly would have to be refactored.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding TableMetrics - currently it includes many metrics,
>> some
>>>> of
>>>>>>> them
>>>>>>>>>>>> are unrelated to sstables at all, but there are metrics which
>> are
>>>>>>>>>>> specific
>>>>>>>>>>>> to the current sstable format, like metrics related to index
>>>>>>> summaries or
>>>>>>>>>>>> bloom filters. The created gauges query certain methods on
>> sstable
>>>>>>>>>>> reader -
>>>>>>>>>>>> I think the only common metrics for sstables we can leave in
>>>>>>> TableMetrics
>>>>>>>>>>>> are those for which there are query methods in generic sstable
>>>>>>> interface.
>>>>>>>>>>>> Other metrics, specific to the certain sstable format should be
>>>>>>>>>>> registered
>>>>>>>>>>>> by the implementation itself.
>>>>>>>>>>>>
>>>>>>>>>>>> Speaking about the implementation, one idea I was thinking about
>>>> was
>>>>>>> that
>>>>>>>>>>>> the factories for formats are registered using Java's native
>>>> service
>>>>>>>>>>>> loader. This way we could get the list of all the factories on
>> the
>>>>>>>>>>>> classpath and call some method, like `registerMetrics` during
>>>> system
>>>>>>>>>>>> initialization. That could be also implemented in static
>>>> initializer
>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>> factory but it would make it less obvious for the implementors
>>>> where
>>>>>>> such
>>>>>>>>>>>> initialization should be done.
>>>>>>>>>>>>
>>>>>>>>>>>> I hope this explains a bit
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Jacek
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org
