[
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424480#comment-16424480
]
Joseph Lynch commented on CASSANDRA-14346:
------------------------------------------
{quote}You're free to spend time implementing any solution for this you’d like.
Just keep in mind that it’s unlikely to get committed until there’s been a
wider discussion on how we should be approaching this problem as a project. You
have 2 people on this ticket concerned that this might not be the right way to
approach the problem of scheduling repairs.
{quote}
My understanding is that there are two alternative proposals, please correct me
if I'm missing some:
# Cassandra ships with repair scheduling in a sidecar process (either with or
without reliable IPC).
# We continue doing nothing and the community just solves this in different
ways.
The current production implementation we have internally is basically #1, it
would be _easier_ for us to open source it as a separate service or jar. When
we talked about this at NGCC (and later conversations) [~kohlisankalp] and
others had objections to us open sourcing it as yet another external tool, so
we changed directions to working on porting into Cassandra itself. We were
further convinced by our production experience with a sidecar based scheduler
that indicates this is not a particularly reliable way to build the scheduler
due to numerous IPC difficulties (which I have described in detail above and in
the design document). To be honest much I don't think we _need_ up front
consensus on sidecar vs in-process since it is reasonably easy to port the code
so that we can switch between them easily (except the table config, that's the
one thing we'd need to figure out).
I don't view #2 (status quo) as a good solution. Do you?
{quote}You also haven't addressed my comments about why ipc difficulties and
being in process making it easier to detect problems aren't strong arguments
for doing this in process.
{quote}
I thought I did, let me try again. If we're inside Cassandra then repair state
(status of repair) and repair scheduling state (status of cluster repair) are
shared in the same process, so we can easily call methods, setup thread
heartbeats, etc ... If we are in separate processes then not only do we have to
deal with differing process lifecycles, but every piece of communication must
go through JMX (or not yet existing reliable IPC); this slows down iteration
and limits repair scheduling capability and reliability. My understanding of
your disagreement is that we _could_ theoretically have reliable IPC but I just
don't agree and welcome any concrete proposal that provides the same level of
reliability that in-process inter-thread communication has.
{quote}Discussing alternate approaches with an open mind is probably a better
use of time.
{quote}
I am trying to be very open in discussing alternate approaches, and I'm happy
to take the sidecar implementation we have an put it literally anywhere
(including just on Netflix's Priam project). I think the only thing that would
change significantly is where the per table configuration lives, and I have
offered an option that might be compatible with either design where we put the
table config in a Cassandra table if we wanted to go that way. That also may be
more convenient for people that already have external tools since interfacing
with Cassandra tables is much easier than with table metadata. If you or others
have an alternative suggestion I of course welcome that.
> Scheduled Repair in Cassandra
> -----------------------------
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
> Issue Type: Improvement
> Components: Repair
> Reporter: Joseph Lynch
> Priority: Major
> Labels: CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes
> sense given that it is necessary to give our users eventual consistency. Most
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar),
> which we spoke about last year at NGCC. Given the positive feedback at NGCC
> we focussed on getting it production ready and have now been using it in
> production to repair hundreds of clusters, tens of thousands of nodes, and
> petabytes of data for the past six months. Also based on feedback at NGCC we
> have invested effort in figuring out how to integrate this natively into
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our
> implementation into Cassandra, and have created a [design
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
> showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would
> be greatly appreciated about the interface or v1 implementation features. I
> have tried to call out in the document features which we explicitly consider
> future work (as well as a path forward to implement them in the future)
> because I would very much like to get this done before the 4.0 merge window
> closes, and to do that I think aggressively pruning scope is going to be a
> necessity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
