[
https://issues.apache.org/jira/browse/BEAM-12164?focusedWorklogId=750614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-750614
]
ASF GitHub Bot logged work on BEAM-12164:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Mar/22 23:45
Start Date: 30/Mar/22 23:45
Worklog Time Spent: 10m
Work Description: nancyxu123 opened a new pull request #17222:
URL: https://github.com/apache/beam/pull/17222
The overall project is to remove watermark updates to the metadata table
from the Connector. To do so, we need to create a new restriction tracker that
keeps track of partition token state, as well as timestmap.
The purpose of this PR is to create the new restriction tracker that keeps
track of both state and timestamp information.
The overall algorithm to remove watermark updates to the connector metadata
table is here:
https://docs.google.com/document/d/1YqpR2iiHV5ESPBg24y_BUuSKYcsgYKDhpxyNoAUzP68/edit?resourcekey=0-m4Rif7BtMjN4OxTzXbMC3g&disco=AAAAWDcj4KU
The design for the new restriction tracker is here:
https://docs.google.com/document/d/1lSLOPSiJFUV2Tr7plSQXn-DcyBNgV5ELJ8p9PRpcO2c/edit?resourcekey=0-tXh0F0I9HK30XV9a7hnfiw
A previous implementation for this restriction tracker is here:
https://github.com/thiagotnunes/beam/tree/spanner-cdc/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/cdc/restriction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 750614)
Time Spent: 63h (was: 62h 50m)
> SpannerIO Change Stream Connector
> ---------------------------------
>
> Key: BEAM-12164
> URL: https://issues.apache.org/jira/browse/BEAM-12164
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Thiago Nunes
> Assignee: Thiago Nunes
> Priority: P2
> Fix For: 2.37.0
>
> Time Spent: 63h
> Remaining Estimate: 0h
>
> We would like to augment the existing Google Cloud SpannerIO connector
> ([https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java)]
> with the support for Spanner Change Streams (CDC). CDC support is just being
> implemented in Spanner and it will be exposed through a gRPC API. We will use
> such API to create a new SpannerIO.readChangeStream(...) implementation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
