[
https://issues.apache.org/jira/browse/FLINK-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692342#comment-16692342
]
ASF GitHub Bot commented on FLINK-9083:
---------------------------------------
vtkstef commented on a change in pull request #6782: [FLINK-9083][Cassandra
Connector] Add async backpressure support to Cassandra Connector
URL: https://github.com/apache/flink/pull/6782#discussion_r234799715
##########
File path:
flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSinkBase.java
##########
@@ -45,73 +48,63 @@
*/
public abstract class CassandraSinkBase<IN, V> extends RichSinkFunction<IN>
implements CheckpointedFunction {
protected final Logger log = LoggerFactory.getLogger(getClass());
+
+ // --------------------------- Cassandra Fields
---------------------------
+
+ private final ClusterBuilder builder;
+
protected transient Cluster cluster;
protected transient Session session;
- protected transient volatile Throwable exception;
- protected transient FutureCallback<V> callback;
+ // ------------------------ Synchronization Fields
------------------------
- private final ClusterBuilder builder;
+ private AtomicReference<Throwable> throwable;
+ private Semaphore semaphore;
+ private Phaser phaser;
Review comment:
Have you considered Guava's rate limiter? If so, I am curious, what made you
choose using Phaser and semaphore together instead of the rate limiter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add async backpressure support to Cassandra Connector
> -----------------------------------------------------
>
> Key: FLINK-9083
> URL: https://issues.apache.org/jira/browse/FLINK-9083
> Project: Flink
> Issue Type: Improvement
> Components: Cassandra Connector
> Reporter: Jacob Park
> Assignee: Jacob Park
> Priority: Minor
> Labels: pull-request-available
>
> As the CassandraSinkBase derivatives utilize async writes, they do not block
> the task to introduce any backpressure.
> I am currently using a semaphore to provide backpressure support by blocking
> at a maximum concurrent requests limit like how DataStax's Spark Cassandra
> Connector functions:
> [https://github.com/datastax/spark-cassandra-connector/blob/v2.0.7/spark-cassandra-connector/src/main/scala/com/datastax/spark/connector/writer/AsyncExecutor.scala#L18]
> This improvement has greatly improved the fault-tolerance of our Cassandra
> Sink Connector implementation on Apache Flink in production. I would like to
> contribute this feature back upstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
