[GitHub] [flink] haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing (ATLEAST_ONCE)

[GitBox]

haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source 
connector with support for checkpointing (ATLEAST_ONCE)
URL: https://github.com/apache/flink/pull/6594#issuecomment-488239680
 
 
   The checkpoints take e.g. `12528 ms` to write, but they are not large at all:
   
   <img width="395" alt="Screenshot 2019-05-01 at 11 01 51" 
src="https://user-images.githubusercontent.com/193115/57011163-8c3b3100-6c00-11e9-88de-5e3f59086ab5.png";>
   
   I put the checkpoint period to:
   
   ```
           env.enableCheckpointing(9000); // checkpoint every N seconds
           env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2000);
   ```
   
   But I've tried 1000 and 1500 respectively, as well with approximately the 
same results.
   
   The creation looks like this:
   
   ```java
           final String subscriptionName = params.get("pubsub-subscription", 
Env.getOrElse("ANALYTICS_PUBSUB_SUBSCRIPTION", "logs-analytics-dev"));
   
           String jsonFilePath =
               Optional
                   .ofNullable(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))
                   .orElse("./analytics-dev.json");
   
           if (! new File(jsonFilePath).exists()) {
               throw new Exception(String.format("Couldn't find path: '%s'", 
jsonFilePath));
           }
           Credentials credentials = GoogleCredentials.fromStream(new 
FileInputStream(jsonFilePath));
           logger.info("Using Credential JSON: {}, subscription name: {}", 
jsonFilePath, subscriptionName);
   
           PubSubSource<AppEvent> source = PubSubSource.newBuilder(new 
AppEventSerializer(), projectId, 
subscriptionName).withCredentials(credentials).build();
           return env.addSource(source).uid(String.format("pubsub-%s", 
subscriptionName));
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services