Xeli commented on a change in pull request #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing (ATLEAST_ONCE) URL: https://github.com/apache/flink/pull/6594#discussion_r257501472
########## File path: docs/dev/connectors/pubsub.md ########## @@ -0,0 +1,110 @@ +--- +title: "Google PubSub" +nav-title: PubSub +nav-parent_id: connectors +nav-pos: 7 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This connector provides a Source and Sink that can read from and write to +[Google PubSub](https://cloud.google.com/pubsub). To use this connector, add the +following dependency to your project: + +{% highlight xml %} +<dependency> + <groupId>org.apache.flink</groupId> + <artifactId>flink-connector-pubsub{{ site.scala_version_suffix }}</artifactId> + <version>{{ site.version }}</version> +</dependency> +{% endhighlight %} + +Note that the streaming connectors are currently not part of the binary +distribution. See +[here]({{ site.baseurl }}/dev/linking.html) +for information about how to package the program with the libraries for +cluster execution. + +#### PubSub Source + +The connector provides a Source for reading data from Google PubSub to Apache Flink. +PubSub has an Atleast-Once guarantee and as such this is the best we can do +right now without adding a lot of extra complexity. + +The class `PubSubSource(…)` has a builder to create PubSubsources. `PubSubSource.newBuilder()` + +There are several optional methods to alter how the PubSubSource is created, +the bare minimum is to provide a google project, pubsub subscription and a way to deserialize the PubSubMessages. Review comment: I don't understand what you are referring to here. Do you mean the capitalization? I've changed those now. I've also updated the documentation with regard to the lastest changes and added a section about backpressure / metrics / tuning. Could you have a look at those as well? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
