stczwd commented on a change in pull request #1332: Doris sink for Spark Structured Streaming URL: https://github.com/apache/incubator-doris/pull/1332#discussion_r296193717
########## File path: docs/documentation/cn/extending-doris/Spark-sink-for-Doris.md ########## @@ -0,0 +1,129 @@ +# Spark sink For Doris + +## Introduction +Doris Sink is a type of Structured Streaming sink with exactly-once fault-tolerance guarantees. It now supports two mode: Broker LOAD & Mini LOAD. + +- **Broker LOAD**: the data will write files on hdfs and Doris use hdfs broker to load the files in. Suitable for the scenario that the data of each batch is larger than 1GB. +- **Mini LOAD**: the data will send to Doris by HTTP PUT request. Suitable for the scenario that the data of each batch is less than 1GB (**Advice mode**) . + +We will support the third mode **Streaming LOAD** recently which has higher performance than the two mode reminded above. + +Note: Doris is also named ***Palo*** in Baidu +## How to build Doris jar + +In `incubator-doris/extension/spark-sink`, run the command `mvn clean install -DskipTests`, which will produce a jar archieve like`spark-sql-palo_2.11-2.4.2.jar` and also install it into local maven repository, then copy the jar to your spark-client/jars + +## How to write first App with Doris Sink +### dependency +Add below dependency into the pom.xml in your project. + +```html +<dependency> + <groupId>org.apache.spark</groupId> + <artifactId>spark-sql-palo_2.11</artifactId> Review comment: palo -> doris ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org For additional commands, e-mail: dev-h...@doris.apache.org
