I’m having trouble getting a flume sink to GCS working. There’s not a lot of documentation out there for GCS and Flume.
Does anyone have a GCS sink in Flume working? I can successfully list the contents of the GCS bucket using “gustils” and the values in core-site-xml seem valid, as I can list the contents of the bucket using "hdfs dfs -ls gs://m <gs://m>y-bucket”. This flume sink when configured for an avro sink to HDFS works perfectly, but pointing to a GCS bucket, no data is sent and flume shows no errors. Here’s my GCS sink in flume.conf agent.sinks.HDFSSink.type = hdfs agent.sinks.HDFSSink.hdfs.path = gs://my-bucket/ agent.sinks.HDFSSink.channel = memoryChannel agent.sinks.HDFSSink.hdfs.useLocalTimeStamp = true agent.sinks.HDFSSink.hdfs.rollCount = 0 agent.sinks.HDFSSink.hdfs.rollInterval = 0 agent.sinks.HDFSSink.hdfs.rollSize = 0 agent.sinks.HDFSSink.hdfs.fileType = DataStream agent.sinks.HDFSSink.hdfs.writeFormat = Text agent.sinks.HDFSSink.hdfs.filePrefix = %H-%M-%S agent.sinks.HDFSSink.hdfs.filePrefix = .json
