- [Kinesis stream name]: The Kinesis stream that this streaming
application receives from
- The application name used in the streaming context becomes the
Kinesis application name
- The application name must be unique for a given account and region.
- The Kinesis backend automatically associates the application name
to the Kinesis stream using a DynamoDB table (always in the us-east-1
region) created during Kinesis Client Library initialization.
- *Changing the application name or stream name can lead to Kinesis
errors in some cases. If you see errors, you may need to manually delete
the DynamoDB table.*
On Fri, May 8, 2015 at 2:06 PM, Mike Trienis <[email protected]>
wrote:
> Hi All,
>
> I am submitting the assembled fat jar file by the command:
>
> bin/spark-submit --jars /spark-streaming-kinesis-asl_2.10-1.3.0.jar
> --class com.xxx.Consumer xxxx-0.1-SNAPSHOT.jar
>
> It reads the data file from kinesis using the stream name defined in a
> configuration file. It turns out that it reads the data perfectly fine for
> one stream name (i.e. the default), however, if I switch the stream name
> and re-submit the jar, it no longer reads the data. From CloudWatch, I can
> see that there is data put into the stream and spark is actually sending
> get requests as well. However, it doesn't seem to be outputting the data.
>
> Has anyone else encountered a similar issue? Does spark cache the stream
> name somewhere? I also have checkpointing enabled as well.
>
> Thanks, Mike.
>
>
>
>
>
>
