Use case is , we want to read/write to kinesis streams using k8s Officially I could not find the connector or reader for kinesis from spark like it has for kafka.
Checking here if anyone used kinesis and spark streaming combination ? On Thu, 6 Apr, 2023, 7:23 pm Mich Talebzadeh, <mich.talebza...@gmail.com> wrote: > Hi Rajesh, > > What is the use case for Kinesis here? I have not used it personally, > Which use case it concerns > > https://aws.amazon.com/kinesis/ > > Can you use something else instead? > > HTH > > Mich Talebzadeh, > Lead Solutions Architect/Engineering Lead > Palantir Technologies > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Thu, 6 Apr 2023 at 13:08, Rajesh Katkar <katkar.raj...@gmail.com> > wrote: > >> Hi Spark Team, >> >> We need to read/write the kinesis streams using spark streaming. >> >> We checked the official documentation - >> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >> >> It does not mention kinesis connector. Alternative is - >> https://github.com/qubole/kinesis-sql which is not active now. This is >> now handed over here - https://github.com/roncemer/spark-sql-kinesis >> >> Also according to SPARK-18165 >> <https://issues.apache.org/jira/browse/SPARK-18165> , Spark officially >> do not have any kinesis connector >> >> We have few below questions , It would be great if you can answer >> >> 1. Does Spark provides officially any kinesis connector which have >> readstream/writestream and endorse any connector for production use cases >> ? >> >> 2. >> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >> This >> documentation does not mention how to write to kinesis. This method has >> default dynamodb as checkpoint, can we override it ? >> 3. We have rocksdb as a state store but when we ran an application >> using official >> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >> rocksdb >> configurations were not effective. Can you please confirm if rocksdb is >> not >> applicable in these cases? >> 4. rocksdb however works with qubole connector , do you have any plan >> to release kinesis connector? >> 5. Please help/recommend us for any good stable kinesis connector or >> some pointers around it >> >>
