Re: Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

2021-08-14 Thread Piotr Nowojski
Hi, FYI, the performance regression after upgrading RocksDB was clearly visible in all of our RocksDB related benchmarks, like for example: http://codespeed.dak8s.net:8000/timeline/?ben=stateBackends.ROCKS&env=2 http://codespeed.dak8s.net:8000/timeline/?ben=stateBackends.ROCKS_INC&env=2 (and many

Exploring Flink for a HTTP delivery service.

2021-08-14 Thread Prasanna kumar
Hi, Aim: Building an event delivery service Scale : Peak load 50k messages/sec. Average load 5k messages/sec Expected to grow every passing month Unique Customer Endpoints : 10k+ Unique events(kafka topics) : 500+ Unique

Running Beam on a native Kubernetes Flink cluster

2021-08-14 Thread Gorjan Todorovski
Hi! I need help implementing a native Kubernetes Flink cluster that needs to run batch jobs (run by TensorFlow Extended), but I am not sure I am configuring it right as I have issues running jobs on more than one task manager, while jobs run fine if there is only one TM. I use the following param

Problems with reading ORC files with S3 filesystem

2021-08-14 Thread Piotr Jagielski
Hi, I want to use Flink SQL filesystem to read ORC file via S3 filesystem on Flink 1.13. My table definition looks like this: create or replace table xxx (..., startdate string) partitioned by (startdate) with ('connector'='filesystem', 'format'='orc', 'path'='s3://xxx/orc/yyy') I followed F

Fwd: PyFlink performance and deployment issues

2021-08-14 Thread Wouter Zorgdrager
Hi all, I'm still dealing with the PyFlink deployment issue as described below. I see that I accidentally didn't forward it to the mailing list. Anyways, my job is stuck in `Initializing` and the logs don't really give me a clue what is going on. In my IDE it runs fine. The command I use to submit