Re: Full Checkpoint information in UI and API

Hi, I've updated the PR [https://github.com/apache/flink/pull/25899] to follow the incremental recommendation. It is short a test or two but otherwise should be ready to review. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?ut

[jira] [Created] (FLINK-37008) Flink UI should show the type of checkpoint (full vs incremental)

Ryan van Huuksloot created FLINK-37008: -- Summary: Flink UI should show the type of checkpoint (full vs incremental) Key: FLINK-37008 URL: https://issues.apache.org/jira/browse/FLINK-37008

Re: Log4j JSONTemplateLayout

I've opened a ticket and a PR. Ticket: https://issues.apache.org/jira/browse/FLINK-36997 PR: https://github.com/apache/flink/pull/25888 It should be easy to include in the Flink 2.0 release. If someone could help with review that would be great. Thanks, Ryan van Huuksloot Sr. Produ

[jira] [Created] (FLINK-36997) Add Log4j JSONTemplateLayout to Flink-Dist

Ryan van Huuksloot created FLINK-36997: -- Summary: Add Log4j JSONTemplateLayout to Flink-Dist Key: FLINK-36997 URL: https://issues.apache.org/jira/browse/FLINK-36997 Project: Flink Issue

Full Checkpoint information in UI and API

is already a checkpoint_type for aligned vs unaligned. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>

Log4j JSONTemplateLayout

ing the additional dependency means we don't need to maintain versions across different distributions. Also, at 230KB, it is a tiny dependency that makes it very easy for users to start using a production grade logging system. Thoughts? If there aren't any objections we can open a ticket

[jira] [Created] (FLINK-36884) GCS 503 Error Codes for `flink-checkpoints//shared/` file after upload complete

Ryan van Huuksloot created FLINK-36884: -- Summary: GCS 503 Error Codes for `flink-checkpoints//shared/` file after upload complete Key: FLINK-36884 URL: https://issues.apache.org/jira/browse/FLINK-36884

Re: [Discussion] Session Clusters Support Heterogeneous Task Manager Images

test streaming jobs. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Thu, Dec 5, 2024 at 2:56 AM Dian Fu wrote: > Hi Ryan, > > It supports configuring the Python dependencie

Re: [Discussion] Session Clusters Support Heterogeneous Task Manager Images

different workloads on this session cluster may require different dependencies / versions. Maybe it is simpler to provide a way to dynamically provide dependencies in Python - similar to Java? (I haven't use the jar submission in Java) Thanks, Ryan van Huuksloot Sr. Production Engineer | Str

Re: [Discussion] Session Clusters Support Heterogeneous Task Manager Images

obs work well on Session Clusters. However, due to the differing images you cannot run different workloads on the same session cluster. Making the session cluster essentially useless. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_

[Discussion] Session Clusters Support Heterogeneous Task Manager Images

image discovery. It would also mean that the operator/api would need the ability to pass different images with the job. Thanks, Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>

[jira] [Created] (FLINK-34991) Flink Operator ClassPath Race Condition Bug

Ryan van Huuksloot created FLINK-34991: -- Summary: Flink Operator ClassPath Race Condition Bug Key: FLINK-34991 URL: https://issues.apache.org/jira/browse/FLINK-34991 Project: Flink

Unable to override PrometheusReporter using PrometheusGateway 'filterLabelValueCharacters'

x27;ll also open a JIRA ticket as soon as someone validates that it doesn't work as intended. Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

I can volunteer to be a release manager. I haven't done it for Apache/Flink or the operator before so I may be a good candidate. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>

[jira] [Created] (FLINK-33803) Add observedGeneration to Operator's status spec

Ryan van Huuksloot created FLINK-33803: -- Summary: Add observedGeneration to Operator's status spec Key: FLINK-33803 URL: https://issues.apache.org/jira/browse/FLINK-33803 Project:

Flink Kubernetes Operator Observed Generation

-implementing-observedgeneration-250728868792 I think we keep the current implementation of the reconciliation spec for now but at the same time, we can update the `status.observedGeneration`? I'll open an issue if folks agree with this addition. Ryan van Huuksloot Sr. Production Eng

Kubernetes Operator Managed Ingress with TLS

open source. Happy to move the discussion to Jira. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

Sounds good - I haven't looked too much into the comparison of the old and new client. Opened a ticket: https://issues.apache.org/jira/browse/FLINK-32508 I may have some time to support this ticket in 2 weeks but anyone can feel free to start working on it. Thanks, Ryan van Huukslo

[jira] [Created] (FLINK-32508) Flink-Metrics Prometheus - Native Histograms / Native Counters

Ryan van Huuksloot created FLINK-32508: -- Summary: Flink-Metrics Prometheus - Native Histograms / Native Counters Key: FLINK-32508 URL: https://issues.apache.org/jira/browse/FLINK-32508 Project

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

a new package to keep things distinct. Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Fri, Jun 30, 2023 at 10:02 AM Martijn Visser wrote: > Hi Patrick, > > Yeah

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

Hi Martijn, Option 2 and 3 would use a single client. It would just register the metrics differently. Does that make sense? Does that change your perspective? Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_med

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

). I lean towards option 4, and we would communicate the change internally as part of a minor version upgrade. I'm open to other ideas and would welcome further discussion on what the OSS community prefers. Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopi

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

Following up, any interest in flink-metrics-prometheus? It is quite a stale package. I would be interested in contributing - time permitting. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=h

Re: [VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

+1 (non-binding) +1 for DynamicKafkaSource Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Mon, Jun 19, 2023 at 8:15 AM Martijn Visser wrote: > +1 (bindin

Flink-Metrics Prometheus - Native Histograms / Native Counters

that there was a closed PR for the simpleclient: https://github.com/apache/flink/pull/21047 - which has what is required for the native metrics but may cause other maintenance tickets. Is there any appetite from the community to update this exporter? Thanks, Ryan van Huuksloot Sr. Production

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

icPartition won't work because the topic may be the same name as this is something that is supported IIRC - I chose to pass a map into starting offsets just for demonstrative purposes, I would be fine with whatever data structure would work best Ryan van Huuksloot Data Developer | Productio

RE: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

Hi Mason, First off, thanks for putting this FLIP together! Sorry for the delay. Full disclosure Mason and I chatted a little bit at Flink Forward 2022 but I have tried to capture the questions I had for him then. I'll start the conversation with a few questions: 1. The concept of streamIds is n

RE: [DISCUSS] Contribution of Multi Cluster Kafka Source

Hi Mason, First off, thanks for putting this FLIP together! Sorry for the delay. Full disclosure Mason and I chatted a little bit at Flink Forward 2022 but I have tried to capture the questions I had for him then. I'll start the conversation with a few questions: 1. The concept of streamIds is n

RE: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

are working with that is public? Could you attach that so we can start looking at it? Let us know if you need anything from us to help move this forward. Thanks! Ryan van Huuksloot Data Developer | Data Platform Engineering | Streaming Capabilities [image: Shopify] <https://www.shopify.