[jira] [Created] (BEAM-11057) Make BigQueryIO.Read::withTestServices publicly accessible

2020-10-12 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-11057: Summary: Make BigQueryIO.Read::withTestServices publicly accessible Key: BEAM-11057 URL: https://issues.apache.org/jira/browse/BEAM-11057 Project: Beam Issue

[jira] [Created] (BEAM-10523) Add support for custom DatumWriters to AvroIO.Write

2020-07-17 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-10523: Summary: Add support for custom DatumWriters to AvroIO.Write Key: BEAM-10523 URL: https://issues.apache.org/jira/browse/BEAM-10523 Project: Beam Issue Type:

[jira] [Created] (BEAM-10395) Dataflow runner should deduplicate files to stage by destination

2020-06-30 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-10395: Summary: Dataflow runner should deduplicate files to stage by destination Key: BEAM-10395 URL: https://issues.apache.org/jira/browse/BEAM-10395 Project: Beam

[jira] [Resolved] (BEAM-8809) AvroWriteRequest should have a public constructor

2020-06-10 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-8809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz resolved BEAM-8809. - Fix Version/s: 2.17.0 Resolution: Fixed > AvroWriteRequest should have a public constructor

[jira] [Commented] (BEAM-14072) XVR_PythonUsingJava_Dataflow and XVR_PythonUsingJavaSQL_Dataflow failing due to a timeout

2022-03-09 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503717#comment-17503717 ] Steve Niemitz commented on BEAM-14072: -- Yeah I saw this last night, do you have any

[jira] [Commented] (BEAM-14072) XVR_PythonUsingJava_Dataflow and XVR_PythonUsingJavaSQL_Dataflow failing due to a timeout

2022-03-09 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503724#comment-17503724 ] Steve Niemitz commented on BEAM-14072: -- Or I guess alternatively how can I run this

[jira] [Created] (BEAM-14134) Many coders cause significant unnecessary allocations

2022-03-19 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14134: Summary: Many coders cause significant unnecessary allocations Key: BEAM-14134 URL: https://issues.apache.org/jira/browse/BEAM-14134 Project: Beam Issue Type

[jira] [Created] (BEAM-14144) Record JFR profiles when GC thrashing is detected

2022-03-22 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14144: Summary: Record JFR profiles when GC thrashing is detected Key: BEAM-14144 URL: https://issues.apache.org/jira/browse/BEAM-14144 Project: Beam Issue Type: Im

[jira] [Created] (BEAM-13993) Python portable SDK harness does not import beam_plugins at startup

2022-02-23 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-13993: Summary: Python portable SDK harness does not import beam_plugins at startup Key: BEAM-13993 URL: https://issues.apache.org/jira/browse/BEAM-13993 Project: Beam

[jira] [Created] (BEAM-14175) "Read loop was aborted" errors are very noisy when large jobs fail

2022-03-25 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14175: Summary: "Read loop was aborted" errors are very noisy when large jobs fail Key: BEAM-14175 URL: https://issues.apache.org/jira/browse/BEAM-14175 Project: Beam

[jira] [Commented] (BEAM-14239) Changing the output timestamp of a timer does not clear the previously set timer

2022-04-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516840#comment-17516840 ] Steve Niemitz commented on BEAM-14239: -- Here's an example of GroupIntoBatches with 3

[jira] [Created] (BEAM-14239) Changing the output timestamp of a timer does not clear the previously set timer

2022-04-04 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14239: Summary: Changing the output timestamp of a timer does not clear the previously set timer Key: BEAM-14239 URL: https://issues.apache.org/jira/browse/BEAM-14239 Projec

[jira] [Updated] (BEAM-14239) Changing the output timestamp of a timer does not clear the previously set timer

2022-04-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-14239: - Attachment: image-2022-04-04-09-57-29-583.png > Changing the output timestamp of a timer does not

[jira] [Assigned] (BEAM-14244) Processing time timers should use outputTimestamp rather than input watermark for their timestamp

2022-04-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz reassigned BEAM-14244: Assignee: Steve Niemitz > Processing time timers should use outputTimestamp rather than in

[jira] [Created] (BEAM-14244) Processing time timers should use outputTimestamp rather than input watermark for their timestamp

2022-04-04 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14244: Summary: Processing time timers should use outputTimestamp rather than input watermark for their timestamp Key: BEAM-14244 URL: https://issues.apache.org/jira/browse/BEAM-14244

[jira] [Updated] (BEAM-14244) Processing time timers should use outputTimestamp rather than input watermark for their timestamp

2022-04-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-14244: - Status: Open (was: Triage Needed) > Processing time timers should use outputTimestamp rather tha

[jira] [Work started] (BEAM-14244) Processing time timers should use outputTimestamp rather than input watermark for their timestamp

2022-04-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-14244 started by Steve Niemitz. > Processing time timers should use outputTimestamp rather than input watermark

[jira] [Created] (BEAM-14265) Flink should hold the watermark at the output timestamp for processing time timers

2022-04-06 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14265: Summary: Flink should hold the watermark at the output timestamp for processing time timers Key: BEAM-14265 URL: https://issues.apache.org/jira/browse/BEAM-14265 Proj

[jira] [Created] (BEAM-14269) PulsarIOTest.testReadFromSimpleTopic is very flaky

2022-04-06 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14269: Summary: PulsarIOTest.testReadFromSimpleTopic is very flaky Key: BEAM-14269 URL: https://issues.apache.org/jira/browse/BEAM-14269 Project: Beam Issue Type: B

[jira] [Commented] (BEAM-14144) Record JFR profiles when GC thrashing is detected

2022-04-07 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518990#comment-17518990 ] Steve Niemitz commented on BEAM-14144: -- Oh interesting I didn't realize it was backp

[jira] [Commented] (BEAM-14239) Changing the output timestamp of a timer does not clear the previously set timer

2022-04-29 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530100#comment-17530100 ] Steve Niemitz commented on BEAM-14239: -- > is it that the previously set timer does f

[jira] [Commented] (BEAM-14239) Changing the output timestamp of a timer does not clear the previously set timer

2022-04-29 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530132#comment-17530132 ] Steve Niemitz commented on BEAM-14239: -- #1 doesn't work if you have multiple timers

[jira] [Commented] (BEAM-14239) Changing the output timestamp of a timer does not clear the previously set timer

2022-04-29 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530138#comment-17530138 ] Steve Niemitz commented on BEAM-14239: -- agreed they'll both fire, my point was they'

[jira] [Commented] (BEAM-11017) Timer with dataflow runner can be set multiple times (dataflow runner)

2022-05-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531900#comment-17531900 ] Steve Niemitz commented on BEAM-11017: -- I'd bet this is caused by https://issues.apa

[jira] [Comment Edited] (BEAM-11017) Timer with dataflow runner can be set multiple times (dataflow runner)

2022-05-04 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531900#comment-17531900 ] Steve Niemitz edited comment on BEAM-11017 at 5/4/22 7:06 PM: -

[jira] [Created] (BEAM-14534) Add an interface to allow users to compress values being written to shuffle

2022-05-31 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14534: Summary: Add an interface to allow users to compress values being written to shuffle Key: BEAM-14534 URL: https://issues.apache.org/jira/browse/BEAM-14534 Project: Be

[jira] [Work started] (BEAM-14534) Add an interface to allow users to compress values being written to shuffle

2022-05-31 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-14534 started by Steve Niemitz. > Add an interface to allow users to compress values being written to shuffle >

[jira] [Created] (BEAM-14545) Optimize copies in dataflow v1 shuffle reader

2022-06-01 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-14545: Summary: Optimize copies in dataflow v1 shuffle reader Key: BEAM-14545 URL: https://issues.apache.org/jira/browse/BEAM-14545 Project: Beam Issue Type: Improv

[jira] [Updated] (BEAM-14244) Processing time timers should use outputTimestamp rather than input watermark for their timestamp

2022-04-07 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-14244: - Fix Version/s: 2.39.0 Resolution: Fixed Status: Resolved (was: In Progress) >

[jira] [Commented] (BEAM-14269) PulsarIOTest.testReadFromSimpleTopic is very flaky

2022-04-07 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519067#comment-17519067 ] Steve Niemitz commented on BEAM-14269: -- I'm not working on it right now, I had just

[jira] [Commented] (BEAM-14265) Flink should hold the watermark at the output timestamp for processing time timers

2022-04-21 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525916#comment-17525916 ] Steve Niemitz commented on BEAM-14265: -- The output timestamp is ignored yes.  I was

[jira] [Created] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-26 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-12229: Summary: WindmillStateCache has a 0% hit rate in 2.29 Key: BEAM-12229 URL: https://issues.apache.org/jira/browse/BEAM-12229 Project: Beam Issue Type: Bug

[jira] [Commented] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-26 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332791#comment-17332791 ] Steve Niemitz commented on BEAM-12229: -- I was working on bisecting now actually, the

[jira] [Commented] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-26 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332832#comment-17332832 ] Steve Niemitz commented on BEAM-12229: -- Can confirm that [https://github.com/apache/

[jira] [Commented] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-26 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332835#comment-17332835 ] Steve Niemitz commented on BEAM-12229: -- So perhaps even more concerning, looking at

[jira] [Commented] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-27 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333216#comment-17333216 ] Steve Niemitz commented on BEAM-12229: -- Oh interesting, I'll apply that patch and tr

[jira] [Commented] (BEAM-12229) WindmillStateCache has a 0% hit rate in 2.29

2021-04-27 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333285#comment-17333285 ] Steve Niemitz commented on BEAM-12229: -- I can confirm that PR fixed it also, thank y

[jira] [Commented] (BEAM-10670) Make non-portable Splittable DoFn the only option when executing Java "Read" transforms

2021-04-28 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334808#comment-17334808 ] Steve Niemitz commented on BEAM-10670: -- I'd be a big +1 on changing the default back

[jira] [Commented] (BEAM-10670) Make non-portable Splittable DoFn the only option when executing Java "Read" transforms

2021-05-07 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340929#comment-17340929 ] Steve Niemitz commented on BEAM-10670: -- Yeah my issues were with the usability of th

[jira] [Created] (BEAM-12451) BigQueryIO leaks gRPC channels

2021-06-03 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-12451: Summary: BigQueryIO leaks gRPC channels Key: BEAM-12451 URL: https://issues.apache.org/jira/browse/BEAM-12451 Project: Beam Issue Type: Bug Compone

[jira] [Created] (BEAM-12754) RowCoderGenerator calls getValue multiple times

2021-08-12 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-12754: Summary: RowCoderGenerator calls getValue multiple times Key: BEAM-12754 URL: https://issues.apache.org/jira/browse/BEAM-12754 Project: Beam Issue Type: Impr

[jira] [Commented] (BEAM-6855) Side inputs are not supported when using the state API

2021-07-16 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382315#comment-17382315 ] Steve Niemitz commented on BEAM-6855: - Just ran into this again today trying to use a

[jira] [Created] (BEAM-12767) Improve UX of PipelineOptions parsing

2021-08-17 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-12767: Summary: Improve UX of PipelineOptions parsing Key: BEAM-12767 URL: https://issues.apache.org/jira/browse/BEAM-12767 Project: Beam Issue Type: Improvement

[jira] [Created] (BEAM-10991) Timers don't release watermark holds in dataflow on 2.24

2020-09-30 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-10991: Summary: Timers don't release watermark holds in dataflow on 2.24 Key: BEAM-10991 URL: https://issues.apache.org/jira/browse/BEAM-10991 Project: Beam Issue T

[jira] [Created] (BEAM-11728) BigQueryIO should should output records on successful inserts

2021-02-01 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-11728: Summary: BigQueryIO should should output records on successful inserts Key: BEAM-11728 URL: https://issues.apache.org/jira/browse/BEAM-11728 Project: Beam I

[jira] [Created] (BEAM-11801) BigtableIO should not set useCachedDataPool when using an emulator

2021-02-11 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-11801: Summary: BigtableIO should not set useCachedDataPool when using an emulator Key: BEAM-11801 URL: https://issues.apache.org/jira/browse/BEAM-11801 Project: Beam

[jira] [Work started] (BEAM-11801) BigtableIO should not set useCachedDataPool when using an emulator

2021-02-11 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-11801 started by Steve Niemitz. > BigtableIO should not set useCachedDataPool when using an emulator > -

[jira] [Commented] (BEAM-7716) PubsubIO returns empty message bodies for all messages read

2021-03-16 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302830#comment-17302830 ] Steve Niemitz commented on BEAM-7716: - Yes correct.  The experiment flag is "enable_cu

[jira] [Commented] (BEAM-7716) PubsubIO returns empty message bodies for all messages read

2021-03-16 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302847#comment-17302847 ] Steve Niemitz commented on BEAM-7716: - I was using it to try to debug pubsub read perf

[jira] [Created] (BEAM-12126) DirectRunner doesn't respect use_deprecated_reads experiment

2021-04-07 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-12126: Summary: DirectRunner doesn't respect use_deprecated_reads experiment Key: BEAM-12126 URL: https://issues.apache.org/jira/browse/BEAM-12126 Project: Beam Is

[jira] [Commented] (BEAM-10670) Make non-portable Splittable DoFn the only option when executing Java "Read" transforms

2021-04-07 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316858#comment-17316858 ] Steve Niemitz commented on BEAM-10670: -- This ticket seems possibly out of date now,

[jira] [Commented] (BEAM-7717) PubsubIO watermark tracking hovers near start of epoch

2021-03-15 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301955#comment-17301955 ] Steve Niemitz commented on BEAM-7717: - yeah it still seems broken.  No one probably no

[jira] [Commented] (BEAM-7716) PubsubIO returns empty message bodies for all messages read

2021-03-15 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301957#comment-17301957 ] Steve Niemitz commented on BEAM-7716: - it's only reproducible if you don't use windmil

[jira] [Created] (BEAM-10075) Allow users to tune the grouping table size in batch dataflow pipelines

2020-05-25 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-10075: Summary: Allow users to tune the grouping table size in batch dataflow pipelines Key: BEAM-10075 URL: https://issues.apache.org/jira/browse/BEAM-10075 Project: Beam

[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment

2020-05-25 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116181#comment-17116181 ] Steve Niemitz commented on BEAM-9383: - I was just validating the 2.22 branch and this

[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment

2020-05-25 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116185#comment-17116185 ] Steve Niemitz commented on BEAM-9383: - There's actually another much more serious bug

[jira] [Created] (BEAM-10076) Dataflow worker status page incorrectly displays work item statuses

2020-05-25 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-10076: Summary: Dataflow worker status page incorrectly displays work item statuses Key: BEAM-10076 URL: https://issues.apache.org/jira/browse/BEAM-10076 Project: Beam

[jira] [Updated] (BEAM-10076) Dataflow worker status page incorrectly displays work item statuses

2020-05-25 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-10076: - Description: The work item status page incorrectly renders its table due to an incorrectly place

[jira] [Updated] (BEAM-10076) Dataflow worker status page incorrectly displays work item statuses

2020-05-25 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-10076: - Attachment: image-2020-05-25-17-13-49-465.png > Dataflow worker status page incorrectly displays

[jira] [Updated] (BEAM-10076) Dataflow worker status page incorrectly displays work item statuses

2020-05-25 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-10076: - Description: The work item status page incorrectly renders its table due to an incorrectly place

[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment

2020-05-26 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116879#comment-17116879 ] Steve Niemitz commented on BEAM-9383: - {quote}Steve, why would two workers jars be sta

[jira] [Assigned] (BEAM-7568) Java dataflow harness re-encodes value state cells even if they haven't changed

2020-05-26 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz reassigned BEAM-7568: --- Assignee: Steve Niemitz > Java dataflow harness re-encodes value state cells even if they hav

[jira] [Created] (BEAM-13081) Portable representation of "packed bitset indicating null fields" in beam Row format is not compatible with jvm representations

2021-10-19 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-13081: Summary: Portable representation of "packed bitset indicating null fields" in beam Row format is not compatible with jvm representations Key: BEAM-13081 URL: https://issues.apache

[jira] [Created] (BEAM-13459) Dataflow python runner should cache uploaded artifacts across job runs

2021-12-14 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-13459: Summary: Dataflow python runner should cache uploaded artifacts across job runs Key: BEAM-13459 URL: https://issues.apache.org/jira/browse/BEAM-13459 Project: Beam

[jira] [Created] (BEAM-13488) ClassGraph scanner is nondeterministic/incomplete when using parallel scans

2021-12-17 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-13488: Summary: ClassGraph scanner is nondeterministic/incomplete when using parallel scans Key: BEAM-13488 URL: https://issues.apache.org/jira/browse/BEAM-13488 Project: Be

[jira] [Created] (BEAM-13672) Window.into() without a windowFn not correctly translated to portable representation

2022-01-17 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-13672: Summary: Window.into() without a windowFn not correctly translated to portable representation Key: BEAM-13672 URL: https://issues.apache.org/jira/browse/BEAM-13672 Pr

[jira] [Created] (BEAM-13689) BigQueryIO batch writes should output "void" elements when successful

2022-01-19 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-13689: Summary: BigQueryIO batch writes should output "void" elements when successful Key: BEAM-13689 URL: https://issues.apache.org/jira/browse/BEAM-13689 Project: Beam

[jira] [Commented] (BEAM-13672) Window.into() without a windowFn not correctly translated to portable representation

2022-01-20 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479562#comment-17479562 ] Steve Niemitz commented on BEAM-13672: -- ah ok.  it's possible this was just a side-e

[jira] [Comment Edited] (BEAM-13672) Window.into() without a windowFn not correctly translated to portable representation

2022-01-20 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479562#comment-17479562 ] Steve Niemitz edited comment on BEAM-13672 at 1/20/22, 6:32 PM: ---

[jira] [Updated] (BEAM-13672) Window.into() without a windowFn not correctly translated to portable representation

2022-01-21 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-13672: - Resolution: Not A Bug Status: Resolved (was: Open) > Window.into() without a windowFn no

[jira] [Updated] (BEAM-13488) ClassGraph scanner is nondeterministic/incomplete when using parallel scans

2022-01-11 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-13488: - Fix Version/s: 2.36.0 Resolution: Fixed Status: Resolved (was: Open) > ClassGr

[jira] [Updated] (BEAM-13488) ClassGraph scanner is nondeterministic/incomplete when using parallel scans

2022-01-11 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-13488: - Status: Resolved (was: Resolved) > ClassGraph scanner is nondeterministic/incomplete when using

[jira] [Updated] (BEAM-13459) Dataflow python runner should cache uploaded artifacts across job runs

2022-01-11 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-13459: - Fix Version/s: 2.36.0 Resolution: Fixed Status: Resolved (was: Triage Needed)

[jira] [Commented] (BEAM-9308) Optimize state cleanup at end-of-window

2022-01-12 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474592#comment-17474592 ] Steve Niemitz commented on BEAM-9308: - The attached PR is closed, we could never reach

[jira] [Updated] (BEAM-12126) DirectRunner doesn't respect use_deprecated_reads experiment

2022-01-12 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-12126: - Resolution: Fixed Status: Resolved (was: Open) > DirectRunner doesn't respect use_deprec

[jira] [Updated] (BEAM-12754) RowCoderGenerator calls getValue multiple times

2022-01-12 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-12754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-12754: - Resolution: Fixed Status: Resolved (was: Open) > RowCoderGenerator calls getValue multip

[jira] [Updated] (BEAM-11057) Make BigQueryIO.Read::withTestServices publicly accessible

2022-01-12 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-11057: - Resolution: Fixed Status: Resolved (was: Open) > Make BigQueryIO.Read::withTestServices

[jira] [Commented] (BEAM-7605) Provide a way for user code to read dataflow runner stats

2022-01-12 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474595#comment-17474595 ] Steve Niemitz commented on BEAM-7605: - This is not resolved, the attached PR was never

[jira] [Commented] (BEAM-6855) Side inputs are not supported when using the state API

2022-01-12 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17474597#comment-17474597 ] Steve Niemitz commented on BEAM-6855: - this is not resolved > Side inputs are not sup

[jira] [Created] (BEAM-7684) WindmillStateCache grossly misunderestimates object size in its weighting function

2019-07-03 Thread Steve Niemitz (JIRA)
Steve Niemitz created BEAM-7684: --- Summary: WindmillStateCache grossly misunderestimates object size in its weighting function Key: BEAM-7684 URL: https://issues.apache.org/jira/browse/BEAM-7684 Project:

[jira] [Commented] (BEAM-7716) PubsubIO returns empty message bodies for all messages read

2019-07-10 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882640#comment-16882640 ] Steve Niemitz commented on BEAM-7716: - I did a little debugging tonight and tracked th

[jira] [Comment Edited] (BEAM-7716) PubsubIO returns empty message bodies for all messages read

2019-07-10 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882640#comment-16882640 ] Steve Niemitz edited comment on BEAM-7716 at 7/11/19 4:14 AM: --

[jira] [Commented] (BEAM-7716) PubsubIO returns empty message bodies for all messages read

2019-07-11 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882993#comment-16882993 ] Steve Niemitz commented on BEAM-7716: - Hacking up PubsubMessagePayloadOnlyCoder to alw

[jira] [Commented] (BEAM-7717) PubsubIO watermark tracking hovers near start of epoch

2019-07-11 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882998#comment-16882998 ] Steve Niemitz commented on BEAM-7717: - [https://github.com/apache/beam/blob/5b68aae06c

[jira] [Commented] (BEAM-7718) PubsubIO to use gRPC API instead of JSON REST API

2019-07-11 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883006#comment-16883006 ] Steve Niemitz commented on BEAM-7718: - One thing I did notice using the gRPC client is

[jira] [Commented] (BEAM-7718) PubsubIO to use gRPC API instead of JSON REST API

2019-07-11 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883084#comment-16883084 ] Steve Niemitz commented on BEAM-7718: - Additionally, the gRPC implementation seems eff

[jira] [Commented] (BEAM-7718) PubsubIO to use gRPC API instead of JSON REST API

2019-07-12 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883837#comment-16883837 ] Steve Niemitz commented on BEAM-7718: - Looks like there was a change very recently ([

[jira] [Closed] (BEAM-7718) PubsubIO to use gRPC API instead of JSON REST API

2019-07-12 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-7718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz closed BEAM-7718. --- Resolution: Done Fix Version/s: 2.14.0 > PubsubIO to use gRPC API instead of JSON REST API > --

[jira] [Created] (BEAM-7745) StreamingSideInputDoFnRunner/StreamingSideInputFetcher have suboptimal state access pattern during normal operation

2019-07-15 Thread Steve Niemitz (JIRA)
Steve Niemitz created BEAM-7745: --- Summary: StreamingSideInputDoFnRunner/StreamingSideInputFetcher have suboptimal state access pattern during normal operation Key: BEAM-7745 URL: https://issues.apache.org/jira/brows

[jira] [Commented] (BEAM-2680) Improve scalability of the Watch transform

2019-08-04 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899655#comment-16899655 ] Steve Niemitz commented on BEAM-2680: - I ran into this today.  We have a long-running

[jira] [Commented] (BEAM-2680) Improve scalability of the Watch transform

2019-08-05 Thread Steve Niemitz (JIRA)
[ https://issues.apache.org/jira/browse/BEAM-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900133#comment-16900133 ] Steve Niemitz commented on BEAM-2680: - Sharding would delay the inevitable more, but t

[jira] [Created] (BEAM-9795) Support custom avro DatumWriters when writing to BigQuery

2020-04-21 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-9795: --- Summary: Support custom avro DatumWriters when writing to BigQuery Key: BEAM-9795 URL: https://issues.apache.org/jira/browse/BEAM-9795 Project: Beam Issue Type

[jira] [Work started] (BEAM-9795) Support custom avro DatumWriters when writing to BigQuery

2020-04-21 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9795 started by Steve Niemitz. --- > Support custom avro DatumWriters when writing to BigQuery > -

[jira] [Updated] (BEAM-9795) Support custom avro DatumWriters when writing to BigQuery

2020-04-21 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz updated BEAM-9795: Status: Open (was: Triage Needed) > Support custom avro DatumWriters when writing to BigQuery > ---

[jira] [Resolved] (BEAM-9795) Support custom avro DatumWriters when writing to BigQuery

2020-05-08 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Niemitz resolved BEAM-9795. - Fix Version/s: 2.22.0 Resolution: Fixed > Support custom avro DatumWriters when writing to

[jira] [Created] (BEAM-9931) Support custom avro DatumReaders in AvroIO

2020-05-08 Thread Steve Niemitz (Jira)
Steve Niemitz created BEAM-9931: --- Summary: Support custom avro DatumReaders in AvroIO Key: BEAM-9931 URL: https://issues.apache.org/jira/browse/BEAM-9931 Project: Beam Issue Type: Improvement

[jira] [Commented] (BEAM-1589) Add OnWindowExpiration method to Stateful DoFn

2020-01-13 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014335#comment-17014335 ] Steve Niemitz commented on BEAM-1589: - this is really cool!  Both of these features ar

[jira] [Commented] (BEAM-9557) Error setting processing time timers near end-of-window

2020-03-23 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065090#comment-17065090 ] Steve Niemitz commented on BEAM-9557: - imo this should be a blocker for 2.20.  It brea

[jira] [Commented] (BEAM-9557) Error setting processing time timers near end-of-window

2020-03-23 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065220#comment-17065220 ] Steve Niemitz commented on BEAM-9557: - I had actually tried that (for other reasons) i

[jira] [Commented] (BEAM-9557) Error setting processing time timers near end-of-window

2020-03-24 Thread Steve Niemitz (Jira)
[ https://issues.apache.org/jira/browse/BEAM-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066280#comment-17066280 ] Steve Niemitz commented on BEAM-9557: - The default case will work as you mentioned, bu

  1   2   >