SQL massively more resource-intensive? Memory leak?

2019-06-01 Thread deklanw
Is using Beam SQL just massively more resource-intensive or is there a bug somewhere here (in my code or elsewhere)? (I'm using Flink runner) Here is my code: https://pastebin.com/nNtc9ZaG Here is the error I get (truncated at the end because it's so long and seemingly repetitive) when I run th

Re: SQL massively more resource-intensive? Memory leak?

2019-06-03 Thread deklanw
Yes, it works but I think it has the same problem. It's a lot slower so it took me hours of running it, but by the end the memory usage was high and the CPU about 100% so it seems to be the same problem. Worth noting perhaps that when I use the DirectRunner I have to turn enforceImmutability of

Out of memory when using SQL transform?

2019-04-29 Thread deklanw
I get two different errors, randomly, as far as I can tell: https://pastebin.com/pEvLk81F https://pastebin.com/aKjsTGub Short offending code: https://pastebin.com/kND1Z0Rr I read records from Kafka, deserialize the Avro as GenericRecords, assign my timestamp, convert to Row, then do a simp