Hi Marcin,
I'm having a little trouble understanding this. I think this is a
summary of your problem statement: You have a pipeline that windows
data on event time. Your event generator has an artificial 30 second
delay. The pipeline appears to be experiencing a 10-20 second delay
instead of the e
+d...@beam.apache.org
I tried reproducing this but was not successful, the output schema was as
expected. I added the following to BeamSqlMultipleSchemasTest.java at head.
(I did discover that PAssert.that(result).containsInAnyOrder(output)
doesn't validate column names however.)
@Test
publ
Yes, that worked.
The issue does not occur if I disable all of the following planner rules:
CoreRules.FILTER_CALC_MERGE, CoreRules.PROJECT_CALC_MERGE,
LogicalCalcMergeRule.INSTANCE (which wraps CoreRules.CALC_MERGE),
and BeamCalcMergeRule.INSTANCE (which wraps CoreRules.CALC_MERGE).
All the rules
Hi Talat,
I did get your test case running and added some logging to
RexProgramBuilder.mergePrograms. There is only one merge that occurs during
the test and it has an output type of RecordType(JavaType(int) ID,
JavaType(class java.lang.String) V). This does seem like the correct output
name but i
Also this is at very least a Beam bug. You can file a Beam issue if you
want, otherwise I will when I get back.
Andrew
On Fri, Jan 27, 2023 at 11:27 AM Andrew Pilloud wrote:
> Hi Talat,
>
> I did get your test case running and added some logging to
> RexProgramBuilder.mergePrograms. There is on
I would guess that you have some existing code that expects random IO
access to the files via the Java IO or NIO interface (the common blocking
IO in a DoFn pattern), so using a Beam IO which is what we recommend and
are discussing here would be a significant rewrite?
I worked on Isilon from 6.5 -
It looks like Calcite stopped considering field names in RelNode equality
as of Calcite 2.22 (which we use in Beam v2.34.0+). This can result in a
planner state where two nodes that only differ by field name are considered
equivalent.
I have a fix for Beam in https://github.com/apache/beam/pull/25
This sounds reasonable to me as well.
I've made swaps like this in the past, the base image of each is probably a
bigger factor than the JDK. The openjdk images were based on Debian 11. The
default eclipse-temurin images are based on Ubuntu 22.04 with an alpine
option. Ubuntu is a Debian derivativ
I have a test case that I believe should reproduce this on both head
and 2.43 but it ends up with a different logical plan. Can you provide your
input types?
We have a class of issues around compex types
https://github.com/apache/beam/issues/19009 I don't believe the
"LogicalFilter(condition=[=($2
Hi Talat,
I managed to turn your test case into something against Calcite. It
looks like there is a bug affecting tables that contain one or more
single element structs and no multi element structs. I've sent the
details to the Calcite mailing list here.
https://lists.apache.org/thread/tlr9hsmx09b
set plannerName doesn't actually do anything on the SQL shell at query
parse time, it will still use the calcite parser. Have you tried calcite
SQL?
Support for struts is somewhat limited. I know there are bugs around nested
structs and structs with single values.
Andrew
On Thu, Apr 20, 2023 at
It means SQL doesn't support the "OneOf" type.
On Mon, Apr 24, 2023 at 1:42 AM Jeff Zhang wrote:
>
> Hi all,
>
> I got the following error when running sql: select * from PCOLLECTION,
> what does this mean?
>
>
> Exception in thread "main"
> org.apache.beam.sdk.extensions.sql.impl.ParseException
12 matches
Mail list logo
