2020-09-16 13:17:05 UTC - sindhushree: Hi ,
we are using pulsar flink connector and getting the millions of offset in the
trace logs .and leading to state restoration error .
2020-09-16 11:44:39.764 [Source: opr_event ->
c9ee23b has remote state
SubtaskState{operatorStateFromBackend=StateObjectCollection{[OperatorStateHandle{stateNameToPartitionOffsets={topic-partition-offset-states_subName=StateMetaInfo{offsets=[1106,
1127
, 1148, 1169, 1190, 1211, 1232, 1253, 1274, 1295, 1316, 1337, 1358, 1379, 1400,
1421, 1442, 1463, 1484, 1505, 1526, 1547, 1568, 1589, 1610, 1631, 1652, 1673,
1694, 1715, 1736, 1757, 1778, 1799, 1820,
it has like millions of offsets . and we are not able to restore.
----
2020-09-16 15:11:11 UTC - hangc: you may turn the flink log level from trace to
info, and check the state storage.
----
2020-09-16 17:53:43 UTC - sindhushree: Am unable to store the state back
because of large frame size is the error am getting after this .
----
2020-09-16 19:40:56 UTC - sindhushree: Does at all flink purges its offset ?
----
2020-09-16 20:48:30 UTC - Rich Adams: @Rich Adams has joined the channel
----
2020-09-16 22:04:13 UTC - Muljadi: When will Zookeeper removed from Pulsar?
----
2020-09-16 22:14:45 UTC - Rich Adams: Does anyone know if the decision to use
prestosql was intentional? Seems like most documentation I read points at
prestodb, but looking at what is going into master pulsar is going with
prestosql. I don’t know a ton about the presto community, but based on what I
have seen the SPIs are not compatible:
-
<https://github.com/prestodb/presto/blob/master/presto-spi/src/main/java/com/facebook/presto/spi/Plugin.java>
-
<https://github.com/prestosql/presto/blob/master/presto-spi/src/main/java/io/prestosql/spi/Plugin.java>
----
2020-09-16 22:48:23 UTC - Addison Higham: prestosql is a fork from prestodb,
there was some drama around the original stewardship of prestodb and prestosql
was formed with a more clear non profit foundation, and hence is why pulsar
initially went with prestosql as that seemed to be where the community was
headed.
In the meantime, prestodb has now been itself is now in a better position with
a new foundation being created to act as stewards.
(see
<https://blog.openbridge.com/prestodb-vs-prestosql-the-new-presto-foundation-341405fb7ba3>
for some good background)
At this point, AFAICT, it isn't entirely clear what the best step forward is or
if there will be some unification between the two projects.
I think moving to prestodb is certainly something that would be considered if
that is where all the community ends up, but for now it is somewhat of a sticky
situation :confused:
----
2020-09-16 23:08:44 UTC - Rich Adams: Yeah, looks fairly dramatic. So Pulsar
switched to prestosql in June, there isn’t actually a released version of
Pulsar with prestosql in it right? I think v2.6.0 and v2.6.1 both use presto
v0.206 which was still in the com.facebook package.
----
2020-09-17 07:57:37 UTC - Linton: @Linton has joined the channel
----
