> [4] http://status.openlabtesting.org/builds?project=apache%2Fflink
> [5]
> https://lists.apache.org/thread.html/5c4c75a2de979ed7ef1c661c15dd252569e598a374c27042b38d078b%40%3Cdev.flink.apache.org%3E
>
> Best,
> Guowei
>
>
> On Thu, Apr 1, 2021 at 3:55 AM Rex Fenley wrote:
&
also
supports running on ARM.
Thank you
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/cli.html#starting-a-job-from-a-savepoint
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/libs/state_processor_api.html
> On 16/03/2021 20:03, Rex Fenley wrote:
>
> Hello,
>
&
here any better strategies?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
[1].
> Be careful to use correct prefix extractor.
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#passing-options-factory-to-rocksdb
>
>
> Best
>
>
> --
> *From:* Rex Fenley
> *Sent:
sdb/wiki/Prefix-Seek
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
ess that no Kafka topics ever contain CDC data
> in this instance since you are not interested in historic data. The only
> CDC exchange is by using the debezium connector of Flink. At this point,
> all discussions of this thread are resolved.
>
>
>
> On Sat, Feb 27, 2021 at
l). Again, a
> cloud-native key/value store would perform much better and be much cheaper
> with better SLAs and solve all issues on the Flink side (final note: it's
> independent of the technology, any stream processor will encounter the same
> issue as it's a conceptual mis
marketing is strongly pushing that Kafka can be used as a database and as a
> key/value store while in reality, it's "just" a good distribution log. I
> can provide pointers that discuss the limitations if there is interest.
> Also note that the final topic should not be in CDC
D)
>
> Regards,
> Timo
>
>
>
>
> On 24.02.21 14:20, Arvid Heise wrote:
> > Hi Rex,
> >
> > just an idea, wouldn't it be possible to just add
> >
> > |UNIX_TIMESTAMP()
> > |
> >
> > |right before your window operation?|
thing with intermediate flink jobs too.
Would appreciate any feedback on this approach, thanks!
On Fri, Feb 26, 2021 at 10:52 AM Rex Fenley wrote:
> Does this also imply that it's not safe to compact the initial topic where
> data is coming from Debezium? I'd think that Flink&#
lution is to split it into two topics
> (like topic-raw for Flink and topic-compacted for applications) and live
> with some amplification.
>
> On Thu, Feb 25, 2021 at 12:11 AM Rex Fenley wrote:
>
>> All of our Flink jobs are (currently) used for web applications at the
>
rd with
>> latency at most 'delete.retention.ms'. This is configurable per topic -
>> default 24 hours. If you want to reprocess the data later, your consumer
>> might see only resulting compacted ("retracted") stream, and not every
>> record actually written to
Apologies, forgot to finish. If the Kafka source performs its own
retractions of old data on key (user_id) for every append it receives, it
should resolve this discrepancy I think.
Again, is this true? Anything else I'm missing?
Thanks!
On Tue, Feb 23, 2021 at 6:12 PM Rex Fenley wrote:
t message
altogether and produce incorrect results.
Is this true? How do we prevent this from happening? We need to use
compaction since all our jobs are based on CDC and we can't just drop data
after x number of days.
Thanks
--
Rex Fenley | Software Engineer - Mobile and Backe
with my plan?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
p. What throws me off is that I cannot tell how you would tell
> > Flink to store a processing timestamp as is in a row (to essentially
> > create something like ingestion time).
> >
> > I'm looping in Timo to provide some clarity.
> >
> > On 2/19/2021 8:39 A
On Thu, Feb 18, 2021 at 10:55 PM Chesnay Schepler
wrote:
> Could you check whether this answers your question?
>
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/timely-stream-processing.html#notions-of-time-event-time-and-processing-time
>
> On 2/19/2021 7:2
27;s being used
directly by some operator, by some machine?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
hanks!
On Wed, Feb 17, 2021 at 9:33 PM Rex Fenley wrote:
> Also, as an example, I've tried
> table.window(Tumble over 1.seconds on proctime() as $"w")...
> and it failed.
>
> On Wed, Feb 17, 2021 at 9:30 PM Rex Fenley wrote:
>
>> Hi,
>>
>>
Also, as an example, I've tried
table.window(Tumble over 1.seconds on proctime() as $"w")...
and it failed.
On Wed, Feb 17, 2021 at 9:30 PM Rex Fenley wrote:
> Hi,
>
> When using streaming api, if I want a tumbling window on proctime all I
> have to do is th
to switch to data streams because my tables have
retractions and the table api is simpler to use in that sense.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/streaming/time_attributes.html
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.co
and drop rows that were in that
> window.
> >
> > On Mon, Feb 1, 2021 at 6:37 AM Timo Walther > <mailto:twal...@apache.org>> wrote:
> >
> > Hi Rex,
> >
> > processing-time gives you no alignment of operators across nodes.
> Each
> > opera
, etc. It is always a best effort timing.
>
> Regards,
> Timo
>
>
> On 27.01.21 18:16, Rex Fenley wrote:
> > Hello,
> >
> > I'm looking at ways to deduplicate data and found [1], but does proctime
> > get committed with operators? How does this wo
checkpoints are
enabled."
Thanks!
On Sun, Jan 31, 2021 at 3:10 PM Rex Fenley wrote:
> Got it, thanks! What is the 0 B part of that?
>
> On Sun, Jan 31, 2021 at 3:43 AM Arvid Heise wrote:
>
>> Processed in-flight data is the size of data that is processed between
>&
cts/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html#history-tab
>
> On Sun, Jan 31, 2021 at 7:45 AM Rex Fenley wrote:
>
>> Hello,
>>
>> In the latest version I now see two byte measures on checkpoints. There's
>> *Checkpointed Data Size*
>>
t?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
is monotonously increasing.
>
> Best,
>
> Arvid
>
> On Fri, Jan 29, 2021 at 1:14 AM Rex Fenley wrote:
>
>> Switching to TumblingProcessingTimeWindows seems to have solved that
>> problem.
>>
>> For my own understanding, this won't have any "late&qu
in a window and
process them and then the aggregate will take whatever is last in-order.
Thanks!
On Thu, Jan 28, 2021 at 1:01 PM Rex Fenley wrote:
> It looks like it wants me to call assignTimestampsAndWatermarks but I
> already have a timer on my window which I'd expect everything en
It looks like it wants me to call assignTimestampsAndWatermarks but I
already have a timer on my window which I'd expect everything entering this
stream would simply be aggregated during that window
.window(TumblingEventTimeWindows.of(Time.seconds(1)))
On Thu, Jan 28, 2021 at 12:59 PM Rex F
sure how to interpret
On Wed, Jan 27, 2021 at 11:57 PM Rex Fenley wrote:
> Ok, that sounds like it confirms my expectations.
>
> So I tried running my above code and had to slightly edit to using java
> Tuple2 because our execution environment stuff is all in Java.
>
> class
t; // This is a required function that we don't use.
>> override def merge(a: (Boolean, Row), b: (Boolean, Row)) =
>> throw new NotImplementedException()
>> }
>>
>> I'm hoping that if the last record in the window is an insert it picks
>> that
the ES
sink we will simply check true or false in the first element of the tuple
for an insert or delete request to ES. Does this seem like it will work?
Thanks!
On Wed, Jan 27, 2021 at 12:45 PM Rex Fenley wrote:
> This is great info, thanks!
>
> My question then becomes, what constitute
Jark
>
> On Tue, 26 Jan 2021 at 18:33, Dawid Wysakowicz
> wrote:
>
>> I am pulling in Jark and Godfrey who are more familiar with the internals
>> of the planner.
>> On 21/01/2021 01:43, Rex Fenley wrote:
>>
>> Just tested this and I couldn't rest
Dawid Wysakowicz
> wrote:
>
>> I am pulling Jark and Godfrey who are more familiar with the planner
>> internals.
>>
>> Best,
>>
>> Dawid
>> On 22/01/2021 20:11, Rex Fenley wrote:
>>
>> Hello,
>>
>> Does anyone have any m
u set
> *max.in.flight.requests.per.connection
> *to 1
> *. [1]*
> *- *Often you also want to set *enable.idempotence* and *acks=all*
>
> That is true for the upstream application and if you plan back to write to
> Kafka you also need to set that in Flink.
>
> [1]
>
Hello,
I'm looking at ways to deduplicate data and found [1], but does proctime
get committed with operators? How does this work against clock skew on
different machines?
Thanks
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/queries.html#deduplication
--
ard a
retract but not an append and remove the state from RocksDB?
Thanks!
On Fri, Dec 11, 2020 at 9:15 AM Rex Fenley wrote:
> Hi,
>
> Does this question make sense or am I missing something?
>
> Thanks!
>
> On Thu, Dec 10, 2020 at 10:24 AM Rex Fenley wrote:
>
>> Ok,
6, 2021 at 8:53 PM Rex Fenley wrote:
> Our data arrives in order from Kafka, so we are hoping to use that same
> order for our processing.
>
> On Tue, Jan 26, 2021 at 8:40 PM Rex Fenley wrote:
>
>> Going further, if "Flink provides no guarantees about the order of the
>
Our data arrives in order from Kafka, so we are hoping to use that same
order for our processing.
On Tue, Jan 26, 2021 at 8:40 PM Rex Fenley wrote:
> Going further, if "Flink provides no guarantees about the order of the
> elements within a window" then with minibatch, which
21 at 5:36 PM Rex Fenley wrote:
> Hello,
>
> We have a job from CDC to a large unbounded Flink plan to Elasticsearch.
>
> Currently, we have been relentlessly trying to reduce our record
> amplification which, when our Elasticsearch index is near fully populated,
> c
heckpoint maintain our unbounded consistency and
will deletes work)?
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/queries.html#deduplication
Thanks!
--
Rex Fenley |
> metrics.scope.tm: .taskmanager.
> metrics.scope.tm.job: .taskmanager..
> metrics.scope.task:
> .taskmanager
> metrics.scope.operator:
> .taskmanager
>
>
> On 1/26/2021 9:28 PM, Rex Fenley wrote:
>
> All taskmanager and jobmanager logs show up. Anything specific to an
&
g metric that is shown in the Flink UI should also appear in
> DataDog.
> If this is not the case then something goes wrong within the reporter.
>
> Is there anything suspicious in the Flink logs?
>
> Can you give some example of metrics that *do* show up in DataDog?
>
> O
eally zoom in
or out of those metrics or compare against multiple operators at a time
which makes it really difficult to get an overview of how rocks is doing.
Is this there any way to get the Rocks Native Metrics forwarded over to
DataDog?
Thanks!
--
Rex Fenley | Software Engineer - Mobile a
Hello,
Does anyone have any more information here?
Thanks!
On Wed, Jan 20, 2021 at 9:13 PM Rex Fenley wrote:
> Hi,
>
> Our job was experiencing high write amplification on aggregates so we
> decided to give mini-batch a go. There's a few things I've noticed that ar
unctions, does it work for UDAF?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Just tested this and I couldn't restore from a savepoint. If I do a new job
from scratch, can I tune the minibatch parameters and restore from a
savepoint without having to make yet another brand new job?
Thanks
On Wed, Jan 20, 2021 at 12:43 PM Rex Fenley wrote:
> Hello,
>
>
Hello,
Is it safe to convert a non-mini-batch job to a mini-batch job when
restoring from a checkpoint or a savepoint?
Thanks
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <http
stored in the state. I
> would suggest you try it out regardless.
>
> If you want to duplicate a job for some testing, each of the duplicated
> jobs will have it's own sets of offsets and they will read records
> independently, but starting from the same starting point (when the j
gt; during this time, causing higher back pressure, which in turns is causing
> longer checkpointing time for the first checkpoint after recovery. Maybe
> RocksDB is needs to load it's state from disks.
>
> Piotrek
>
> wt., 19 sty 2021 o 20:11 Rex Fenley napisał(a):
>
>
same way as if this would be a freshly
> started new job without any state. Those new offsets would be as
> specified/defined via `auto.offset.reset`.
>
> Piotrek
>
>
> pon., 18 sty 2021 o 18:12 Rex Fenley napisał(a):
>
>> Thank you,
>>
>> Some parts that stic
r, so sometimes they are represented as
> List/Maps on heap.
>
> Regards,
> Timo
>
>
> On 18.01.21 18:28, Rex Fenley wrote:
> > Fascinating, do you have an estimate of what qualifies as a lot of data
> > and therefore when this should be used?
> >
> >
would refrain from using unaligned checkpoints on the production
> because of some bugs [2] that we are fixing right now. On Flink 1.11.x it
> should be fine.
>
> Cheers,
> Piotrek
>
> [1]
> https://flink.apache.org/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.h
Hello,
When we are recovering on a checkpoint it will take multiple minutes. The
time is usually taken by "Start Delay". What is Start Delay and how can we
optimize for it?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/&g
e problem that way.
>
> Best,
>
> Dawid
>
> [1] https://issues.apache.org/jira/browse/FLINK-21009
> On 15/01/2021 21:28, Rex Fenley wrote:
>
> Yes, use the same SQL and change '0's to '-1'. We received "Caused by:
> java.lang.IllegalArgumentExcepti
abstractions.
>
> Regards,
> Timo
>
>
> On 16.01.21 20:09, Rex Fenley wrote:
> > Hello,
> >
> > In the recent version of Flink docs I read the following [1]:
> > > If an accumulator needs to store large amounts of data,
> > |org.apache.flink.table.
ution only
> occurs when performing a GROUP BY or when having operators with
> different parallelism. For Table API and SQL, you can print the
> shuffling steps via `Table.explain()`. They are indicated with an
> `Exchange` operation
>
> I hope this helps.
>
> Regards,
>
Wups, my bad, I should have refreshed. I will close my bug report.
On Mon, Jan 18, 2021 at 9:16 AM Rex Fenley wrote:
> Here you go https://issues.apache.org/jira/browse/FLINK-21014 .
>
> Thanks!
>
> On Mon, Jan 18, 2021 at 1:28 AM Timo Walther wrote:
>
>> Hi Rex,
&g
aving different timeout might be reasonable.
>
> Regards,
> Timo
>
>
> On 17.01.21 02:43, Rex Fenley wrote:
> > Thanks for the quick response.
> >
> > Is this something that can be added as a feature request? Given that the
> > time it takes to restore fro
0 赵一旦 napisał(a):
>
>> If you changed the consumer group in your new job, the group id will be
>> the new one you set.
>> The job will continue to consumer the topics from the
>> savepoint/checkpoint you specified no matter whether the group id is the
>> original
e records.
>
> Best,
> Piotrek
>
> [1] This step can be avoided if you are using local recovery
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#task-local-recovery
>
> sob., 16 sty 2021 o 06:15 Rex Fenley napisał(a):
>
>> A
!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L204
> .
>
> Best,
> Yangze Guo
>
> On Sun, Jan 17, 2021 at 11:33 AM Rex Fenley wrote:
> >
> > I found the following, indicating that there is no concurrency for th
/ElasticsearchSinkBase.java#L382
Does each subtask of an Elasticsearch sink have it's own separate Bulk
Processor to allow for parallel bulk writes?
Thanks!
On Sat, Jan 16, 2021 at 10:33 AM Rex Fenley wrote:
> Hello,
>
> How many connections does the ES connector use to write to Elast
, 2021 at 2:43 PM Khachatryan Roman <
khachatryan.ro...@gmail.com> wrote:
> Hi Rex,
>
> Unfortunately not: the same timeout value is used both for savepoints and
> checkpoints.
>
> Regards,
> Roman
>
>
> On Sat, Jan 16, 2021 at 9:42 AM Rex Fenley wrote:
>
>
on 1.11.2), did we
simply miss this feature? Does it work for 1.11.x too?
Thanks!
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/functions/udfs.html#mandatory-and-optional-methods
[2]
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/a
Hello,
I'm wondering what sort of algorithm flink uses to map an Integer ID to a
subtask when distributing data. Also, what operators from the TableAPI
cause data to be redistributed? I know Joins will, what about Aggregates,
Sources, Filters?
Thanks!
--
Rex Fenley | Software Eng
nfirm this?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
duration than a savepoint.
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
tion though!
On Fri, Jan 15, 2021 at 7:50 PM Rex Fenley wrote:
> Hello,
>
> We have a savepoint that's ~0.5 TiB in size. When we try to restore from
> it, we time out because it takes too long (write now checkpoint timeouts
> are set to 2 hours which is way above where we want th
trieving a savepoint. Are there any
constraining factors in Flink's design that would slow down the network
download of a savepoint this much (from S3)?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.re
> 'index'='table-api',
> 'document-type'='MyType',
> 'hosts'='http://127.0.0.1:9200',
> 'sink.flush-on-checkpoint'='false',
> 'sink.bulk-flush.max-actions'='-1',
> '
'-1' to disable
> it. I created a ticket[1] to track that. And as far as I can tell and I
> quickly checked that it should work. As for the sink.bulk-flush.max-size
> you should be able to disable it with a value of '0'.
>
> Could you share with us how do you use the c
?
On Wed, Jan 13, 2021 at 6:09 PM Rex Fenley wrote:
> Hello,
>
> I have incremental checkpoints turned on and there seems to be no relation
> at all to how often the job checkpoints and how much data exists. Whether
> checkpoints are set to every 1 min or every 5 seconds they'
would expect that
if the system only runs for 5s then it would have significantly less data
to checkpoint than if it runs for 1 min.
Would someone mind clarifying the meaning of checkpoint data size when
incremental checkpoints are turned on? Possibly I'm misinterpreting it.
Thank you!
--
27;sink.bulk-flush.max-size'.
How can I disable these two settings?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
een investigating socket time outs for some time now and are looking
for a way to fix. If things are synchronous that may be a contributing
factor.
Thanks
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/&
27;s been
deprecated in ES 6 and removed in ES 7.
[1] https://github.com/elastic/elasticsearch/pull/38085
On Wed, Jan 13, 2021 at 12:50 AM Aljoscha Krettek
wrote:
> On 2021/01/12 15:04, Rex Fenley wrote:
> >[2]
> >
> https://github.com/apache/flink/blob/97bfd049951f8d52a2e0ae
For further clarity, we're on Elasticsearch 7.
On Tue, Jan 12, 2021 at 4:53 PM Rex Fenley wrote:
> Correction, by HTTP timeout I mean BulkProcessor timeout.
>
> On Tue, Jan 12, 2021 at 4:40 PM Rex Fenley wrote:
>
>> Hello,
>>
>> For the past number of days
Correction, by HTTP timeout I mean BulkProcessor timeout.
On Tue, Jan 12, 2021 at 4:40 PM Rex Fenley wrote:
> Hello,
>
> For the past number of days we've been investigating continuous
> SocketTimeoutException like the following:
>
> 2021-0
github.com/elastic/elasticsearch/blob/67be92c72386848ef52f029248b350ddaba0fd7e/client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java#L213
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http
Should [2] be removed?
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Got it. Thanks!
On Tue, Jan 12, 2021 at 2:12 AM Chesnay Schepler wrote:
> A normal FileAppender that does not do any rolling or limiting of the log
> file.
>
> On 1/12/2021 3:09 AM, Rex Fenley wrote:
>
> Thanks, I'll check them out. What's the default in 1.11.2?
>
multiple input
>>> channels
>>> 1. does NOT block processing to wait for barriers from all inputs,
>>> meaning the task keeps processing data after receiving a barrier even if it
>>> has multiple inputs.
>>> 2. but still, a task takes a snapshot after
coming from Epoch N+1;
> that's where "at least once" comes from.
>
> On Tue, Jan 12, 2021 at 1:03 PM Rex Fenley wrote:
>
>> Hello,
>>
>> We're using the TableAPI and want to optimize for checkpoint alignment
>> times. We received some advice to po
ork for us.
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
eckpoint is still supported in current Flink version.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint
>
> Best
> Yun Tang
> ----------
> *From:* Rex Fenley
> *Sent:* Tuesday
Hello,
When using the TableAPI, is it safe to run a flink job with a different
`-p` parallelism while restoring from a checkpoint (not a savepoint) using
`-s`, without any rescaling of actual machines? I don't seem to find this
documented anywhere.
Thanks!
--
Rex Fenley | Software Eng
n 1.12 .
>
> On 1/12/2021 12:53 AM, Rex Fenley wrote:
>
> Hello,
>
> We've collected over 150 MiB of log lines in 5 days. Is there a way to
> tell Flink to eject log lines after a certain length so we don't eventually
> run out of disk?
>
> ThankS1
Hello,
We've collected over 150 MiB of log lines in 5 days. Is there a way to tell
Flink to eject log lines after a certain length so we don't eventually run
out of disk?
ThankS1
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/&g
Hello,
Is there some proxy to seeing the relative time it takes for records to
make it through an entire job plan? Maybe checkpoint alignment time would
be a proxy for this? Is there metrics for that or something else that would
provide signal here?
Thanks!
--
Rex Fenley | Software Engineer
ou are running out of direct buffer, you will see
>> "java.lang.OutOfMemoryError:
>> Direct buffer memory"
>>
>> On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley wrote:
>>
>>> Thanks for the reply. If what I'm understanding is correct there's no
>>> chance of an
you could run out of the
> budget.
> But Flink is usually properly configuring the memory limits correctly, to
> avoid running into those situations.
>
> tl;dr: you don't need to worry about this.
>
>
> On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley wrote:
>
>
with something like
> kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
> --group my-flink-application.
>
> On Tue, Dec 15, 2020 at 9:39 PM Rex Fenley wrote:
> >
> > Hi,
> >
> > Is there any way to fetch the current kafka topic of
Hi,
Is there any way to fetch the current kafka topic offsets for the kafka
sources for flink?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remind
Hello,
Our job consistently shows
Outside JVM
Type
Count
Used
Capacity
*Direct* 32,839 1.03 GB 1.03 GB
for direct memory.
Is it typical for it to be full? What are the consequences that we may not
be noticing of direct memory being full?
Thanks!
--
Rex Fenley | Software Engineer - Mobile
;
> FLINK-19013 <https://issues.apache.org/jira/browse/FLINK-19013>
>
> On 12/14/2020 6:40 AM, Rex Fenley wrote:
>
> Hi,
>
> Every time I restore from a savepoint it looks like it can take 20+ min to
> restore given the network i/o graphs I'm seeing. However,
flink is in the middle
of restoring or not?
Thanks
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
disks first,
locally before being written to S3.
Is there a way to tell flink or the os to divide this work up across
mounted disks so it's not all up to 1 disk?
Thanks!
On Sat, Dec 12, 2020 at 10:12 AM Rex Fenley wrote:
> Also, small correction from earlier, there are 4 volumes of 25
1 - 100 of 202 matches
Mail list logo
