/72870074/apache-flink-restoring-state-from-checkpoint-with-changes-kafka-topic
>>> I think you'll need to change the UID for your KafkaSource and restart your
>>> job with allowNonRestoredState enabled.
>>>
>>> Best regards,
>>>
>>> Martijn
scriber.java:57)
...
10 moreCaused by:
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition.
--
Robert Cullen
240-475-4490
met then send an alert?
>
> If that is the case, you can use a UDF for checking and sending alerts.
> See [1] for detailed explanation about UDF.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/functions/udfs/
>
> Robert Cullen 于2021年12月14日
Hello,
I'm looking for some guidance on how to send alert notifications from a
DataStream to a Slack channel and possibly other alerting tools (ie. AWS
sns, mattermost)
--
Robert Cullen
240-475-4490
Does flink have a connector for TCP/IP messages?
--
Robert Cullen
240-475-4490
consuming update changes which is produced by node
GroupAggregate(groupBy=[logId, s3Path, message, hostname], select=[logId,
s3Path, message, hostname, MAX(insertTime) AS insertTime])
```
--
Robert Cullen
240-475-4490
a:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:624)
at java.lang.Thread.run(Thread.java:748)
--
Robert Cullen
240-475-4490
.. which is weird, because "setUnbounded" should not make
> something bounded?!
>
> Are there any log messages from the Source that can give us any hints?
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#boundedness
>
&
(Arrays.asList("fluentd"))
.setGroupId("")
.setDeserializer(new FluentdRecordDeserializer())
//.setStartingOffsets(OffsetsInitializer.earliest())
//.setBounded(OffsetsInitializer.latest())
.setUnbounded(OffsetsInitializer.latest())
.bui
build a
collection for caching. Then query the cache with an ID that is being
streamed. But it appears that you cannot mix APIs (Streaming, Table).
Any ideas? thanks!
--
Robert Cullen
240-475-4490
args: ["jobmanager", "-Ds3.endpoint=https://192.173.0.0:9000";,
"-Ds3.access-key=key", "Ds3.secret-key=secret"]
ports
Robert Cullen
240-475-4490
sql to query the data. The current output adds open/closed parentheses
around the row. Is there a way to format this without the parens?
(00136627-8e1e-4c84-9d8d-b6cfe9d092aa,1)
(00136627-8e1e-4c84-9d8d-b6cfe9d092aa,2)
--
Robert Cullen
240-475-4490
on in MINIO is
missing? Any ideas?
--
Robert Cullen
240-475-4490
>
>> A remedy for application states larger than your available RAM is to use
>> the RocksDB State backend, which allows for states larger than your
>> application RAM. But that requires your kubernetes nodes to be equipped
>> with a fast hard drive (SSD, optimally).
>&g
onding taskmanager/jobmanager
> logs. That would help in finding out why the savepoint creation failed.
> Just to verify: The savepoint data wasn't written to S3 even after the
> timeout happened, was it?
>
> Best,
> Matthias
>
> On Thu, May 27, 2021 at 7:50 PM Robert
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at
org.apache.flink.client.cli.CliFrontend.triggerSavepoint(CliFrontend.java:771)
... 7 more
root@flink-client:/opt/flink#
--
Robert Cullen
240-475-4490
g?
>
> From the first looks of it, I have seen this behaviour mostly in cases
> where one or more taskmanagers shut down due to GarbageCollection issues or
> OutOfMemory-Errors.
>
>
> Best regards
>
> Jan
> On 27.05.21 16:44, Robert Cullen wrote:
>
> I have
TaskManager with id 10.42.0.49:6122-e26293 timed out.
at
org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1299)
... 27 more
--
Robert Cullen
240-475-4490
th-style-access
>
> Thanks,
> Youngwoo
>
> 2021년 5월 17일 (월) 오후 10:47, Robert Cullen 님이 작성:
>
>> Arvid, Is there a way to set environment variables in the
>> flink-conf.yaml? Setting them on the CLI isn't working.
>>
>> On Sat, May 15, 2021 at 3:56 PM Ar
ink/blob/master/flink-end-to-end-tests/test-scripts/test_batch_wordcount.sh#L42
>
> On Fri, May 14, 2021 at 6:48 PM Robert Cullen
> wrote:
>
>> I’m testing out my flink 1.13.0 cluster with the latest Minio
>> RELEASE.2021-04-06T23-11-00Z but flink fails to connect to the bucke
or Code: 400 Bad Request;
Request ID: null; S3 Extended Request ID: null; Proxy: null), S3
Extended Request ID: null
--
Robert Cullen
240-475-4490
>> manually specify rest.address, rest.port for the JobManager address. If
>> you can manually figure out an address to the JobManager service, and pass
>> it to Flink, the submission should work.
>>
>> On Wed, May 5, 2021 at 7:15 PM Robert Cullen
>> wrote:
>&
rmissions
> (through a service account) to make such calls to K8s. My hope is that the
> logs or previous messages are giving an indication into what Flink is
> trying to do.
> Can you also try running on DEBUG log level? (should be the
> log4j-cli.properties file).
>
>
>
)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.flink.client.deployment.ClusterRetrieveException:
Could not get the rest endpoint of job-manager
... 9 more
root@flink-client:/opt/flink#
--
Robert Cullen
240-475-4490
I've been using the Community Edition v2.4. Just wondering if there is a
python integration coming in future versions.
tnanks
--
Robert Cullen
240-475-4490
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/streamfile_sink.html#important-considerations
>
> Best,
> Guowei
>
>
> On Thu, Apr 1, 2021 at 3:56 AM Robert Cullen
> wrote:
>
>> I’m using a local instance of MIN
st_2.12-1.12.2-stream1.jar:1.12.2-stream1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
~[flink-dist_2.12-1.12.2-stream1.jar:1.12.2-stream1]
--
Robert Cullen
240-475-4490
build a portable python environment:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/faq.html#preparing-python-virtual-environment
>
> Best,
> Wei
>
> 在 2021年2月24日，01:38，Roman Khachatryan 写道：
>
> Hi,
>
> I'm pulling in Wei Zhong and Xingbo
ge?
>
> On Thu, Mar 25, 2021 at 7:17 PM Robert Cullen
> wrote:
>
>> When I run a job on my Kubernetes session cluster only the checkpoint
>> directories are created but not the savepoints. (Filesystem configured to
>> S3 Minio) Any ideas?
>>
>> --
>> Robert Cullen
>> 240-475-4490
>>
>
--
Robert Cullen
240-475-4490
When I run a job on my Kubernetes session cluster only the checkpoint
directories are created but not the savepoints. (Filesystem configured to
S3 Minio) Any ideas?
--
Robert Cullen
240-475-4490
e running on.
>
> Regards,
> Dian
>
> 2021年3月24日 上午2:40，Robert Cullen 写道：
>
> I’m running this script taken from the Flink website: tutorial.py
>
> python tutorial.py
>
> from pyflink.common.serialization import SimpleStringEncoder
> from pyflink.common.typeinfo
. However when I run this on my kubernetes session cluster there is
no output. Any ideas?
./bin/flink run \
--target kubernetes-session \
-Dkubernetes.cluster-id=flink-jobmanager -Dkubernetes.namespace=cmdaa \
--pyModule tutorial \
--pyFiles /opt/flink-1.12.0/examples/tutorial.py \
--detached
--
Robert Cullen
240-475-4490
3) Is it possible to dump the stack trace? It would help us understanding
> what’s happening.
>
> Thanks,
> Dian
>
> 2021年3月16日 下午11:51，Robert Cullen 写道：
>
> Thanks All,
>
> I've added python and pyflink to the TM image which fixed the problem.
> Now howeve
details, you can refer to the document[1].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/installation.html
>
> Best,
> Xingbo
>
> Robert Cullen 于2021年3月16日周二 上午2:58写道：
>
>> Okay, I added the jars and fixed that exception. Howe
the lib/ folder of all machines / instances,
> and you've restarted Flink after adding the files to lib/ ?
>
> On Mon, Mar 15, 2021 at 3:42 PM Robert Cullen
> wrote:
>
>> Shuiqiang,
>>
>> I added the flink-connector-kafka_2-12 jar to the /opt/flink/lib direct
AM Shuiqiang Chen wrote:
> Hi Robert,
>
> You can refer to
> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py
> for the whole example.
>
> Best,
> Shuiqiang
>
> Robert Cullen 于2021年3月13日周六 上午4
SimpleStringSchema(), kafka_props)
>
> # set the kafka source to consume data from earliest offset.
> kafka_consumer.set_start_from_earliest()
>
> # create a DataStream from kafka consumer source
> ds = env.add_source(kafka_consumer)
>
> result_stream = ...
>
> # write the
I’ve scoured the web looking for an example of using a Kafka source for a
DataStream in python. Can someone finish this example?
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
ds = env.from_collection( KAFKA_SOURCE )
...
--
Robert Cullen
240-475-4490
e by starting a
> pod as the Flink client.
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#accessing-flinks-web-ui
>
> Best,
> Yang
>
> Robert Cullen 于2021年3月6日周六 上午12:43写道：
>
>> Trying to
.java:246)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
--
Robert Cullen
240-475-4490
e.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Do I need to create an ingress for pyflink?
Robert Cullen
240-475-4490
.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
--
Robert Cullen
240-475-4490
My customer wants us to install this package in our Flink Cluster:
https://github.com/twitter/AnomalyDetection
One of our engineers developed a python version:
https://pypi.org/project/streaming-anomaly-detection/
Is there a way to install this in our cluster?
--
Robert Cullen
240-475-4490
org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
... 35 more
Any ideas?
Robert Cullen
240-475-4490
I have a Kubernetes cluster with Flink running in Session Mode.
Is there a way to drop the jar file into a folder and/or add it to the
Docker image?
--
Robert Cullen
240-475-4490
.42.255:9000
# s3.endpoint: http://10.43.70.109:9000
s3.path-style-access: true
s3.path.style.access: true
s3.access-key: qCEwcLzhi7xfhl5R6sXLn93a5brgRtBs
s3.secret-key: fJWZFviCxWUrfjQoXZ4UAjN4YviQXQOz
--
Robert Cullen
240-475-4490
> > How are you running the Flink cluster? What is your deplyment?
> >
> > The exception clearly indicates that you found a bug. Could you open an
> > ticket in Flink's JIRA? We need details how to reproduce it.
> >
> > Thanks,
> > Timo
> >
ate over the results but using this causes an error:
log_counts.execute().collect();
Has there been an update in how to traverse results?
Robert Cullen
240-475-4490
2021 at 11:29 AM Arvid Heise wrote:
> Sorry Robert for not checking the complete example. New sources are used
> with fromSource instead of addSource. It's not ideal but hopefully we can
> remove the old way rather soonish to avoid confusion.
>
> On Tue, Jan 5, 2021 at 5:23 PM Robert Cul
o explicitely set but that shouldn't be necessary (and may make
> things more complicated once you also want to execute the application in
> streaming mode).
>
> env.setRuntimeMode(RuntimeExecutionMode.BATCH);
>
>
> On Tue, Jan 5, 2021 at 4:51 PM Robert Cullen
> wrote:
>
while it's safe to use for development, I'd wait for 1.12.1 to roll it out
> on production.
>
> If you have specific questions on the migration from DataSet and
> DataStream, please let me know.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12
I have a Kafka source that I would like to run a batch job on. Since
Version 1.12.0 is now soft deprecating the DataSet API in favor of the
DataStream API, can someone show me an example of this? (Using DataStream)
thanks
--
Robert Cullen
240-475-4490
Does anyone know when an official 1.12.0 image will be available on Docker
Hub?
--
Robert Cullen
240-475-4490
53 matches
Mail list logo
