HI Team
Iam working on flink with IBM cloud object storage.We have setted up a
dockerImage for flink and pushed it to IBM cloud. So we cannot configure
our access_key & secret_key in flink-conf.yaml. So I was trying to use
customway to do that functionality & Iam unable to do it.
Please find t
Hi Jungtaek,
Yes. Your understanding is correct :)
Best,
Jark
On Tue, 28 Apr 2020 at 11:58, Jungtaek Lim
wrote:
> Thanks Kurt and Jark for the detailed explanation! Pretty much helped to
> understand about FLIP-66.
>
> That sounds as Flink won't leverage timestamp in StreamRecord (which is
> h
Thanks Kurt and Jark for the detailed explanation! Pretty much helped to
understand about FLIP-66.
That sounds as Flink won't leverage timestamp in StreamRecord (which is
hidden and cannot modified easily) and handles the time semantic by the
input schema for the operation, to unify the semantic b
Xintong Song created FLINK-17422:
Summary: Create user document for the external resource framework
and the GPU plugin..
Key: FLINK-17422
URL: https://issues.apache.org/jira/browse/FLINK-17422
Project
Hi Jungtaek,
Kurt has said what I want to say. I will add some background.
Flink Table API & SQL only supports to define processing-time attribute and
event-time attribute (watermark) on source, not support to define a new one
in query.
The time attributes will pass through the query and time-base
+1
best,
Forward
Yadong Xie 于2020年4月28日周二 上午10:03写道：
> Hi all
>
> sorry for we have an issue that was not discovered in advance
>
> When users run multiple standalone TMs on the same host, the JM log list
> may confuse the users since all JM and TM logs are stored in the same
> folder, and they
Hi all
sorry for we have an issue that was not discovered in advance
When users run multiple standalone TMs on the same host, the JM log list
may confuse the users since all JM and TM logs are stored in the same
folder, and they will appear on the same list.
I have a discussion with Gary and Lin
The current behavior is later. Flink gets time attribute column from source
table, and tries to analyze and keep
the time attribute column as much as possible, e.g. simple projection or
filter which doesn't effect the column
will keep the time attribute, window aggregate will generate its own time
Hi Jiayi Liao.
Thanks your replying. Add attachment . And can’t get any useful messages;
> 2020年4月27日 12:40，Jiayi Liao 写道：
>
> <粘贴的图形-1.tiff>
Hi devs,
I'm interesting about the new change on FLIP-66 [1], because if I
understand correctly, Flink hasn't been having event-time timestamp field
(column) as a part of "normal" schema, and FLIP-66 tries to change it.
That sounds as the column may be open for modification, like rename (alias)
o
One more point I forgot to mention.
Flink SQL supports Hive UDF's[1]. I haven't tested it, but the datasketch
hive package should just work out of the box.
Seth
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html
On Mon, Apr 27, 2020 at 2:27 PM Se
Hi Marta,
I think this is a great idea, I'd be happy to help mentor a table
documentation project.
Seth
On Thu, Apr 23, 2020 at 8:38 AM Marta Paes Moreira
wrote:
> Thanks for the feedback!
>
> So far, the projects on the table are:
>
>1. Improving the Table API/SQL documentation.
>2. I
Hi Lee,
I really like this project, I used it with Flink a few years ago when it
was still Yahoo DataSketches. The projects clearly complement each other.
As Arvid mentioned, the Flink community is trying to foster an ecosystem
larger than what is in the main Flink repository. The reason is that t
If this can encourage Lee I'm one of the Flink users that already use
datasketches and I found it an amazing library.
When I was trying it out (lat year) I tried to stimulate some discussion[1]
but at that time it was probably too early..
I really hope that now things are mature for both communitie
Hi,
In my opinion the docker images are essentially simply differently packed
binary releases.
This becomes more true when in the future deploying a Flink application to
kubernetes simply pulls the correct binary from a docker hub. Because of
these kinds of use cases I disagree with Robert that t
Hi Arvid,
Note: I am dual listing this thread on both dev lists for better tracking.
1. I'm curious on how you would estimate the effort to port datasketches
>to Flink? It already has a Java API, but how difficult would it be to
>subdivide the tasks into parallel chunks of work? Since
Piotr Nowojski created FLINK-17421:
--
Summary: Backpressure new checkpoints if previous were not managed
to be cleaned up yet
Key: FLINK-17421
URL: https://issues.apache.org/jira/browse/FLINK-17421
P
Gyula Fora created FLINK-17420:
--
Summary: Cannot alias Tuple and Row fields when converting
DataStream to Table
Key: FLINK-17420
URL: https://issues.apache.org/jira/browse/FLINK-17420
Project: Flink
Hi
The image is not very clear.
For RocksDBStateBackend, do you enable incremental checkpoint?
Currently, checkpoint on TM side contains some steps:
1 barrier align
2 sync snapshot
3 async snapshot
For expired checkpoint, could you please check the tasks in the first
operator of the DAG to find o
+1 for xyz.[min|max]
This is already mentioned in the Code Style Guideline [1].
Best,
Jark
[1]:
https://flink.apache.org/contributing/code-style-and-quality-components.html#configuration-changes
On Mon, 27 Apr 2020 at 21:33, Flavio Pompermaier
wrote:
> +1 for Chesnay approach
>
> On Mon, Apr
This also sounds good to me. The master branch will continue to be the
publishing area for the release Dockerfiles and we can support changes
between Flink versions without additional complexity.
My only comment would be that I hope we can still keep eventual convergence
in mind—it should be possi
xiaodao created FLINK-17419:
---
Summary: kafka sink json data convert issuse,type compare error
Key: FLINK-17419
URL: https://issues.apache.org/jira/browse/FLINK-17419
Project: Flink
Issue Type: Bug
+1 for Chesnay approach
On Mon, Apr 27, 2020 at 2:31 PM Chesnay Schepler wrote:
> +1 for xyz.[min|max]; imo it becomes obvious if think of it like a yaml
> file:
>
> xyz:
> min:
> max:
>
> opposed to
>
> min-xyz:
> max-xyz:
>
> IIRC this would also be more in-line with the hierarchical
Thanks for starting this discussion Chesnay. Your proposal sounds good to
me. I can see how the current setup makes the development of version
specific features impractical. Hence, +1 for the proposed changes.
Cheers,
Till
On Mon, Apr 27, 2020 at 12:19 PM David Anderson
wrote:
> Makes sense to
Simon Tao created FLINK-17418:
-
Summary: Windows system is not supported currently
Key: FLINK-17418
URL: https://issues.apache.org/jira/browse/FLINK-17418
Project: Flink
Issue Type: New Feature
Wei Zhong created FLINK-17417:
-
Summary: The flink-avro-sql-jar is not really sql-jar
Key: FLINK-17417
URL: https://issues.apache.org/jira/browse/FLINK-17417
Project: Flink
Issue Type: Bug
Thanks Aljoscha.
On Mon, Apr 27, 2020 at 5:29 PM Aljoscha Krettek
wrote:
> Hi Manish,
>
> welcome to the community! You could start from a user program example
> and then try and figure out how that leads to job execution. So probably
> start with the DataStream WordCount example, figure out wha
wangxiyuan created FLINK-17416:
--
Summary: Flink-kubernetes doesn't work on java 8 8u252
Key: FLINK-17416
URL: https://issues.apache.org/jira/browse/FLINK-17416
Project: Flink
Issue Type: Bug
+1 for xyz.[min|max]; imo it becomes obvious if think of it like a yaml
file:
xyz:
min:
max:
opposed to
min-xyz:
max-xyz:
IIRC this would also be more in-line with the hierarchical scheme for
config options we decided on months ago.
On 27/04/2020 13:25, Xintong Song wrote:
+1 for
I see no reason why we shouldn't put reporters into the plugins
directory by default, was already planning to do this for the JMX
reporter (FLINK-16970) and intend to do this for all remaining reporters.
I'm not sure about filesystems though; is there a clear 1:1 mapping of
scheme <-> filesyst
Aljoscha Krettek created FLINK-17415:
Summary: Fold API-agnostic documentation into DataStream
documentation (chinese)
Key: FLINK-17415
URL: https://issues.apache.org/jira/browse/FLINK-17415
Proje
Hi Manish,
welcome to the community! You could start from a user program example
and then try and figure out how that leads to job execution. So probably
start with the DataStream WordCount example, figure out what the methods
on DataStream do, that is how they build up a graph of Transformati
+1 for Robert's idea about adding tests/tools checking the pattern of new
configuration options, and migrate the old ones in release 2.0.
Concerning the preferred pattern, I personally agree with Till's opinion. I
think 'xyz.[min|max]' somehow expresses that 'min' and 'max' are properties
of 'xyz'
Huang Xingbo created FLINK-17414:
Summary: Change the steps of building PyFlink in doc
Key: FLINK-17414
URL: https://issues.apache.org/jira/browse/FLINK-17414
Project: Flink
Issue Type: Sub-t
Zhijiang created FLINK-17413:
Summary: Remove redundant states from ThreadSafeUnaligner
Key: FLINK-17413
URL: https://issues.apache.org/jira/browse/FLINK-17413
Project: Flink
Issue Type: Sub-task
zhangzhiyuan created FLINK-17412:
Summary: Make the FLINK_YARN_FILES value configurable
Key: FLINK-17412
URL: https://issues.apache.org/jira/browse/FLINK-17412
Project: Flink
Issue Type: Wish
Hi,
Looking into the codebase, its quite huge.
Any suggestions/guidelines on which parts should one explore first, and to
maintain whole picture too?
Manish
Makes sense to me. I think this would align well enough with user
expectations, and be more straightforward.
David
On Thu, Apr 23, 2020 at 1:23 PM Chesnay Schepler wrote:
> Hello everyone,
>
> Currently, all development in docker-flink occurs on the master branch,
> for all releases at once. Th
Overall I like this proposal; thanks for bringing it forward, Aljoscha.
I also like the idea of making the Watermark generator a rich function --
this should make it more straightforward to implement smarter watermark
generators. Eg, one that uses state to keep statistics about the actual
out-of-o
Simon Tao created FLINK-17411:
-
Summary: Add async mode in JDBCLookupFunction
Key: FLINK-17411
URL: https://issues.apache.org/jira/browse/FLINK-17411
Project: Flink
Issue Type: Improvement
Hi everyone,
as Robert said I think the problem is that we don't have strict guidelines
and every committer follows his/her personal taste. I'm actually not sure
whether we can define bullet-proof guidelines but we can definitely
make them more concrete.
In this case here, I have to admit that I
This naturally depends on the issue you're trying to reproduce.
On 25/04/2020 11:18, Manish G wrote:
Hi,
While working on an issue, is there a specific approach to quickly
reproduce the issue locally?
With regards
Manish
Hi Aljoscha,
Thanks for opening the discussion!
I have two comments on the FLIP:
1) we could add lifecycle methods to the Generator, i.e. open()/
close(), probably with a Context as argument: I have not fully thought
this through but I think that this is more aligned with the rest of
our rich fun
Hello Aljoscha,
unfortunately not, I'm not really familiar with the optimizer code and it's
really complex to debug :(
this method is as far as I got -
https://github.com/apache/flink/blob/master/flink-optimizer/src/main/java/org/apache/flink/optimizer/dataproperties/RequestedGlobalProperties.jav
Zhu Zhu created FLINK-17410:
---
Summary: BlockingPartitionBenchmark compilation failed due to
changed StreamGraph interface
Key: FLINK-17410
URL: https://issues.apache.org/jira/browse/FLINK-17410
Project: Fli
On 27.04.20 09:34, David Morávek wrote:
When we include `flatMap` in between rebalances ->
`.rebalance().flatMap(...).rebalance()`, we need to reshuffle again,
because dataset distribution may have changed (eg. you can possibli emit
unbouded stream from a single element). Unfortunatelly `flatMap
zhanghang-dev created FLINK-17409:
-
Summary: SqlCreateView's SqlParserPos return wrong value
Key: FLINK-17409
URL: https://issues.apache.org/jira/browse/FLINK-17409
Project: Flink
Issue Type:
Yangze Guo created FLINK-17408:
--
Summary: Introduce GPUDriver
Key: FLINK-17408
URL: https://issues.apache.org/jira/browse/FLINK-17408
Project: Flink
Issue Type: Sub-task
Components: Ru
Yangze Guo created FLINK-17407:
--
Summary: Introduce external resource framework.
Key: FLINK-17407
URL: https://issues.apache.org/jira/browse/FLINK-17407
Project: Flink
Issue Type: Sub-task
Hi Lee,
I must admit that I also heard of data sketches for the first time (there
are really many Apache projects).
Datasketches sounds really exciting. As a (former) data engineer, I can
100% say that this is something that (end-)users want and need and it would
make so much sense to have it in
Kurt Young created FLINK-17406:
--
Summary: add documentation about dynamic table options
Key: FLINK-17406
URL: https://issues.apache.org/jira/browse/FLINK-17406
Project: Flink
Issue Type: Sub-tas
godfrey he created FLINK-17405:
--
Summary: add test cases for cancel job in SQL client
Key: FLINK-17405
URL: https://issues.apache.org/jira/browse/FLINK-17405
Project: Flink
Issue Type: Improveme
Thanks for starting this discussion.
I believe the different options are a lot about personal taste, there are
no objective arguments why one option is better than the other.
I agree with your proposal to simply go with the "max-xyz" pattern, as this
is the style of the majority of the current con
Thanks for starting the thread!
I would consider the docker images of Flink convenience binary releases
that can happen any time. I believe a simplified, but formal release
process would be appropriate (preview / staging images for the community to
validate & vote, then release to docker hub).
On
Hi all,
The voting time for FLIP-108[1] has passed. I'm closing the vote now.
There were 3 + 3 votes, 3 of which are binding:
- Till (binding)
- Becket (binding)
- Stephan (binding)
- Xintong Song (non-binding)
- Canbin Zheng (non-binding)
- Yang Wang (non-binding)
There were no -1 votes.
Thu
Thanks Dian for the release work and thanks everyone involved.
Best,
Zhijiang
--
From:Till Rohrmann
Send Time:2020 Apr. 27 (Mon.) 15:13
To:Jingsong Li
Cc:dev ; Leonard Xu ; Benchao Li
; Konstantin Knauf ; jincheng
sun ; Hequn C
+1
On Thu, Apr 16, 2020 at 4:17 AM Yangze Guo wrote:
> Hi Aljoscha,
>
> Thanks for your advice. +1 to align the config pattern.
>
> I also agree that we need to move the long discussion to the [DISCUSS]
> thread. Sorry if it bothers you.
>
> Best,
> Yangze Guo
>
> On Thu, Apr 16, 2020 at 7:52 AM
Hello Flinkers,
we have run into unexpected behaviour with chained Reshuffles in Apache
Beam's Flink runner (batch).
In flink optimizer, when we `.rebalance()` dataset, is output channel is
marked as `FORCED_REBALANCED`. When we chain this with another
`.rebalance()`, the latter is ignored becaus
Robert Metzger created FLINK-17404:
--
Summary: Running HA per-job cluster (rocks, non-incremental) gets
stuck
Key: FLINK-17404
URL: https://issues.apache.org/jira/browse/FLINK-17404
Project: Flink
Thanks Dian for being our release manager and thanks to everyone who helped
making this release possible.
Cheers,
Till
On Mon, Apr 27, 2020 at 3:26 AM Jingsong Li wrote:
> Thanks Dian for managing this release!
>
> Best,
> Jingsong Lee
>
> On Sun, Apr 26, 2020 at 7:17 PM Jark Wu wrote:
>
>> Th
60 matches
Mail list logo
