Hi Lorenzo,
> As I mentioned above, in the end I think hardcoding in Flink support for
the `cpulimit` doesn't seem appropriate and at the same time it is not
actually necessary.
I agree.
What's suggested is similar to the way where Flink currently allow users to
limit resource usage on s3a filesystem.
I appreciate the effort spent in looking into limiting s5cmd using cpu
limit.
> To use `s5cmd` with `cpulimit` via some bash script wrapping the
`s5cmd`. Users instead of passing a path to the `s5cmd` via the
`s3.s5cmd.path` config option, can pass a path to that wrapper bash script.
That wrapper could then apply `cpulimit` to the `s5cmd` call (or use a
completely different mechanism, like CGroups if desired).
Would appreciate if we have the CPU limit captured under rejected
alternatives and also mention the option of implementing limits within
wrapper.
It can be very useful for users who may want to implement said limiting.
Best regards
Keith Lee
On Mon, Jun 17, 2024 at 5:50 PM Piotr Nowojski <pnowoj...@apache.org> wrote:
> Hi all!
>
> >> Do you know if there is a way to limit the CPU utilisation of s5cmd? I
> see
> >> worker and concurrency configuration but these do not map directly to
> cap
> >> in CPU usage. The experience for feature user in this case will be one
> of
> >> trial and error.
> >
> > Those are good points. As a matter of fact, shortly after publishing this
> FLIP, we started experimenting with using `cpulimit` to achieve just that.
> If everything will work out fine, we are planning to expose this as a
> configuration option for the S3 file system. I've added that to the FLIP.
>
> Before finally putting this FLIP under vote, I would like to revisit/share
> with you our production experience when it comes to preventing s5cmd from
> overloading the machines.
>
> When trying out `cpulimit`, we had some problems on our particular setup to
> make `cpulimit` behave the way we want. I think in general that's still a
> viable option, but we have internally settled upon just limiting the number
> of workers via:
>
> "s3.s5cmd.args": "-r 0 --numworkers 5"
>
> That seems to work reliably with a combination of FLINK-35501 [1], without
> affecting the actual performance. But I will be curious to learn what will
> eventually work best for others.
>
> Hence, I would propose also to modify the FLIP slightly, and remove
> "native" support for cpulimit from this FLIP. In other words, I would
> propose to drop the config option:
>
> public static final ConfigOption<String> S5CMD_CPULIMIT =
> ConfigOptions.key("s3.s5cmd.cpulimit")
> .doubleType()
> .withDescription(
> "Optional cpulimit value to set for the s5cmd
> to prevent TaskManager from overloading");
>
> The thing is, that it seems a bit platform specific, and there is actually
> no need for this dedicated config option. Instead I would propose to
> document in some best practices section two solutions to limit the resource
> usage of the s5cmd.
>
> 1. Adding ` --numworkers X` to the `s3.s5cmd.args` ConfigOption and/or
> adjusting IO Thread Pool size (thanks to [1])
> 2. To use `s5cmd` with `cpulimit` via some bash script wrapping the
> `s5cmd`. Users instead of passing a path to the `s5cmd` via the
> `s3.s5cmd.path` config option, can pass a path to that wrapper bash script.
> That wrapper could then apply `cpulimit` to the `s5cmd` call (or use a
> completely different mechanism, like CGroups if desired).
>
> As I mentioned above, in the end I think hardcoding in Flink support for
> the `cpulimit` doesn't seem appropriate and at the same time it is not
> actually necessary.
>
> WDYT?
>
> Best,
> Piotrek
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-35501
>
> pt., 17 maj 2024 o 15:27 <lorenzo.affe...@ververica.com.invalid>
> napisał(a):
>
> > Perfectly agree with all your considerations.
> > Wee said.
> >
> > Thank you!
> > On May 16, 2024 at 10:53 +0200, Piotr Nowojski <pnowoj...@apache.org>,
> > wrote:
> > > Hi Lorenzo,
> > >
> > > > • concerns about memory and CPU used out of Flink's control
> > >
> > > Please note that using AWS SDKv2 would have the same concerns. In both
> > > cases Flink can control only to a certain extent what either the SDKv2
> > > library does under the hood or the s5cmd process, via configuration
> > > parameters. SDKv2, if configured improperly, could also overload TMs
> CPU,
> > > and it would also use extra memory. To an extent that also applies to
> the
> > > current way we are downloading/uploading files from the S3.
> > >
> > > > • deployment concerns from the usability perspective (would need to
> > > install s5cmd on all TMs prior to the job deploy)
> > >
> > > Yes, that's the downside.
> > >
> > > > Also, invoking an external binary would incur in some performance
> > > degradation possibly. Might it be that using AWS SDK would not, given
> its
> > > Java implementation, and performance could be similar?
> > >
> > > We haven't run the benchmarks for very fast checkpointing and we
> haven't
> > > measured latency impact, but please note `s5cmd` is very fast to
> startup.
> > > It's not Java after all ;) It's at least an order of magnitude below
> 1s,
> > so
> > > the impact on Flink should be negligible, as Flink doesn't support
> > > checkpointing under < 1s. Especially not with uploading files to S3. I
> > > wouldn't be actually surprised that `s5cmd` (or AWS
> > > SDKv2's `TransferManager`) would actually both improve minimal e2e
> > > checkpointing times.
> > >
> > > > Wrapping up, I see using AWS SDK having PROs that could be traded
> with
> > > the CON of slightly worse perf than s5cmd:
> > > >
> > > > • no hurdles for the user, as the SDK would be a Flink dependency
> > > > • less config on the Flink side
> > > >
> > > > Do you agree?
> > >
> > > To an extent, yes. AWS SDKv2's TransferManager would also have to be
> > > configured properly. But I agree that the largest hurdle with `s5cmd`
> is
> > > the added operational complexity of having to supply 3rd party binary.
> > >
> > > Please also keep in mind that a lot of work for that FLIP will be in
> > > defining, creating and using the interfaces for batch files copy. The
> > > actual implementation of the fast copying file system interface and
> using
> > > `s5cmd` is not the dominant factor. So all in all, I wouldn't object if
> > > someone would like to take over my AWS SDKv2 PoC, and finish it off in
> > the
> > > future, as an alternative for the `s5cmd`. Indeed AWS SDKv2 could at
> some
> > > point become the default setting, while `s5cmd` could remain as a
> faster
> > > alternative.
> > >
> > > Best,
> > > Piotrek
> > >
> > > czw., 16 maj 2024 o 09:03 <lorenzo.affe...@ververica.com.invalid>
> > > napisał(a):
> > >
> > > > Hello Piotr and thanks for this proposal!
> > > > The idea sounds smart and very well grounded thank you!
> > > >
> > > > Also here, as others, I have some doubts about invoking an external
> > binary
> > > > (namely s5cmd):
> > > >
> > > > • concerns about memory and CPU used out of Flink's control
> > > > • deployment concerns from the usability perspective (would need to
> > > > install s5cmd on all TMs prior to the job deploy)
> > > >
> > > >
> > > > Also, invoking an external binary would incur in some performance
> > > > degradation possibly. Might it be that using AWS SDK would not, given
> > its
> > > > Java implementation, and performance could be similar?
> > > >
> > > > Wrapping up, I see using AWS SDK having PROs that could be traded
> with
> > the
> > > > CON of slightly worse perf than s5cmd:
> > > >
> > > > • no hurdles for the user, as the SDK would be a Flink dependency
> > > > • less config on the Flink side
> > > >
> > > >
> > > > Do you agree?
> > > > On May 13, 2024 at 07:42 +0200, Hangxiang Yu <master...@gmail.com>,
> > wrote:
> > > > > > > >
> > > > > > > > Note that for both recovery and checkpoints, there are no
> > retring
> > > > > > > > mechanisms. If any part of downloading or
> > > > > > > > uploading fails, the job fails over, so actually using such
> > interface
> > > > > > > > extension would be out of scope of this FLIP. In
> > > > > > > > that case, maybe if this could be extended in the future
> > without
> > > > breaking
> > > > > > > > compatibility we could leave it as a
> > > > > > > > future improvement?
> > > > > > > >
> > > > > >
> > > > > > Thanks for the reply.
> > > > > > It makes sense to consider as a future optimization.
> > > > > >
> > > > > > On Fri, May 10, 2024 at 4:31 PM Piotr Nowojski <
> > piotr.nowoj...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > Thanks for your suggestions!
> > > > > > > >
> > > > > > > > > > > > I'd prefer a unified one interface
> > > > > > > >
> > > > > > > > I have updated the FLIP to take that into account. In this
> > case, I
> > > > would
> > > > > > > > also propose to completely drop `DuplicatingFileSystem` in
> > favour of a
> > > > > > > > basically renamed version of it `PathsCopyingFileSystem`.
> > > > > > > > `DuplicatingFileSystem` was not marked as
> > PublicEvolving/Experimental
> > > > > > > > (probably by mistake), so technically we can do it. Even if
> > not for
> > > > that
> > > > > > > > mistake, I would still vote to replace it to simplify the
> > code, as any
> > > > > > > > migration would be very easy. At the same time to the best of
> > my
> > > > knowledge,
> > > > > > > > no one has ever implemented it.
> > > > > > > >
> > > > > > > > > > > > The proposal mentions that s5cmd utilises 100% of CPU
> > similar to
> > > > Flink
> > > > > > > > > > > > 1.18. However, this will be a native process outside
> > of the JVM.
> > > > Are
> > > > > > > > there
> > > > > > > > > > > > risk of large/long state download starving the TM of
> > CPU cycle
> > > > causing
> > > > > > > > > > > > issues such as heartbeat or ask timeout?
> > > > > > > > > > > >
> > > > > > > > > > > > Do you know if there is a way to limit the CPU
> > utilisation of
> > > > s5cmd? I
> > > > > > > > see
> > > > > > > > > > > > worker and concurrency configuration but these do not
> > map directly
> > > > to cap
> > > > > > > > > > > > in CPU usage. The experience for feature user in this
> > case will be
> > > > one of
> > > > > > > > > > > > trial and error.
> > > > > > > >
> > > > > > > > Those are good points. As a matter of fact, shortly after
> > publishing
> > > > this
> > > > > > > > FLIP, we started experimenting with using `cpulimit` to
> > achieve just
> > > > that.
> > > > > > > > If everything will work out fine, we are planning to expose
> > this as a
> > > > > > > > configuration option for the S3 file system. I've added that
> > to the
> > > > FLIP.
> > > > > > > >
> > > > > > > > > > > > 2. copyFiles is not an atomic operation, How could we
> > handle the
> > > > > > > > situation
> > > > > > > > > > > > when some partial files fail ?
> > > > > > > > > > > > Could we return the list of successful files then the
> > caller could
> > > > decide
> > > > > > > > > > > > to retry or just know them ?
> > > > > > > >
> > > > > > > > Do you have some suggestions on how that should be
> implemented
> > in the
> > > > > > > > interface and how should it be used?
> > > > > > > >
> > > > > > > > Note that for both recovery and checkpoints, there are no
> > retring
> > > > > > > > mechanisms. If any part of downloading or
> > > > > > > > uploading fails, the job fails over, so actually using such
> > interface
> > > > > > > > extension would be out of scope of this FLIP. In
> > > > > > > > that case, maybe if this could be extended in the future
> > without
> > > > breaking
> > > > > > > > compatibility we could leave it as a
> > > > > > > > future improvement?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Piotrek
> > > > > > > >
> > > > > > > >
> > > > > > > > pt., 10 maj 2024 o 07:40 Hangxiang Yu <master...@gmail.com>
> > > > napisał(a):
> > > > > > > >
> > > > > > > > > > > > Hi Piotr.
> > > > > > > > > > > > Thanks for your proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > I have some comments, PTAL:
> > > > > > > > > > > > 1. +1 about unifying the interface with
> > DuplicatingFileSystem.
> > > > > > > > > > > > IIUC, DuplicatingFileSystem also covers the logic
> > from/to both
> > > > local and
> > > > > > > > > > > > remote paths.
> > > > > > > > > > > > The implementations could define their own logic
> about
> > how to fast
> > > > > > > > > > > > copy/duplicate files, e.g. hard link or transfer
> > manager.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. copyFiles is not an atomic operation, How could we
> > handle the
> > > > > > > > situation
> > > > > > > > > > > > when some partial files fail ?
> > > > > > > > > > > > Could we return the list of successful files then the
> > caller could
> > > > decide
> > > > > > > > > > > > to retry or just know them ?
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 9, 2024 at 3:46 PM Keith Lee <
> > > > leekeiabstract...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Piotr,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for the proposal. Looks great.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Along similar line of Aleks' question on
> > memory usage.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The proposal mentions that s5cmd utilises
> 100%
> > of CPU similar
> > > > to Flink
> > > > > > > > > > > > > > > > 1.18. However, this will be a native process
> > outside of the
> > > > JVM. Are
> > > > > > > > > > > > there
> > > > > > > > > > > > > > > > risk of large/long state download starving
> the
> > TM of CPU cycle
> > > > causing
> > > > > > > > > > > > > > > > issues such as heartbeat or ask timeout?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Do you know if there is a way to limit the
> CPU
> > utilisation of
> > > > s5cmd? I
> > > > > > > > > > > > see
> > > > > > > > > > > > > > > > worker and concurrency configuration but
> these
> > do not map
> > > > directly to
> > > > > > > > cap
> > > > > > > > > > > > > > > > in CPU usage. The experience for feature user
> > in this case
> > > > will be one
> > > > > > > > of
> > > > > > > > > > > > > > > > trial and error.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > Keith
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, May 8, 2024 at 12:47 PM Ahmed Hamdy <
> > > > hamdy10...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Piotr
> > > > > > > > > > > > > > > > > > > > +1 for the proposal, it seems to have
> > a lot of gains.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best Regards
> > > > > > > > > > > > > > > > > > > > Ahmed Hamdy
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, 6 May 2024 at 12:06, Zakelly
> > Lan <
> > > > zakelly....@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Piotrek,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for your answers!
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Good question. The intention
> > and use case behind
> > > > > > > > > > > > > > > > `DuplicatingFileSystem`
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > > different. It marks
> if
> > `FileSystem` can quickly
> > > > copy/duplicate
> > > > > > > > > > > > files
> > > > > > > > > > > > > > > > > > > > > > > > > > > > in the remote
> > `FileSystem`. For example an
> > > > equivalent of a hard
> > > > > > > > > > > > link
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > > > bumping a reference
> > count in the remote system.
> > > > That's a bit
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > > > > > > to copy paths between
> > remote and local file
> > > > systems.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > However, it could
> > arguably be unified under one
> > > > interface where
> > > > > > > > we
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > > > > > re-use or re-name
> > `canFastDuplicate(Path, Path)` to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > `canFastCopy(Path,
> > Path)` with the following use
> > > > cases:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > -
> > `canFastCopy(remoteA, remoteB)` returns true -
> > > > current
> > > > > > > > equivalent
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > `DuplicatingFileSystem` - quickly duplicate/hard
> > > > link remote path
> > > > > > > > > > > > > > > > > > > > > > > > > > > > - `canFastCopy(local,
> > remote)` returns true - FS
> > > > can natively
> > > > > > > > > > > > upload
> > > > > > > > > > > > > > > > > > > > > > > > local
> > > > > > > > > > > > > > > > > > > > > > > > > > > > file to a remote
> > location
> > > > > > > > > > > > > > > > > > > > > > > > > > > > -
> `canFastCopy(remote,
> > local)` returns true - FS
> > > > can natively
> > > > > > > > > > > > > > > > download
> > > > > > > > > > > > > > > > > > > > > > > > > > > > local file from a
> > remote location
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe indeed that's a
> > better solution vs having
> > > > two separate
> > > > > > > > > > > > > > > > interfaces
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > copying and
> > duplicating?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'd prefer a unified one
> > interface, `canFastCopy(Path,
> > > > Path)` looks
> > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > me. This also resolves my
> > question 1 about the
> > > > destination.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at
> 6:36 PM
> > Piotr Nowojski <
> > > > > > > > pnowoj...@apache.org>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All!
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your
> > comments.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Muhammet and Hong,
> > about the config options.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Could you
> > please also add the configuration
> > > > property for this?
> > > > > > > > An
> > > > > > > > > > > > > > > > > > > > > > > > example
> > > > > > > > > > > > > > > > > > > > > > > > > > > > showing how users
> > would set this parameter would
> > > > be helpful.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1/ Configure
> > the implementation of
> > > > PathsCopyingFileSystem used
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2/ Configure
> > the location of the s5cmd binary
> > > > (version control
> > > > > > > > > > > > > > > > etc.)
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Ops, sorry I added
> the
> > config options that I had
> > > > in mind to the
> > > > > > > > > > > > > > > > FLIP. I
> > > > > > > > > > > > > > > > > > > > > > > > > > > > don't know why I have
> > omitted this. Basically I
> > > > suggest that in
> > > > > > > > > > > > order
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > use native file
> > copying:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. `FileSystem` must
> > support it via implementing
> > > > > > > > > > > > > > > > > > > > `PathsCopyingFileSystem`
> > > > > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. That `FileSystem`
> > would have to be configured
> > > > to actually use
> > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > > > > > > > > > > example S3 file
> system
> > would return `true` that it
> > > > can copy paths
> > > > > > > > > > > > > > > > > > > > > > > > > > > > only if
> > `s3.s5cmd.path` has been specified.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Would this
> > affect any filesystem connectors
> > > > that use
> > > > > > > > > > > > > > > > FileSystem[1][2]
> > > > > > > > > > > > > > > > > > > > > > > > > > > > dependencies?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Definitely not out of
> > the box. Any place in Flink
> > > > that is
> > > > > > > > currently
> > > > > > > > > > > > > > > > > > > > > > > > > > > > uploading/downloading
> > files from a FileSystem
> > > > could use this
> > > > > > > > > > > > feature,
> > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > > > would have to be
> > implemented. The same way this
> > > > FLIP will
> > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > > > > > native
> > > > > > > > > > > > > > > > > > > > > > > > > > > > files copying when
> > downloading state during
> > > > recovery,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > but the old code path
> > will be still used for
> > > > uploading state
> > > > > > > > files
> > > > > > > > > > > > > > > > > > > > > > > > during a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > checkpoint.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How adding a
> > s5cmd will affect memory
> > > > footprint? Since this is
> > > > > > > > a
> > > > > > > > > > > > > > > > > > > > native
> > > > > > > > > > > > > > > > > > > > > > > > > > > > binary, memory
> > consumption will not be controlled
> > > > by JVM or
> > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > As you mentioned the
> > memory usage of `s5cmd` will
> > > > not be
> > > > > > > > > > > > controlled,
> > > > > > > > > > > > > > > > so
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > memory footprint will
> > grow. S5cmd integration with
> > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > has been tested quite
> > extensively on our
> > > > production environment
> > > > > > > > > > > > > > > > > > > > already,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and we haven't
> > observed any issues so far despite
> > > > the fact we
> > > > > > > > > > > > > > > > > > > > > > > > > > > > are using quite small
> > pods. But of course if your
> > > > setup is
> > > > > > > > working
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > edge of OOM, this
> > could tip you over that edge.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Zakelly:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. What is
> the
> > semantic of `canCopyPath`?
> > > > Should it be
> > > > > > > > associated
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > specific
> > destination path? e.g. It can be
> > > > copied to local, but
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > remote FS.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > For the S3 (both for
> > SDKv2 and s5cmd
> > > > implementations), the
> > > > > > > > copying
> > > > > > > > > > > > > > > > > > > > > > > > > > > > direction
> > (upload/download) doesn't matter. I
> > > > don't know about
> > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > > > > > > > file systems, I
> > haven't investigated anything
> > > > besides S3.
> > > > > > > > > > > > > > > > Nevertheless
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > > > > > > wouldn't worry too
> > much about it, since we can
> > > > start with the
> > > > > > > > > > > > simple
> > > > > > > > > > > > > > > > > > > > > > > > > > > > `canCopyPath` that
> > handles both directions. If
> > > > this will become
> > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > > > > > > > in the future, adding
> > directional
> > > > `canDownloadPath` or
> > > > > > > > > > > > > > > > `canUploadPath`
> > > > > > > > > > > > > > > > > > > > > > > > > > > > would be a backward
> > compatible change, so we can
> > > > safely extend it
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > future if needed.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Is the
> > existing interface
> > > > `DuplicatingFileSystem`
> > > > > > > > > > > > > > > > feasible/enough
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > this case?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Good question. The
> > intention and use case behind
> > > > > > > > > > > > > > > > > > > > `DuplicatingFileSystem`
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > > different. It marks
> if
> > `FileSystem` can quickly
> > > > copy/duplicate
> > > > > > > > > > > > files
> > > > > > > > > > > > > > > > > > > > > > > > > > > > in the remote
> > `FileSystem`. For example an
> > > > equivalent of a hard
> > > > > > > > > > > > link
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > > > bumping a reference
> > count in the remote system.
> > > > That's a bit
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > > > > > > to copy paths between
> > remote and local file
> > > > systems.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > However, it could
> > arguably be unified under one
> > > > interface where
> > > > > > > > we
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > > > > > re-use or re-name
> > `canFastDuplicate(Path, Path)` to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > `canFastCopy(Path,
> > Path)` with the following use
> > > > cases:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > -
> > `canFastCopy(remoteA, remoteB)` returns true -
> > > > current
> > > > > > > > equivalent
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > `DuplicatingFileSystem` - quickly duplicate/hard
> > > > link remote path
> > > > > > > > > > > > > > > > > > > > > > > > > > > > - `canFastCopy(local,
> > remote)` returns true - FS
> > > > can natively
> > > > > > > > > > > > upload
> > > > > > > > > > > > > > > > > > > > > > > > local
> > > > > > > > > > > > > > > > > > > > > > > > > > > > file to a remote
> > location
> > > > > > > > > > > > > > > > > > > > > > > > > > > > -
> `canFastCopy(remote,
> > local)` returns true - FS
> > > > can natively
> > > > > > > > > > > > > > > > download
> > > > > > > > > > > > > > > > > > > > > > > > > > > > local file from a
> > remote location
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe indeed that's a
> > better solution vs having
> > > > two separate
> > > > > > > > > > > > > > > > interfaces
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > copying and
> > duplicating?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Will the
> > interface extracting introduce a
> > > > break change?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > No. The signature of
> > the existing abstract
> > > > `FileSystem` class
> > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > remain
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the same. Only
> > most/all of the abstract methods
> > > > would be
> > > > > > > > > > > > > > > > > > > > > > > > > > > > pulled out to the
> > interface and abstract
> > > > `FileSystem` would
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > > > > new interface.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Piotrek
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > pon., 6 maj 2024 o
> > 04:55 Zakelly Lan <
> > > > zakelly....@gmail.com>
> > > > > > > > > > > > > > > > > > > > napisał(a):
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Piotr,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for
> the
> > proposal. It's meaningful to
> > > > speed up the state
> > > > > > > > > > > > > > > > > > > > > > > > download.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > get into some
> > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. What is
> the
> > semantic of `canCopyPath`?
> > > > Should it be
> > > > > > > > associated
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > specific
> > destination path? e.g. It can be
> > > > copied to local, but
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > remote FS.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Is the
> > existing interface
> > > > `DuplicatingFileSystem`
> > > > > > > > > > > > > > > > feasible/enough
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > this case?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Will the
> > interface extracting introduce a
> > > > break change?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Zakelly
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May
> 2,
> > 2024 at 6:50 PM Aleksandr
> > > > Pilipenko <
> > > > > > > > > > > > > > > > z3d...@gmail.com
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi
> > Piotr,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> Thanks
> > for the proposal.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How
> > adding a s5cmd will affect memory
> > > > footprint? Since this
> > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > native
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > binary, memory consumption will not be
> > > > controlled by JVM or
> > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > Aleksandr
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On
> > Thu, 2 May 2024 at 11:12, Hong Liang <
> > > > h...@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Hi Piotr,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Thanks for the FLIP! Nice to see work
> > > > to improve the
> > > > > > > > > > > > filesystem
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > performance. +1 to future work to
> > > > improve the upload speed
> > > > > > > > as
> > > > > > > > > > > > > > > > > > > > well.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > would be useful for jobs with large
> > > > state and high Async
> > > > > > > > > > > > > > > > > > > > > > > > > > > > checkpointing
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > times.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Some thoughts on the configuration, it
> > > > might be good for us
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2x
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > points of configurability for future
> > > > proofing:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > 1/ Configure the implementation of
> > > > PathsCopyingFileSystem
> > > > > > > > > > > > used,
> > > > > > > > > > > > > > > > > > > > > > > > maybe
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > config, or by ServiceResources (this
> > > > would allow us to use
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > alternative clouds/Implement S3 SDKv2
> > > > support if we want
> > > > > > > > this
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > future). Also this could be used as a
> > > > feature flag to
> > > > > > > > > > > > determine
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > should be using this new native file
> > > > copy support.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > 2/ Configure the location of the s5cmd
> > > > binary (version
> > > > > > > > > > > > control
> > > > > > > > > > > > > > > > > > > > > > > > etc.),
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > you have mentioned in the FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > Hong
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > On Thu, May 2, 2024 at 9:40 AM
> > > > Muhammet Orazov
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > <mor+fl...@morazow.com.invalid> wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Hey Piotr,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Thanks for the proposal! It would
> > > > be great improvement!
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Some questions from my side:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > In order to configure s5cmd
> > > > Flink’s user would need
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > to specify path to the s5cmd
> > > > binary.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Could you please also add the
> > > > configuration property
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > for this? An example showing how
> > > > users would set this
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > parameter would be helpful.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Would this affect any filesystem
> > > > connectors that use
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > FileSystem[1][2] dependencies?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > Muhammet
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > [1]:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > [2]:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > On 2024-04-30 13:15, Piotr
> > > > Nowojski wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > Hi all!
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > I would like to put under
> > > > discussion:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > FLIP-444: Native file copy
> > > > support
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > https://cwiki.apache.org/confluence/x/rAn9EQ
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > This proposal aims to speed up
> > > > Flink recovery times, by
> > > > > > > > > > > > > > > > > > > > > > > > speeding
> > > > > > > > > > > > > > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > state
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > download times. However in the
> > > > future, the same
> > > > > > > > mechanism
> > > > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > used to speed up state
> > > > uploading
> > > > > > > > > > > > > > > > > > > > (checkpointing/savepointing).
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > I'm curious to hear your
> > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > Piotrek
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Hangxiang.
> > > > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Hangxiang.
> > > >
> >
>
