Hi Gabor,
The issue is that, read permission is not getting checked when Flink FileSource
is listing the files under given source directory.
This is happening as Security Manager is coming as null.
public String[] list() {
SecurityManager security = System.getSecurityManager(); -> Here Secur
Hi Yanfei,
I am facing this issue on jdk1.8/11.
Thanks for giving pointer, I will try to set Security manager and check the
behaviour.
Regards,
Kirti Dhar
-Original Message-
From: Yanfei Lei
Sent: Wednesday, March 6, 2024 4:37 PM
To: Kirti Dhar Upadhyay K
Cc: User@flink.apache.org
S
Hi Hang,
You got it right. The problem is exactly at the same line where you pointed [1].
I have used below solution as of now.
```
If(!Files.isReadable(Paths.get(fileStatus.getPath().getPath( {
throw new FlinkRuntimeException("Cannot list files under " +
fileStatus.getPath());
}
fi
Hi, can you provide more details about this Flink batch job? For instance,
through a flame graph, the threads are found spending most of their time on
some certain tasks.
--
Best！
Xuyang
At 2024-03-07 08:40:32, "Charles Tan" wrote:
Hi all,
I have been looking into using Flink
Hi, IMO, both the SQL Client and the Restful API can provide connections to the
SQL Gateway service for submitting jobs. A slight difference is that the SQL
Client also offers a command-line visual interface for users to view results.
In your production scenes, placing the SQL to be submitted in
Hi all,
I have been looking into using Flink in batch mode to process Iceberg
tables. I noticed that the performance for queries in Flink's batch mode is
quite slow, especially when compared to Spark. I'm wondering if there are
any configurations that I'm missing to get better performance out of
r
I'm reading the deployment guide[1] and wanted to check my understanding.
For deploying a SQL job into production, would the pattern be to write the
SQL in a file that's under source control, and pass that file as an
argument to SQL Client with -f argument (as in this docs example[2])?
Or script a
Hi Kirti,
Not sure what is the exact issue here but I'm not convinced that
having FlinkSecurityManager is going to solve it.
Here is the condition however:
* cluster.intercept-user-system-exit != DISABLED (this must be changed)
* cluster.processes.halt-on-fatal-error == false (this is good by defa
Hi, Kirti.
Could you please provide the stack trace of this NPE? I check the code and
I think maybe the problem lies in LocalFileSystem#listStatus.
The code in line 161[1] may return null, which will let
LocalFileSystem#listStatus return null. Then the `containedFiles` is null
and the NPE occurs.
Hi Kirti Dhar,
What is your java version? I guess this problem may be related to
FLINK-33309[1]. Maybe you can try adding "-Djava.security.manager" to
the java options.
[1] https://issues.apache.org/jira/browse/FLINK-33309
Kirti Dhar Upadhyay K via user 于2024年3月6日周三 18:10写道：
>
> Hi Team,
>
>
>
>
Hi Gabriele, use (or extend) the window operator provided by Flink is a
better idea. A window operator in Flink manages two types of state:
- Window state: accumlate data for windows, and provide data to window
function when a window comes to its end time.
- Timer state: store the end tim
Hi Team,
I am using Flink File Source with Local File System.
I am facing an issue, if source directory does not has read permission, it is
returning the list of files as null instead of throwing permission exception
(refer the highlighted line below), resulting in NPE.
final FileStatus[] conta
12 matches
Mail list logo
