[jira] [Created] (FLINK-32142) Apple Silicon Support: Unable to Build Flink Project due to "Bad CPU Type" Error
Elphas Toringepi created FLINK-32142:
Summary: Apple Silicon Support: Unable to Build Flink Project due
to "Bad CPU Type" Error
Key: FLINK-32142
URL: https://issues.apache.org/jira/browse/FLINK-32142
Project: Flink
Issue Type: Bug
Components: Build System, Runtime / Web Frontend
Affects Versions: 1.15.4, 1.16.1, 1.15.3, 1.15.2, 1.17.0, 1.16.0, 1.15.1,
1.15.0, 1.16.2, 1.18.0, 1.17.1, 1.15.5, 1.16.3, 1.17.2
Environment: Apple Silicon architecture (M2 Pro)
macOS Ventura (Version 13.3.1)
Reporter: Elphas Toringepi
Fix For: 1.18.0
Attempting to build the Flink project on Apple Silicon architecture results in
an error related to the execution of the frontend-maven-plugin.
The error message indicates that the plugin fails to run
"flink/flink-runtime-web/web-dashboard/node/node" program due to a "Bad CPU
type in executable" error.
{code:java}
[ERROR] Failed to execute goal
com.github.eirslett:frontend-maven-plugin:1.11.0:npm (npm install) on project
flink-runtime-web: Failed to run task: 'npm ci --cache-max=0 --no-save
${npm.proxy}' failed. java.io.IOException: Cannot run program
"flink/flink-runtime-web/web-dashboard/node/node" (in directory
"/Users/elphas/src/flink/flink-runtime-web/web-dashboard"): error=86, Bad CPU
type in executable{code}
Steps to Reproduce:
# Clone the Flink project repository.
# Attempt to build the project on an Apple Silicon device.
# Observe the error message mentioned above.
{code:java}
git clone https://github.com/apache/flink.git
cd flink
./mvnw clean package -DskipTests=
{code}
Proposed Solution
Upgrade frontend-maven-plugin from version 1.11.0 to the latest version, 1.12.1.
frontend-maven-plugin version 1.11.0 downloads x64 binaries
node-v16.13.2-darwin-x64.tar.gz instead of the arm64 binaries.
Support for arm64 has been available for frontend-maven-plugin since version
2. https://github.com/eirslett/frontend-maven-plugin/pull/970
{code:java}
[DEBUG] Executing command line
[/Users/elphas/src/flink/flink-runtime-web/web-dashboard/node/node, --version]
[INFO] Installing node version v16.13.2 [DEBUG] Creating temporary directory
/flink/flink-runtime-web/web-dashboard/node/tmp [INFO] Unpacking
~/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-darwin-x64.tar.gz
into flink/flink-runtime-web/web-dashboard/node/tmp{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Maven plugin to detect issues early on
Hi Emre, Thanks for your proposal. It looks very interesting! Please pay attention that most connectors have been externalized. Will your proposed plug be used for building Flink Connectors or Flink itself? Furthermore, it would be great if you could elaborate features wrt best practices so that we could understand how the plugin will help us. Afaik, FLIP is recommended for improvement ideas that will change public APIs. I am not sure if a new maven plugin belongs to it. Best regards, Jing On Tue, May 16, 2023 at 11:29 AM Kartoglu, Emre wrote: > Hello all, > > Myself and 2 colleagues developed a Maven plugin (no support for Gradle or > other build tools yet) that we use internally to detect potential issues in > Flink apps at compilation/packaging stage: > > > * Known connector version incompatibilities – so far covering Kafka > and Kinesis > * Best practices e.g. setting operator IDs > > We’d like to make this open-source. Ideally with the Flink community’s > support/mention of it on the Flink website, so more people use it. > > Going forward, I believe we have at least the following options: > > * Get community support: Create a FLIP to discuss where the plugin > should live, what kind of problems it should detect etc. > * We still open-source it but without the community support (if the > community has objections to officially supporting it for instance). > > Just wanted to gauge the feeling/thoughts towards this tool from the > community before going ahead. > > Thanks, > Emre > >
Re: [DISCUSS] Release Flink 1.17.1
Thanks Weijie for the effort. Best regards, Yuxia - 原始邮件 - 发件人: "Jing Ge" 收件人: "dev" 抄送: "kevin lam" 发送时间: 星期五, 2023年 5 月 19日 下午 7:52:18 主题: Re: [DISCUSS] Release Flink 1.17.1 Hi Weijie, Thanks for your effort and sharing the status! Best regards, Jing On Fri, May 19, 2023 at 11:50 AM weijie guo wrote: > Hi Jing, > > Thank you for your attention! > > I have cut off the code for release-1.17.1-rc1 and started deploying it to > the apache repository, while preparing the pull request for the blog post. > The specific released time depends on the speed at which they are > completed. VOTE thread will be launched no later than next Monday at the > latest. > > Best regards, > > Weijie > > > Jing Ge 于2023年5月19日周五 14:48写道： > > > Hi Weijie, > > > > just wondering how the process is going. Will the RC1 be released today? > > Thanks for driving this! > > > > Best regards, > > Jing > > > > On Wed, May 17, 2023 at 5:02 AM weijie guo > > wrote: > > > > > Hi Kevin, > > > > > > If everything goes smoothly, I will release the RC1 version of 1.17.1 > > this > > > Thursday or Friday. > > > > > > Best regards, > > > > > > Weijie > > > > > > > > > Kevin Lam 于2023年5月16日周二 22:53写道： > > > > > > > Hi! Thanks for doing this release. I'm looking forward to some of the > > bug > > > > fixes, is there a date set for the release of 1.17.1? > > > > > > > > On Mon, May 15, 2023 at 6:10 AM Lijie Wang > > > > > wrote: > > > > > > > > > +1 for the release. > > > > > > > > > > Best, > > > > > Lijie > > > > > > > > > > Jing Ge 于2023年5月15日周一 17:07写道： > > > > > > > > > > > +1 for releasing 1.17.1 > > > > > > > > > > > > Best Regards, > > > > > > Jing > > > > > > > > > > > > On Thu, May 11, 2023 at 10:03 AM Martijn Visser < > > > > > martijnvis...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > +1, thanks for volunteering! > > > > > > > > > > > > > > On Thu, May 11, 2023 at 9:23 AM Xintong Song < > > > tonysong...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > I'll help with the steps that require PMC privileges. > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 11, 2023 at 3:12 PM Jingsong Li < > > > > jingsongl...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 for releasing 1.17.1 > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Jingsong > > > > > > > > > > > > > > > > > > On Thu, May 11, 2023 at 1:29 PM Gyula Fóra < > > > gyula.f...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > +1 for the release > > > > > > > > > > > > > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > > > On Thu, 11 May 2023 at 05:35, Yun Tang > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 for release flink-1.17.1 > > > > > > > > > > > > > > > > > > > > > > The blocker issue might cause silent incorrect data, > it's > > > > > better > > > > > > to > > > > > > > > > have a > > > > > > > > > > > fix release ASAP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > > Yun Tang > > > > > > > > > > > > > > > > > > > > > > From: weijie guo > > > > > > > > > > > Sent: Thursday, May 11, 2023 11:08 > > > > > > > > > > > To: dev@flink.apache.org ; > > > > > > > > tonysong...@gmail.com > > > > > > > > > < > > > > > > > > > > > tonysong...@gmail.com> > > > > > > > > > > > Subject: [DISCUSS] Release Flink 1.17.1 > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would like to discuss creating a new 1.17 patch > release > > > > > > (1.17.1). > > > > > > > > The > > > > > > > > > > > last 1.17 release is nearly two months old, and since > > then, > > > > 66 > > > > > > > > tickets > > > > > > > > > have > > > > > > > > > > > been closed [1], of which 14 are blocker/critical [2]. > > > Some > > > > of > > > > > > > them > > > > > > > > > are > > > > > > > > > > > quite important, such as FLINK-31293 [3] and > FLINK-32027 > > > > [4]. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am not aware of any unresolved blockers and there are > > no > > > > > > > > in-progress > > > > > > > > > > > tickets [5]. > > > > > > > > > > > Please let me know if there are any issues you'd like > to > > be > > > > > > > included > > > > > > > > in > > > > > > > > > > > this release but still not merged. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the community agrees to create this new patch > > release, I > > > > > could > > > > > > > > > > > volunteer as the release manager > > > > > > > > > > > and Xintong can help with actions that require a PMC > > role. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Weijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]
[jira] [Created] (FLINK-32143) Automatic cleanup of terminated flinkdeployments
Paul Lin created FLINK-32143: Summary: Automatic cleanup of terminated flinkdeployments Key: FLINK-32143 URL: https://issues.apache.org/jira/browse/FLINK-32143 Project: Flink Issue Type: New Feature Components: Kubernetes Operator Reporter: Paul Lin Currently, if a job turns into terminated status (e.g. FINISHED or FAILED), the flinkdeployment remains until a manual cleanup is performed. We could add a configuration named `kubernetes.operator.deployment.cleanup.delay`, and clean up terminated deployments after the delay. The delay is default to `-1` which means the cleanup is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-312: Add Yarn ACLs to Flink Containers
Thanks for creating this FLIP. This sounds like a useful feature to make the Flink applications running on YARN cluster more securely. However, I think we still miss some important parts in the FLIP. 1. Which hadoop versions this FLIP relies on? 2. We need to describe a bit more about how the YARN ACLs works. 3. Does the ACLs only apply to the logs? How about the Flink JobManager UI? Best, Yang Venkatakrishnan Sowrirajan 于2023年5月13日周六 08:12写道： > Thanks for the FLIP, Archit. > > +1 from me as well. This would be very useful for us and others in the > community given the same issue was raised earlier as well. > > Regards > Venkata krishnan > > > On Fri, May 12, 2023 at 4:03 PM Becket Qin wrote: > > > Thanks for the FLIP, Archit. > > > > The motivation sounds reasonable and it looks like a straightforward > > proposal. +1 from me. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Fri, May 12, 2023 at 1:30 AM Archit Goyal > > > > > wrote: > > > > > Hi all, > > > > > > I am opening this thread to discuss the proposal to support Yarn ACLs > to > > > Flink containers which has been documented in FLIP-312 < > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP*312*3A*Add*Yarn*ACLs*to*Flink*Containers__;KyUrKysrKys!!IKRxdwAv5BmarQ!bQiA3GX9bFf-w6A9M4Aez7RSMYLdvFtjZnlrOSf6N2nQUFuDdnoJ20uujW8RPY1VbLS9P4AfpnqPmkZZOuQ$ > > > >. > > > > > > This FLIP mentions about providing Yarn application ACL mechanism on > > Flink > > > containers to be able to provide specific rights to users other than > the > > > one running the Flink application job. This will restrict other users > in > > > two ways: > > > > > > * view logs through the Resource Manager job history > > > * kill the application > > > > > > Please feel free to reply to this email thread and share your opinions. > > > > > > Thanks, > > > Archit Goyal > > > > > > > > >
Troubleshooting pyflink import error regarding Json, Kafka, Substring, DataTypes and FileSystem libraries
Gentlemen, I don't if dev is the right place to ask questions about pyflink mismatch problems. If this is not the case kindly guide me where can i register to ask these matters. I have problem with some apache-flink modules. I am running a 1.17.0 apache- flink and I write codes in Colab I faced a problem for import modules from pyflink.table import DataTypes from pyflink.table.descriptors import Schema, Kafka, Json, Rowtime from pyflink.table.catalog import FileSystem not working for me (python version 3.10) All other modules importing fine. I checked with your Github but didn't find these on yours too which means modules are not inside your descriptor.py too. I think it needed installation of connectors but it failed too. Please see the link below: https://github.com/aomidvar/scrapper-price-comparison/blob/d8a10f74101bf96974e769813c33b83d7a71f02b/kafkaconsumer1.ipynb I am running a test after producing the stream ( https://github.com/aomidvar/scrapper-price-comparison/blob/main/kafkaproducer1.ipynb) to Confluent server and I like to do a flink job but the above mentioned modules are not found with the following error in python: --- ImportError Traceback (most recent call last) in () 1 from pyflink.table import DataTypes > 2 from pyflink.table.descriptors import Schema, Kafka, Json, Rowtime 3 from pyflink.table.catalog import FileSystem ImportError: cannot import name 'Kafka' from 'pyflink.table.descriptors' (/usr/local/lib/python3.10/dist-packages/pyflink/table/descriptors.py) --- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt. To view examples of installing some common dependencies, click the "Open Examples" button below. --- I have doubt that if current error is related to a version and dependencies then I have to ask the developer if I do this python 3.8 env is that possible to get solved? Thanks for your time , Alireza Omidvar
[VOTE] Release 1.17.1, release candidate #1
Hi everyone, Please review and vote on the release candidate #1 for the version 1.17.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.17.1-rc1" [5], * website pull request listing the new release and adding announcement blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Release Manager [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352886 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.17.1-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1635/ [5] https://github.com/apache/flink/releases/tag/release-1.17.1-rc1 [6] https://github.com/apache/flink-web/pull/650
