Hi Denys,
On 24/01/25 04:05, Denys Dmytriyenko wrote:
> >+do_configure[network] = "1" > >+ > >+do_configure: prepend() { >
>+ if [ -n "${http_proxy}" ]; then > >+ export
HTTP_PROXY=${http_proxy} > >+ export http_proxy=${http_proxy} > >+ fi
> >+ >
ZjQcmQRYFpfptBannerStart
This message was sent from outside of Texas Instruments.
Do not click links or open attachments unless you recognize the source
of this email and know the content is safe.
Report Suspicious
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/G3vK!uPdqXRfP1m07amYkfLMjnW_0WPoGAUc_xiEhDJ8PA6NpT0xA8jTf-KB9808zSr3fgbwdqDV_S4_t$>
ZjQcmQRYFpfptBannerEnd
> >+do_configure[network] = "1"
> >+
> >+do_configure:prepend() {
> >+ if [ -n "${http_proxy}" ]; then
> >+ export HTTP_PROXY=${http_proxy}
> >+ export http_proxy=${http_proxy}
> >+ fi
> >+
> >+ if [ -n "${https_proxy}" ]; then
> >+ export HTTPS_PROXY=${https_proxy}
> >+ export https_proxy=${https_proxy}
> >+ fi
>
>
> Are these checks needed? If it is unset then exporting doesn't hurt,
> could replace all this with:
>
> export http_proxy
> export https_proxy
> do_configure[network] = "1"
>
> >+do_compile[network] = "1"
> >+
> >+do_compile:prepend() {
> >+ # Used to download cmake dependencies when behind a proxy
> >+ if [ -n "${http_proxy}" ]; then
> >+ export HTTP_PROXY=${http_proxy}
> >+ export http_proxy=${http_proxy}
> >+ fi
> >+
> >+ if [ -n "${https_proxy}" ]; then
> >+ export HTTPS_PROXY=${https_proxy}
> >+ export https_proxy=${https_proxy}
> >+ fi
>
> Do you need network during compile? Seems to only be used in configure.
Both of the above will get you in trouble later on - these are considered a
very bad antisocial behavior and an almost sure way to strip off of any Yocto
Project Compatible status...
I completely understand your concerns about following best practices in
Yocto.
I was aware that the Yocto documentation now recommends limiting network
access
to the do_fetch task. However, I'd like to point out that many official
Yocto Project
layers still enable network access after do_fetch.
For example, in meta-openembedded, there are several recipes that enable
network access
during do_configure and do_compile. Following are few references,
- meta-oe/recipes-support/crucible [1]
- meta-oe/recipes-dbs/influxdb [2]
- meta-oe/recipes-test/syzkaller [3]
- meta-arago-extras/recipes-devtools/websocketd [4]
These are few examples which I founded just my doing a grep on layers
used in TI SDK
but I'm sure there would be more such recipes in other layers which are
under Yocto
Project banner & still enabling network after do_fetch.
It's interesting to note that the Yocto Project documentation itself
provides example
of how to enable network access outside of do_fetch [5]. This highlights
the complexity
of building certain packages, especially those with dynamic dependencies.
Projects like Golang and TensorFlow have more traction towards x86-based
systems, making
it challenging to revamp their build workflows. As a result, many Yocto
layers, including
meta-imx-ml [6], meta-st-x-linux-ai [7], and even meta-tensorflow (the
official Yocto
Project layer for TensorFlow), builds TFLite by enabling network access
during
do_configure and do_compile [8]
Furthermore, the TensorFlow CMake build system is not intelligent enough
to detect the
presence of existing dependencies. This makes it difficult to cleanly
build TensorFlow
Lite without network connectivity. I'd like to point out that
meta-webosose:recipes-upstreamable
which attempts to build TFLite without network connectivity [9], but it
ends up with a
recipe that uses dirty hacks to delete CMake files for certain delegates.
This approach makes the recipe non-scalable and difficult to migrate to
newer versions
(I believe that's one of the reason they are still on TFLite v2.9 for
past 2 years!)
Similarly, Golang-based projects face similar challenges. That's why
we're enabling network
access at do_compile for websocketd in meta-arago [4]
Btw, I'd like to mention that I've already addressed the do_compile
network access issue
by switching to setuptools3 [10]. Nevertheless, I still require network
access at the
do_configure stage to complete the build process. I hope this provides
more context
to the issue.
Ideally, if we want to fix this problem, Yocto Project should educate
projects like Golang,
Tensorflow, etc. about the traction of these projects in non x86 based
systems
where majority of people end up using Yocto project which doesn't like
fetching dependencies
at runtime. Post which several folks can collaborate & ensure to add a
build mechanism
which is Yocto Project friendly!
[1]:
https://git.openembedded.org/meta-openembedded/plain/meta-oe/recipes-support/crucible/crucible_2023.11.02.bb?h=scarthgap
[2]:
https://git.openembedded.org/meta-openembedded/plain/meta-oe/recipes-dbs/influxdb/influxdb_1.8.10.bb?h=scarthgap
[3]:
https://git.openembedded.org/meta-openembedded/plain/meta-oe/recipes-test/syzkaller/syzkaller_git.bb?h=scarthgap
[4]:
https://git.ti.com/cgit/arago-project/meta-arago/commit/meta-arago-extras/recipes-devtools/websocketd/websocketd_0.4.1.bb?h=scarthgap&id=29ef988fae74cd26c418d0e728f757b3c26f23da
[5]:
https://docs.yoctoproject.org/5.0.6/migration-guides/migration-4.0.html#fetching-changes
[6]:
https://github.com/nxp-imx/meta-imx/blob/scarthgap-6.6.36-2.1.0/meta-imx-ml/recipes-libraries/tensorflow-lite/tensorflow-lite_2.16.2.bb#L46
[7]:
https://github.com/STMicroelectronics/meta-st-x-linux-ai/blob/scarthgap/recipes-frameworks/tensorflow-lite/tensorflow-lite_git.bb#L75
[8]:
https://git.yoctoproject.org/meta-tensorflow/tree/recipes-framework/tensorflow/tensorflow.inc#n64
[9]:
https://github.com/webosose/meta-webosose/blob/master/meta-webos/recipes-upstreamable/tensorflow-lite/tensorflow-lite_2.9.3.inc#L140
[10]: https://lists.yoctoproject.org/g/meta-arago/message/15754
---
Chirag
I'd suggest spending some time trying to find a way to do any and all
downloads inside the do_fetch() task, which is the only one allowed to have
network access.
In other words - do_configure() and do_compile() are definitely not the right
places to do any network fetching...
--
Denys
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15755):
https://lists.yoctoproject.org/g/meta-arago/message/15755
Mute This Topic: https://lists.yoctoproject.org/mt/110775616/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/meta-arago/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
