[
https://issues.apache.org/jira/browse/CAMEL-25670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25670:
--------------------------------
Description:
With {{camel run --source-dir}}, dev mode watches every file under the source
directory, recursively ({{routesReloadPattern=*}} and
{{routesReloadDirectoryRecursive=true}}, set in {{Run.java}}). When the routes
read and write data files in a folder under it, their own file activity counts
as a change and reloads the routes, again and again.
An example from a local-model benchmark:
{noformat}
- route:
id: courier
from:
uri: file
parameters:
directoryName: samples
noop: true
steps:
- to:
uri: file
parameters:
directoryName: inbox
- route:
id: invoices
from:
uri: file
parameters:
directoryName: inbox
steps:
- log:
message: "Received ${file:name}"
{noformat}
Every reload restarts {{courier}} with an empty in-memory idempotent
repository, so it copies {{samples/*}} into {{inbox/}} again. That is a change
under the source directory, so the routes reload again. With the default 2 s
scan, the routes reloaded every 2.0 s for as long as the integration ran (63
reloads in 2 minutes), and every reload processed the same files again. To a
beginner it looks like duplicate processing, not like dev mode.
h3. Proposal
# {{--data-dir}} for {{camel run}}: one or more folders, comma-separated and
relative to the source directory, that dev mode does not watch, for example
{{camel run --source-dir . --data-dir=samples,inbox,archive}}. Also as a
property, so it can be set in application.properties. Other tools can use it
too: {{camel export}} can leave the data out, and the AI file tools can tell
data from source.
# A warning when reloads repeat: if the routes reload, say, 5 times within 15
seconds, log one WARN that names the folder of the files that triggered them,
for example "Dev mode reloaded 5 times in 10 s, each time for files under
inbox/. If that folder holds data, start with --data-dir=inbox." The reload
strategy already knows which files changed, so this needs no look at the
endpoints.
# The built-in examples (the {{camel run --example}} and the example browser
ones, {{dsl/camel-jbang/camel-jbang-core/src/main/resources/examples}}): check
the ones that use files, and give those that write into their own folder a data
dir in their application.properties. Of the 11 that use the file component,
three write into their own folder:
** {{connect/file-processing}}: to {{inbox}} and {{archive}}, and its {{inbox}}
consumer moves files to {{done}} and {{failed}};
** {{fail-well/error-handling}}: to {{orders}} and {{parked}};
** {{transform/csv-to-json}}: to {{outbox}}.
The other eight only read their files ({{noop=true}}): no change, so no loop.
A shorter scan interval (CAMEL-25661) makes the loop faster: at 250 ms the same
example reloaded 178 times.
_Claude Code on behalf of davsclaus_
was:
With {{camel run --source-dir}}, dev mode watches every file under the source
directory, recursively ({{routesReloadPattern=*}} and
{{routesReloadDirectoryRecursive=true}}, set in {{Run.java}}). When the routes
read and write data files in a folder under it, their own file activity counts
as a change and reloads the routes, again and again.
An example from a local-model benchmark:
{noformat}
- route:
id: courier
from:
uri: file
parameters:
directoryName: samples
noop: true
steps:
- to:
uri: file
parameters:
directoryName: inbox
- route:
id: invoices
from:
uri: file
parameters:
directoryName: inbox
steps:
- log:
message: "Received ${file:name}"
{noformat}
Every reload restarts {{courier}} with an empty in-memory idempotent
repository, so it copies {{samples/*}} into {{inbox/}} again. That is a change
under the source directory, so the routes reload again. With the default 2 s
scan, the routes reloaded every 2.0 s for as long as the integration ran (63
reloads in 2 minutes), and every reload processed the same files again. To a
beginner it looks like duplicate processing, not like dev mode.
h3. Proposal
# {{--data-dir}} for {{camel run}}: one or more folders, comma-separated and
relative to the source directory, that dev mode does not watch, for example
{{camel run --source-dir . --data-dir=samples,inbox,archive}}. Also as a
property, so it can be set in application.properties. Other tools can use it
too: {{camel export}} can leave the data out, and the AI file tools can tell
data from source.
# A warning when reloads repeat: if the routes reload, say, 5 times within 15
seconds, log one WARN that names the folder of the files that triggered them,
for example "Dev mode reloaded 5 times in 10 s, each time for files under
inbox/. If that folder holds data, start with --data-dir=inbox." The reload
strategy already knows which files changed, so this needs no look at the
endpoints.
A shorter scan interval (CAMEL-25661) makes the loop faster: at 250 ms the same
example reloaded 178 times.
_Claude Code on behalf of davsclaus_
> camel-jbang - camel run: --data-dir for data folders that dev mode does not
> watch
> ---------------------------------------------------------------------------------
>
> Key: CAMEL-25670
> URL: https://issues.apache.org/jira/browse/CAMEL-25670
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> With {{camel run --source-dir}}, dev mode watches every file under the source
> directory, recursively ({{routesReloadPattern=*}} and
> {{routesReloadDirectoryRecursive=true}}, set in {{Run.java}}). When the
> routes read and write data files in a folder under it, their own file
> activity counts as a change and reloads the routes, again and again.
> An example from a local-model benchmark:
> {noformat}
> - route:
> id: courier
> from:
> uri: file
> parameters:
> directoryName: samples
> noop: true
> steps:
> - to:
> uri: file
> parameters:
> directoryName: inbox
> - route:
> id: invoices
> from:
> uri: file
> parameters:
> directoryName: inbox
> steps:
> - log:
> message: "Received ${file:name}"
> {noformat}
> Every reload restarts {{courier}} with an empty in-memory idempotent
> repository, so it copies {{samples/*}} into {{inbox/}} again. That is a
> change under the source directory, so the routes reload again. With the
> default 2 s scan, the routes reloaded every 2.0 s for as long as the
> integration ran (63 reloads in 2 minutes), and every reload processed the
> same files again. To a beginner it looks like duplicate processing, not like
> dev mode.
> h3. Proposal
> # {{--data-dir}} for {{camel run}}: one or more folders, comma-separated and
> relative to the source directory, that dev mode does not watch, for example
> {{camel run --source-dir . --data-dir=samples,inbox,archive}}. Also as a
> property, so it can be set in application.properties. Other tools can use it
> too: {{camel export}} can leave the data out, and the AI file tools can tell
> data from source.
> # A warning when reloads repeat: if the routes reload, say, 5 times within 15
> seconds, log one WARN that names the folder of the files that triggered them,
> for example "Dev mode reloaded 5 times in 10 s, each time for files under
> inbox/. If that folder holds data, start with --data-dir=inbox." The reload
> strategy already knows which files changed, so this needs no look at the
> endpoints.
> # The built-in examples (the {{camel run --example}} and the example browser
> ones, {{dsl/camel-jbang/camel-jbang-core/src/main/resources/examples}}):
> check the ones that use files, and give those that write into their own
> folder a data dir in their application.properties. Of the 11 that use the
> file component, three write into their own folder:
> ** {{connect/file-processing}}: to {{inbox}} and {{archive}}, and its
> {{inbox}} consumer moves files to {{done}} and {{failed}};
> ** {{fail-well/error-handling}}: to {{orders}} and {{parked}};
> ** {{transform/csv-to-json}}: to {{outbox}}.
> The other eight only read their files ({{noop=true}}): no change, so no loop.
> A shorter scan interval (CAMEL-25661) makes the loop faster: at 250 ms the
> same example reloaded 178 times.
> _Claude Code on behalf of davsclaus_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
