File.ls doesn’t appear to work with wildcards, I get this error `{:error,
:enoent}`. But File.ls(“lib”) is also fast. The C script and NIF are beyond my
knowledge.
> On Oct 14, 2019, at 12:35 PM, José Valim <[email protected]>
> wrote:
>
> Right. Now try running File.ls in some directories and see if that is enough
> to reproduce the issue. Then I would implement a C script that calls ls as a
> NIF, see if it is fast or not. And so on.
>
>
> José Valim
> www.plataformatec.com.br <http://www.plataformatec.com.br/>
> Skype: jv.ptec
> Founder and Director of R&D
>
>
> On Mon, Oct 14, 2019 at 8:29 PM Michael St Clair <[email protected]
> <mailto:[email protected]>> wrote:
> What is the best way to experiment with that? As I am showing that using the
> ls command is much faster (times below)
>
> On my Mac:
> iex(3)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc
> <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
> 0.095601
>
> iex(4)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc
> <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
> 0.071896
>
>
>
> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |>
> String.split("\n") end |> :timer.tc <http://timer.tc/> |> elem(0) |>
> Kernel./(1_000_000)
>
> 0.034176
>
> iex(6)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |>
> to_string() |> String.split("\n") end |> :timer.tc <http://timer.tc/> |>
> elem(0) |> Kernel./(1_000_000)
>
> 0.028481
>
>
>
> On our production instance running docker (elixir:1.9.1-alpine):
>
> iex(zipbooks@phoenix-360h)1> fn -> Path.wildcard("lib/**/*.ex") end |>
> :timer.tc <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
> 0.448345
>
> iex(zipbooks@phoenix-360h)2> fn -> Path.wildcard("lib/**/*.ex") end |>
> :timer.tc <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
>
> 0.35429
>
>
>
> iex(zipbooks@phoenix-360h)7> fn -> "ls lib/**/*.ex" |> String.to_charlist()
> |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc
> <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
> 0.00361
>
> iex(zipbooks@phoenix-360h)8> fn -> "ls lib/**/*.ex" |> String.to_charlist()
> |> :os.cmd() |> to_string() |> String.split("\n") end |> :timer.tc
> <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
> 0.003731
>
>
>
> Local docker container with mounted directories(elixir:1.9.1-alpine):
>
> iex(1)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc
> <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
> 0.881692
>
> iex(2)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc
> <http://timer.tc/> |> elem(0) |> Kernel./(1_000_000)
>
>
> 0.897629
>
>
>
> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> to_string() |>
> String.split("\n") end |> :timer.tc <http://timer.tc/> |> elem(0) |>
> Kernel./(1_000_000)
>
> 0.031227
>
> iex(4)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |>
> to_string() |> String.split("\n") end |> :timer.tc <http://timer.tc/> |>
> elem(0) |> Kernel./(1_000_000)
>
>
> 0.031241
>
>
> On Monday, October 14, 2019 at 12:14:57 PM UTC-6, José Valim wrote:
> There is no way to patch out Mix.Utils to run your own code. The proper fix
> would be to figure out why the path traversal is slow on Elixir (it is
> implemented on top of filelib:wildcard) so we can fix it rather permanently.
> How long does "Path.wildcard("lib/**/*.ex")" take? Can you implement your own
> path traversal in Elixir? Is it faster than Path.wildcard?
>
>
> José Valim
> www.plataformatec.com.br <http://www.plataformatec.com.br/>
> Skype: jv.ptec
> Founder and Director of R&D
>
>
> On Mon, Oct 14, 2019 at 8:08 PM Michael St Clair <[email protected] <>>
> wrote:
> I've been doing some digging around to see how to make phoenix run faster in
> Docker. Without this plug the app runs just as fast as native
> `plug(Phoenix.CodeReloader`. I think I have traced it down to this
> https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L237
>
> <https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L237>.
> Running just that function explains most if not all of the extra load time
> on my endpoints. However this code `"ls lib/**/*.ex" |> String.to_charlist()
> |> :os.cmd() |> to_string() |> String.split("\n") |> IO.inspect` appears to
> do the same thing and much faster. I'm just running into the problem of how
> to override that Mix.Utils file to test it here
> https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L36
>
> <https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L36>.
> Any suggestions?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com
>
> <https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com
>
> <https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K8G5py3P0eGxQ83C%2Bk0Ajwyjc4BGFDUf3dPt%3DS2rDGUQ%40mail.gmail.com
>
> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K8G5py3P0eGxQ83C%2Bk0Ajwyjc4BGFDUf3dPt%3DS2rDGUQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/9C339FAC-8D80-47F3-B6A2-F90AAC7AFD88%40gmail.com.
