Trying doing File.ls a couple hundred times in the same measurement. Try to get a folder with many files. Also take a look at doing the same for File.stat.
*José Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D On Mon, Oct 14, 2019 at 8:47 PM Michael St Clair <[email protected]> wrote: > File.ls doesn’t appear to work with wildcards, I get this error `{:error, > :enoent}`. But File.ls(“lib”) is also fast. The C script and NIF are > beyond my knowledge. > > On Monday, October 14, 2019 at 12:35:23 PM UTC-6, José Valim wrote: >> >> Right. Now try running File.ls in some directories and see if that is >> enough to reproduce the issue. Then I would implement a C script that calls >> ls as a NIF, see if it is fast or not. And so on. >> >> >> *José Valim* >> www.plataformatec.com.br >> Skype: jv.ptec >> Founder and Director of R&D >> >> >> On Mon, Oct 14, 2019 at 8:29 PM Michael St Clair <[email protected]> >> wrote: >> >>> What is the best way to experiment with that? As I am showing that using >>> the ls command is much faster (times below) >>> >>> On my Mac: >>> >>> iex(3)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) >>> |> Kernel./(1_000_000) >>> >>> 0.095601 >>> >>> iex(4)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) >>> |> Kernel./(1_000_000) >>> >>> 0.071896 >>> >>> >>> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> >>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> >>> Kernel./(1_000_000) >>> >>> 0.034176 >>> >>> iex(6)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> >>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> >>> Kernel./(1_000_000) >>> >>> 0.028481 >>> >>> >>> On our production instance running docker (elixir:1.9.1-alpine): >>> >>> iex(zipbooks@phoenix-360h)1> fn -> Path.wildcard("lib/**/*.ex") end |> : >>> timer.tc |> elem(0) |> Kernel./(1_000_000) >>> >>> 0.448345 >>> >>> iex(zipbooks@phoenix-360h)2> fn -> Path.wildcard("lib/**/*.ex") end |> : >>> timer.tc |> elem(0) |> Kernel./(1_000_000) >>> >>> 0.35429 >>> >>> >>> iex(zipbooks@phoenix-360h)7> fn -> "ls lib/**/*.ex" |> >>> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end >>> |> :timer.tc |> elem(0) |> Kernel./(1_000_000) >>> >>> 0.00361 >>> >>> iex(zipbooks@phoenix-360h)8> fn -> "ls lib/**/*.ex" |> >>> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") end >>> |> :timer.tc |> elem(0) |> Kernel./(1_000_000) >>> >>> 0.003731 >>> >>> >>> Local docker container with mounted directories(elixir:1.9.1-alpine): >>> >>> iex(1)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) >>> |> Kernel./(1_000_000) >>> >>> 0.881692 >>> >>> iex(2)> fn -> Path.wildcard("lib/**/*.ex") end |> :timer.tc |> elem(0) >>> |> Kernel./(1_000_000) >>> >>> 0.897629 >>> >>> >>> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> >>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> >>> Kernel./(1_000_000) >>> >>> 0.031227 >>> >>> iex(4)> fn -> "ls lib/**/*.ex" |> String.to_charlist() |> :os.cmd() |> >>> to_string() |> String.split("\n") end |> :timer.tc |> elem(0) |> >>> Kernel./(1_000_000) >>> >>> 0.031241 >>> >>> On Monday, October 14, 2019 at 12:14:57 PM UTC-6, José Valim wrote: >>>> >>>> There is no way to patch out Mix.Utils to run your own code. The proper >>>> fix would be to figure out why the path traversal is slow on Elixir (it is >>>> implemented on top of filelib:wildcard) so we can fix it rather >>>> permanently. How long does "Path.wildcard("lib/**/*.ex")" take? Can you >>>> implement your own path traversal in Elixir? Is it faster than >>>> Path.wildcard? >>>> >>>> >>>> *José Valim* >>>> www.plataformatec.com.br >>>> Skype: jv.ptec >>>> Founder and Director of R&D >>>> >>>> >>>> On Mon, Oct 14, 2019 at 8:08 PM Michael St Clair <[email protected]> >>>> wrote: >>>> >>>>> I've been doing some digging around to see how to make phoenix run >>>>> faster in Docker. Without this plug the app runs just as fast as native >>>>> `plug(Phoenix.CodeReloader`. I think I have traced it down to this >>>>> https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/utils.ex#L237. >>>>> Running just that function explains most if not all of the extra load time >>>>> on my endpoints. However this code `"ls lib/**/*.ex" |> >>>>> String.to_charlist() |> :os.cmd() |> to_string() |> String.split("\n") |> >>>>> IO.inspect` appears to do the same thing and much faster. I'm just running >>>>> into the problem of how to override that Mix.Utils file to test it here >>>>> https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L36. >>>>> Any suggestions? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elixir-lang-core" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elixir-lang-core/b811d808-04a2-427e-9281-3d337f05de16%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/b46f9a3c-4f0a-479d-98df-a921ffeb65fc%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/13ce15ce-7bad-468a-84c5-10be4044935c%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/13ce15ce-7bad-468a-84c5-10be4044935c%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LB7RwWN2MKewGtHwQKXYeSwo%3DsMqf%2BmsKKFM0m%2Bh5zDQ%40mail.gmail.com.
