Hi Zach! The only way you can minimize the transitive dependency problem is by removing the compile-time deps. I struggled with this for years with Surface, and as you did, I tried the approach of removing the runtime deps to avoid them from turning into transitive compile-time deps. Eventually, I realized that avoiding compile-time deps would be the only way to have a permanent solution and started removing any implementation that required information at compile-time. As soon as LV v0.8 is out, we'll wrap up that work, so we don't expect further issues after that. In the meantime, to solve the problem, I added a fix that drastically cut down those dependencies, which was to convert the `compile` deps into `export` deps, by using `import` instead of `require`. For this to work properly, I have to generate and automatically rename a signature function to force the recompilation of direct callers whenever the component's metadata (props, slots, etc) changes. Otherwise, it would only recompile them if you added or removed another function.
It seems to me that Elixir could provide a way to hook into the compiler allowing DSL developers to inform a criteria that could be used to trigger callers' recompilation. Similar to `__mix_recompile__?`. Maybe something like `__export_changed__?` or anything else. With this approach, DSL authors could replace the `compile` deps with `export` deps as long as the changes in that dependency don't need to be propagated to other modules other than the callers. This is the case for Surface and many other libs, like Commanded, for instance. I'm not sure the proposed solution would solve your issue with Ash but I thought it was worth bringing it here since it may solve issues on other libs. José, is something like that feasible? On Tuesday, September 13, 2022 at 8:37:24 PM UTC-3 [email protected] wrote: > Well, its runtime dependencies that create transitive compile time > dependencies, isn't it? So if I have a function like > > defmodule Foo do > # callers are aware that they can't use the module at compile time > def returns_a_module() do > SomeSpecificModule > end > > def something_used_at_compile_time() do > 10 > end > end > > # and then some other module > > defmodule Bar do > @something Foo.something_used_at_compile_time() > end > > This induces a transitive compile time dependency from Bar to > SomeSpecificModule. > > In the case of Ash DSLs, for example, this happens because we store the > resulting configuration in a module attribute. Users are aware that you > can't call these pluggable modules at compile time. Only the framework code > calls those modules. The above example is contrived, of course, I'm not > suggesting that we need a feature to make *that* work, just trying to draw > some kind of parallel. > > I don't see a reasonable way to handle this without essentially removing > the ability to use modules in the way that we are. It would be a pretty > unreasonable amount of work/change for users to change the way that we plug > behavior in Ash. > > To me, another kind of module dependency, like `runtime-only`, would solve > for this, and it would have to be explicitly requested, i.e > `expand_literal(.., runtime_only: true)`. Then when determining transitive > compile time dependencies, the compiler would not use those modules. > > > On Tue, Sep 13, 2022 at 6:51 PM, José Valim <[email protected]> wrote: > >> As I mentioned, the issue on transitive compile-time dependencies are the >> compile-time deps, not the runtime ones. So I would focus on how to >> eliminate those. Otherwise I am not sure we will agree on the problem >> statement. :) >> >> On Wed, Sep 14, 2022 at 12:49 AM Zach Daniel <[email protected]> >> wrote: >> > Although the solution I originally proposed may not be correct (totally >>> fine with me [image: 😃]), I think the problem statement is still >>> valid. Can we agree that there are some cases where you may want to >>> reference a module without creating transitive compile time dependencies, >>> i.e in the case of a DSL like what Ash provides? >>> >>> >>> On Sun, Sep 11, 2022 at 3:08 PM, Zach Daniel <[email protected] >>> > wrote: >>> >> Hm...yeah, that makes sense. Are there other things a runtime dependency >>>> is meant to do aside from ensure that transitive compile time dependencies >>>> are properly tracked? If so, is there a way to solve for the transitive >>>> dependencies without removing the runtime dependency? Because ultimately >>>> the idea here is that this module is essentially just a big >>>> validated/extensible configuration. The resource itself doesn't do >>>> anything. So requiring things that refer to a resource to recompile when >>>> any of the modules it refers to in a `change` like that is unnecessary. >>>> However we can express that reality is totally fine with me. >>>> >>>> >>>> On Sunday, September 11, 2022 at 3:05:18 PM UTC-4 José Valim wrote: >>>> >>>>> The issue is in the transitive compile time dependencies, not the >>>>> runtime dependencies. >>>>> >>>>> I don't think we should be encouraging removing the runtime >>>>> dependencies when they are explicitly listed in the code as above. When >>>>> done via configuration, at least you are not literally listing it. >>>>> >>>>> On Sun, Sep 11, 2022 at 8:59 PM Zach Daniel <[email protected]> >>>>> wrote: >>>>> >>>>>> So all we we do is hold onto the module, and then at runtime we go >>>>>> through the list of modules that we should call and call a specific >>>>>> function on them. Requiring a runtime dependency for that is causing >>>>>> really >>>>>> slow compile times because of transitive dependencies. Maybe there is >>>>>> some >>>>>> consequence I don't see, but I removed the runtime dependencies by >>>>>> disabling the lexical tracker when expanding the alias, and its been >>>>>> that >>>>>> way for months w/o anyone reporting any issues with that implementation. >>>>>> Aside from having to use `warn: false` if they use aliases. >>>>>> >>>>>> To me, its the same as if they gave us, instead of a module, an >>>>>> `atom` that referred to application configuration, i.e the adapter >>>>>> pattern. >>>>>> That would work without a runtime dependency, so why couldn't this? >>>>>> >>>>>> >>>>>> On Sun, Sep 11, 2022 at 2:56 PM, José Valim <[email protected]> >>>>>> wrote: >>>>>> >>>>> Sorry, correction: If, at any moment, you call any function from that >>>>>>> module at runtime, you must not remove the *runtime* time >>>>>>> dependency. >>>>>>> >>>>>>> On Sun, Sep 11, 2022 at 8:55 PM José Valim <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>> Why do you want to remove the runtime dependency when, per your >>>>>>>> description: >>>>>>>> >>>>>>>> > In Ash Framework, we have declarative ways to construct *runtime* >>>>>>>> behavior using behaviors. >>>>>>>> >>>>>>>> Emphasis mine. If, at any moment, you call any function from that >>>>>>>> module at runtime, you must not remove the compile time dependency. >>>>>>>> >>>>>>>> On Sun, Sep 11, 2022 at 8:52 PM Zach Daniel <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>> `expand_literal` removes the compile time dependency, but leaves a >>>>>>>>> runtime dependency when used inside of a module. >>>>>>>>> >>>>>>>>> What I'm trying to do is remove both the compile time dependency >>>>>>>>> *and* the runtime dependency, without requiring the use of `warn: >>>>>>>>> false` on >>>>>>>>> aliases. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sunday, September 11, 2022 at 2:31:42 PM UTC-4 José Valim wrote: >>>>>>>>> >>>>>>>>>> Sorry, I don't understand the proposal. You mentioned >>>>>>>>>> expand_literal, which already removes the compile-time dependency >>>>>>>>>> but keeps >>>>>>>>>> the remaining functionality such as warnings. Can you please expand? >>>>>>>>>> >>>>>>>>>> On Sun, Sep 11, 2022 at 7:57 PM Zach Daniel < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> For clarity, the dependency I'm talking about there is the >>>>>>>>>>> dependency from `MyApp.User` to `MyApp.User.Changes.HashPassword`. >>>>>>>>>>> On Sunday, September 11, 2022 at 1:55:51 PM UTC-4 Zach Daniel >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> In Ash Framework, we have declarative ways to construct runtime >>>>>>>>>>>> behavior using behaviors. So an Ash resource might look like this: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ```elixir >>>>>>>>>>>> defmodule MyApp.User do >>>>>>>>>>>> use Ash.Resource >>>>>>>>>>>> >>>>>>>>>>>> alias MyApp.User.Changes.HashPassword >>>>>>>>>>>> >>>>>>>>>>>> attributes do >>>>>>>>>>>> uuid_primary_key :id >>>>>>>>>>>> .... >>>>>>>>>>>> end >>>>>>>>>>>> >>>>>>>>>>>> actions do >>>>>>>>>>>> create :register do >>>>>>>>>>>> change HashPassword >>>>>>>>>>>> end >>>>>>>>>>>> end >>>>>>>>>>>> end >>>>>>>>>>>> ``` >>>>>>>>>>>> >>>>>>>>>>>> However, by default, this would incur a compile time >>>>>>>>>>>> dependency. This compile time dependency is unnecessary, as we >>>>>>>>>>>> won't call >>>>>>>>>>>> any functions on this module or use it in any way until runtime. >>>>>>>>>>>> >>>>>>>>>>>> That optimization is well and good, but due to transitive >>>>>>>>>>>> compile time dependencies, we see some interesting behavior. >>>>>>>>>>>> Something >>>>>>>>>>>> you'd often see in a change module is things like pattern matching >>>>>>>>>>>> on other >>>>>>>>>>>> resources, or the resource in question in function heads. >>>>>>>>>>>> Resources are >>>>>>>>>>>> meant to be introspectable at compile time, and so this runtime >>>>>>>>>>>> dependency >>>>>>>>>>>> on a change, with a compile time dependency on a resource, incurs >>>>>>>>>>>> a >>>>>>>>>>>> transitive compile time dependency. This problem multiplies over >>>>>>>>>>>> time, and >>>>>>>>>>>> causes users to have to do things solely to optimize compile >>>>>>>>>>>> times, like >>>>>>>>>>>> only use map pattern matches instead of struct pattern matches. >>>>>>>>>>>> >>>>>>>>>>>> So what we do is we actually disable the lexical tracker when >>>>>>>>>>>> accepting certain parts of the DSL. This prevents *any* >>>>>>>>>>>> dependency. >>>>>>>>>>>> Naturally, at compile time you are no longer safe to call a >>>>>>>>>>>> resource's >>>>>>>>>>>> change module as changes in that module won't cause recompiles, >>>>>>>>>>>> but that >>>>>>>>>>>> was never a thing you should have done in the first place so I'm >>>>>>>>>>>> not >>>>>>>>>>>> worried about that. >>>>>>>>>>>> >>>>>>>>>>>> This leads us to the primary issue: disabling the lexical >>>>>>>>>>>> tracker when expanding aliases also causes warnings about unused >>>>>>>>>>>> aliases, >>>>>>>>>>>> even though they *are* used. I believe I've brought this issue up >>>>>>>>>>>> before >>>>>>>>>>>> and we were hoping that the feature introduced in 1.14 for >>>>>>>>>>>> `defimpl` would >>>>>>>>>>>> help, but that only helps prevent compile time issues, which is >>>>>>>>>>>> something I >>>>>>>>>>>> had already solved for in the same way it was solved for 1.14. >>>>>>>>>>>> I've laid it >>>>>>>>>>>> all out to help clarify exactly *why* I need it so perhaps someone >>>>>>>>>>>> can >>>>>>>>>>>> point me in a better direction. >>>>>>>>>>>> >>>>>>>>>>>> The simplest thing that could help: >>>>>>>>>>>> >>>>>>>>>>>> A way to tell the lexical tracker that an alias has just been >>>>>>>>>>>> referenced without inducing any kind of compile or runtime >>>>>>>>>>>> dependency. The >>>>>>>>>>>> idea is to just prevent it from warning about the alias. >>>>>>>>>>>> >>>>>>>>>>>> I'm open to other solutions as well. >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/54627973-7b74-47d7-9e35-4270621e6c91n%40googlegroups.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/54627973-7b74-47d7-9e35-4270621e6c91n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elixir-lang-core" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/22887a7e-a1b6-4ccc-98bf-69a5ad3551a5n%40googlegroups.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/22887a7e-a1b6-4ccc-98bf-69a5ad3551a5n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elixir-lang-core" group. >>>>>>> >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BVkL%2B1V7LTDVyzhCqcNWrmHFPoWx2Fp916Ur%2ByL%2BiVBA%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BVkL%2B1V7LTDVyzhCqcNWrmHFPoWx2Fp916Ur%2ByL%2BiVBA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/l7xp6eut.444ff913-f528-4b88-805a-cac120cdb4d6%40we.are.superhuman.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/l7xp6eut.444ff913-f528-4b88-805a-cac120cdb4d6%40we.are.superhuman.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elixir-lang-core/072bb99d-09a0-4567-a934-f8893015dd91n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/elixir-lang-core/072bb99d-09a0-4567-a934-f8893015dd91n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> >> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >> >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-core/l80s9s37.13dfe243-83b3-4133-8aad-7a86910f56fa%40we.are.superhuman.com >>> >>> <https://groups.google.com/d/msgid/elixir-lang-core/l80s9s37.13dfe243-83b3-4133-8aad-7a86910f56fa%40we.are.superhuman.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> > You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J8qxM%3DKxMUUW6X%2Bt%2BrD9rKn0yDHbDY1fDgopxRvtNn5A%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J8qxM%3DKxMUUW6X%2Bt%2BrD9rKn0yDHbDY1fDgopxRvtNn5A%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/325ea272-674a-4a5a-9228-6af11579bd05n%40googlegroups.com.
