After a brief discussion with the Elixir team, we are inclined to go more with option 2. However, in order to make it efficient, we want to use persistent_term, which is only available on 21.3 and Elixir currently supports 21.0+.
We will drop Erlang/OTP 21 support once Erlang/OTP 24 is out but it is unclear if that will be before Elixir v1.12. Alternatively we can require Erlang/OTP 21.3. In any case, I have added an item to track this here: https://github.com/elixir-lang/elixir/issues/8414 Thank you! On Fri, Nov 6, 2020 at 10:25 AM Allen Wyma <[email protected]> wrote: > Looks like they just use the derive functionality from inspect: "@derive { > Inspect, except: @ecto_redact_fields}" so you can also look into using > that. If it's for struct outside of your control, that may be more difficult > > On Fri, Nov 6, 2020 at 5:21 PM 'Jayson Vantuyl' via elixir-lang-core < > [email protected]> wrote: > >> That would help with Ecto. Unfortunately we still see a bunch of stuff >> from the likes of Elixir GRPC and various structs from libraries that don't >> have that kind of functionality. We were kind of hoping for a top-down, >> whitelist approach. We figure that it's easier to plug the leak from the >> top (*inspect/2* itself) than it is to try to deal with all of the leaks >> at the bottom (everything that tries to use it). >> >> On Friday, November 6, 2020 at 1:15:37 AM UTC-8 [email protected] wrote: >> >>> I believe that the new changeset "redact" attribute should help with >>> this? >>> >>> On Fri, Nov 6, 2020 at 4:56 PM 'Jayson Vantuyl' via elixir-lang-core < >>> [email protected]> wrote: >>> >>>> Hey everybody, >>>> >>>> *Use Case* >>>> >>>> I have what I think is actually a pretty common problem. All over my >>>> code base, there are uses of *inspect/2*. This is a wonderful thing >>>> that helps with debugging. It is less wonderful when it spews PII all over >>>> your logs / error pages and you find yourself having sent somebody's social >>>> security number to DataDog or disclosed their home address on a 500 page. >>>> Then your lawyer starts doing that thing where their eye twitches, you need >>>> to notify four different regulators on three continents, alert all of your >>>> customers with scary messages, are made to attend 3-hours of mandatory >>>> re-education wherein you learn to recite the GDPR from memory, and >>>> eventually sacrifice three interns to appease the compliance gods. >>>> >>>> *Temporary Hack* >>>> >>>> What myself and some co-conspirators have done to address this is >>>> overriding the *Inspect* protocol for the built-in types to redact >>>> things by default and then have a whitelist for the bits we want to show. >>>> Given that our top-level logging metadata is a map, we can pretty much just >>>> whitelist keys there and call it a day. This works fabulously, but it >>>> violates Erlang's expectations significantly. While Erlang is probably >>>> used to that, it gets quite cross and refuses to generate a release because >>>> it doesn't have a good way to know which .beam file to put in the release. >>>> >>>> As you might imagine, I'm pretty bummed that I can't use releases and >>>> have to ignore tons of very alarming sounding warnings about redefining >>>> modules. >>>> >>>> *Options* >>>> >>>> Could we consider some solutions to make redaction require less >>>> unforgivable sins against code loading? To start, three directions have >>>> been proposed by the various people I've talked to: >>>> >>>> >>>> 1. Instead of implementing *Inspect* for the built-in types, do >>>> that inspection in a handler for *Any*; thereby allowing overriding >>>> of the built-in types easily. >>>> 2. Wedge something into a common entry point (maybe in >>>> *Inspect.Algebra*?) allows us to specify a global redaction >>>> function. Perhaps configure this with a global config value? >>>> 3. Implement some sort of overriding layer for just the *Inspect* >>>> protocol. >>>> >>>> >>>> In terms of pros and cons, for #1... >>>> >>>> - *Pro:* Works well for built-ins. >>>> - *Pro:* Implementing this is very straightforward. >>>> - *Pro:* This probably doesn't break any existing code, very small >>>> blast radius. >>>> - *Con:* Doesn't work at all as soon as anything defines its own >>>> inspection protocol. >>>> - *Con:* Isn't amenable to configuration at runtime (maybe this is >>>> not an issue?). >>>> >>>> As for #2... >>>> >>>> - *Pro:* *Can* be configured at runtime. >>>> - *Pro:* I have no idea how to implement this and *Inspect.Algebra* >>>> scares me. >>>> - *Pro:* This probably doesn't break any existing code, very small >>>> blast radius. >>>> - *Con:* Given that the Inspect protocol is pretty much "turn X >>>> into string", I'm not sure how much redaction we could really do if we >>>> allow the existing protocol to run. >>>> >>>> As for #3... >>>> >>>> - *Pro:* This provides a clear way to just replace the protocol for >>>> a given type. >>>> - *Pro:* Implementing this is very straightforward. >>>> - *Pro: *This probably doesn't break any existing code, very small >>>> blast radius. >>>> - *Con:* It's all fun and games until a library does it, then you >>>> need Override2 or 3 or 4... >>>> - *Con:* Probably gets redundant if there ever is a blessed way to >>>> override protocols. >>>> - *Con:* I already pitched this to a few Elixir celebrities and >>>> they thought it was a bit too hacky. >>>> >>>> *In Closing* >>>> >>>> So, yeah, in the long term, maybe we'll have a blessed way to override >>>> protocols; but, short of that, there's got to be some stopgap, right? What >>>> do people think of adopting something like one of these approaches so that >>>> my PII problems evaporate and I can finally build some sweet, sweet >>>> releases? I'll even implement it myself! I promise! >>>> >>>> Thanks, >>>> >>>> - J >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elixir-lang-core/d5d9a932-930c-46f7-a3fb-2f1a9ae06112n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elixir-lang-core/d5d9a932-930c-46f7-a3fb-2f1a9ae06112n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/38307a23-2568-4c48-ae8c-76cb70e7541bn%40googlegroups.com >> <https://groups.google.com/d/msgid/elixir-lang-core/38307a23-2568-4c48-ae8c-76cb70e7541bn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/CAJuWSyL8xhXbZf0ca_nZ4-dJU_Xej3UJ_8-dQFJp0%2B9XX1znpw%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAJuWSyL8xhXbZf0ca_nZ4-dJU_Xej3UJ_8-dQFJp0%2B9XX1znpw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L_CuMDq2JVLz2tYog33eTBLR_YF36B-c_DY5VUaVQNsQ%40mail.gmail.com.
