This was the final version I'd landed on for UTF-8: https://github.com/elixir-unicode/unicode/blob/main/lib/unicode/validation/utf8.ex
Along with the following modules for testing: https://github.com/elixir-unicode/unicode/blob/main/test/support/unicode_validation_helpers.ex https://github.com/elixir-unicode/unicode/blob/main/test/unicode_validation_test.exs I think it's ideal functionality to have in the String module, and the implementation's "reasonable enough" until a native solution's available in OTP. Testing is my only uncertainty - How much is prudent and in what manner? On Tuesday, October 31, 2023 at 12:35:48 PM UTC-4 José Valim wrote: > Folks, I am following up on this, where did we land? > > The new implementation is roughly ~70LOC for UTF-8, so at first I don't > see an issue with adding it to Elixir. However, the Elixir version would be > UTF-8 only (part of the String module). > > Thoughts? > > On Saturday, October 7, 2023 at 10:51:34 PM UTC+2 Kip wrote: > >> Cameron, I think this would be a useful addition to the Unicode library >> <https://github.com/elixir-unicode/unicode> I maintain. If that works >> for you, please open an issue there and we can collaborate. I think it >> being part of the Erlang `:unicode` module makes good sense too as José >> says but that's a longer "sales" and implementation cycle. >> >> On Saturday, October 7, 2023 at 7:40:50 PM UTC+11 José Valim wrote: >> >>> Hi Cameron, >>> >>> If the goal is to include this handling for UTF-16 and UTF-32, I suggest >>> proposing this to Erlang/OTP as new functions in the "unicode" module. >>> Otherwise, Elixir only has facilities to deal with UTF-8. You could propose >>> such a feature in their issues tracker. >>> >>> Also note that "rolling your own" or "depending on packages" is usually >>> not enough reasons for adding features to Elixir. Otherwise, one could >>> easily argue Decimal and Jason would be more important additions to the >>> language. :) We do describe which features we would consider part of the >>> language here: https://elixir-lang.org/development.html >>> >>> Other than that, awesome job on the library and benchmarks. :) >>> >>> On Sat, Oct 7, 2023 at 1:03 AM Kip <kipc...@gmail.com> wrote: >>> >>>> Your implementation is definitely fast and memory efficient so I >>>> retract my implementation comments. Now that I've run the benchmarking >>>> script and tested out a few different approaches leveraging the std lib I >>>> understand better why you've taken the approach you have. Nice work. >>>> >>>> On Saturday, October 7, 2023 at 9:26:37 AM UTC+11 Kip wrote: >>>> >>>>> Cameron, I think this is a useful proposal. Elixir has means to check >>>>> validity (String.valid?/1) and a mechanism to split valid and invalid >>>>> code >>>>> points (String.chunk/2 with the :valid trait). But there isn't, to my >>>>> knowledge, a means to coerce validity. A couple of thoughts: >>>>> >>>>> 1. Since Elixir strings are, by definition, UTF8, I don't know that >>>>> special handling of UTF16 and UTF32 code points makes much sense - >>>>> although >>>>> I accept this may be more Unicode compliant. >>>>> 2. What would the function be called? Since we have String.valid?/1 >>>>> maybe String.validate/2 with an option `replace_invalid: utf8_string`. >>>>> The >>>>> default `:replace_invalid` could be U+FFFD or it could be `nil`. If >>>>> the default is `nil` then there could also be a `String.validate!/2` that >>>>> raises if there is no `:replace_invalid` option. >>>>> 3. I think the implementation could leverage the code of >>>>> `String.chunk/2` which uses `String.next_codepoint/1`. That would >>>>> simplify >>>>> implementation and be more consistent in code style. >>>>> >>>>> On Friday, October 6, 2023 at 12:24:28 PM UTC+11 cameron...@gmail.com >>>>> wrote: >>>>> >>>>>> As far as I can tell, neither Elixir nor Erlang have a built in >>>>>> function for replacing invalid sequences in Unicode. There's a suggested >>>>>> method on this page >>>>>> <https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf#page=153> >>>>>> >>>>>> of the Unicode standard for handling this. Several other languages ( >>>>>> Go <https://pkg.go.dev/bytes#ToValidUTF8>, Python >>>>>> <https://docs.python.org/3/library/stdtypes.html#bytes.decode>, C# >>>>>> <https://github.com/dotnet/docs/issues/13547>, etc) now follow this >>>>>> spec. >>>>>> >>>>>> Invalid Unicode's encountered frequently enough that I think it's >>>>>> worth incorporating a solution into Elixir itself. >>>>>> >>>>>> Present alternatives to handling invalid unicode (and json by >>>>>> extension <https://github.com/michalmuskala/jason/issues/174>) are: >>>>>> >>>>>> - Crashing (not ideal in many cases) >>>>>> - Roll your own (lot of overhead for accidental complexity) >>>>>> - Depend on a package (+1 package towards dependency hell) >>>>>> >>>>>> This is my college try >>>>>> <https://github.com/Moosieus/UniRecover/tree/main>, but I'm certain >>>>>> there's a performant and far cleaner solution to be had in pure Elixir. >>>>>> If >>>>>> not, perhaps this is a request for OTP. >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to elixir-lang-co...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elixir-lang-core/197620a2-6a96-41c6-a6e7-5da03e351080n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/elixir-lang-core/197620a2-6a96-41c6-a6e7-5da03e351080n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/26e4c761-caf0-46ea-ab08-a33407febdb8n%40googlegroups.com.
