I like your examples, Richard, of strings that should not be considered as
numbers. There are times you want to recognize something like "221B Baker
Street" and extract the 221 or perhaps the 221B as a part of a full address and
ignore the rest. But one goal of a function like as.numeric() is to handle
fully-formed numbers ready to be converted to another storage format.
People with special uses and needs generally need to roll their own. A Bank may
want to be able to handle this:
as.currency("five hundred twenty eight dollars and thirty two cents")
And someone else may want to handle Roman numerals, or calculate the value of
text in a form of numerology like:
gematria("חַי") generates 18.
But what you describe with μsoft is quite inconsiderate and I wonder if it
should be automatically handled by an enhanced version of the as.numeric
family. There is indeed a general problem with different character sets and how
to know what you have. But breaking lots of software this way is not ...
-----Original Message-----
From: Richard O'Keefe <rao...@gmail.com>
Sent: Monday, March 3, 2025 5:34 PM
To: avi.e.gr...@gmail.com
Cc: Ivan Krylov <ikry...@disroot.org>; Christofer Bogaso
<bogaso.christo...@gmail.com>; r-help <r-help@r-project.org>
Subject: Re: [R] Failed to convert data to numeric
The zero-width no-break space character is used as the Byte Order
Mark. That is, an official function for it at the beginning of a
character sequence
is to indicate whether you have 2-byte or 4-byte big-endian or
little-endian encoding. It was not intended for use in UTF-8, where
there is nothing for
it to tell you, but Microsoft jumped in with all six feet and said
"hey, we'll use this to indicate that it's Unicode in UTF-8 and not
one of the hundreds
of other 8-bit coded character sets." I've lost count of the number
of programs that have choked because they were given a BOM where they
didn't expect one.
So there is no great mystery about why there is a BOM at the beginning
of this particular string.
The real mystery is why it was there and NOT at the beginning of all the others.
I suggest that it is a good idea to remove the BOM character from the
beginning of microsofted strings,
but a bad idea to remove any other character. If you are given bad
data like "Bond-007" when you
expect a number, you want to know about it, and not mistake it for
-007. Still less do you want a
phone number like "+61 3 555 1234 x77" to be mistaken for a plain
number "613555123477".
On Tue, 4 Mar 2025 at 06:24, <avi.e.gr...@gmail.com> wrote:
>
> The second solution Ivan offers looks good, and a bit more general than his
> first that simply removes one non-visible character.
>
> It begs the question of why the data has that anomaly at all. Did the data
> come from a text-processing environment where it was going to wrap there and
> was protected?
>
> As Ivan points out, there is a question of what format you expect numbers in
> and what "as.numeric" should do when it does not see an integer or floating
> point number.
>
> If you test it, you can see that as.numeric ignores leading and/or trailing
> blanks and tabs and even newlines sometimes and some other irrelevant ASCII
> characters. In that spirit, the UNICODE character being mentioned should be
> one that any UNICODE-aware version of as.numeric should ignore.
>
> But UNICODE supports a much wider vision of numeric so that there are
> numeric-equivalent symbols in other languages and groupings and even
> something like the symbols for numerals in light or dark circles count as
> numbers. Those can likely safely be excluded in this context but perhaps not
> in a more general function.
>
> But I note as.numeric seems to handle scientific notation as in:
>
> as.numeric("1.23e8")
> [1] 1.23e+08
>
> So a single instance of the letters "e" and "E" must be supported if your
> numbers in string form may contain them. Further, the E cannot be the first
> or last letter. It cannot have adjacent whitespace. Still, if you are OK with
> getting an NA in such situations, it should be OK.
>
> It gets worse. Hexadecimal is supported:
>
> > as.numeric("0X12")
> [1] 18
>
> You now need to support the letters x and X. But only if preceded by a zero!
>
> It gets still worse as any characters from [0-9A-F] are supported:
>
> > as.numeric("0xAE")
> [1] 174
>
> There may be other scenarios it handles. The filter applied might remove
> valid numbers so you may want to carefully document it if your program only
> handles a restricted set.
>
> A possible idea might be to make two passes and only evaluate any resulting
> NA from as.numeric() by doing a substitution like Ivan suggests to try to fix
> any broken ones. But note it may fix too much as "1.2 e 5" might become
> "1.2e5" as spaces are removed.
>
> -----Original Message-----
> From: R-help <r-help-boun...@r-project.org> On Behalf Of Ivan Krylov via
> R-help
> Sent: Monday, March 3, 2025 3:09 AM
> To: Christofer Bogaso <bogaso.christo...@gmail.com>
> Cc: r-help <r-help@r-project.org>
> Subject: Re: [R] Failed to convert data to numeric
>
> В Mon, 3 Mar 2025 13:21:31 +0530
> Christofer Bogaso <bogaso.christo...@gmail.com> пишет:
>
> > Is there any way to remove all possible "Unicode character" that may
> > be present in the array at once?
>
> Define a range of characters you consider acceptable, and you'll be
> able to use regular expressions to remove everything else. For example,
> the following expression should remove everything except ASCII digits,
> dots, and hyphen-minus:
>
> gsub('[^0-9.-]+', '', dat2)
>
> There is a brief introduction to regular expressions in ?regex and
> various online resources such as <https://regex101.com/>.
>
> --
> Best regards,
> Ivan
>
> ______________________________________________
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
