[
https://issues.apache.org/jira/browse/IO-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795002#comment-17795002
]
Marcono1234 commented on IO-781:
--------------------------------
[[email protected]], no this is not limited to surrogate chars. Here is
an example without surrogate chars based on the {{CodingErrorAction.IGNORE}}
variant shown above:
{code}
Charset charset = StandardCharsets.US_ASCII;
CharSequenceInputStream in = CharSequenceInputStream.builder()
.setCharsetEncoder(charset.newEncoder().onUnmappableCharacter(CodingErrorAction.IGNORE))
.setCharSequence("\u0080")
.get();
System.out.println("Available: " + in.available());
// Note: readAllBytes() is a method added in Java 9
System.out.println("Actually read: " + in.readAllBytes().length);
{code}
As mentioned above the underlying issue is that
{{CharSequenceInputStream.available()}} makes potentially wrong guesses about
how many bytes will be available by considering {{this.cBuf.remaining()}},
instead of just returning the number of bytes which are guaranteed to be
available ({{this.bBuf.remaining()}} I assume).
{quote}
So it shouldn't be hard to scan and identify [surrogate chars]
{quote}
This is in my opinion something which should not be done. Because this issue is
not limited to surrogate chars in the first place, and also because the
underlying {{CharsetEncoder}} handles the encoding already;
{{CharSequenceInputStream}} should just not make any assumptions which are not
guaranteed to be true.
> CharSequenceInputStream.available() returns too large numbers in some cases
> ---------------------------------------------------------------------------
>
> Key: IO-781
> URL: https://issues.apache.org/jira/browse/IO-781
> Project: Commons IO
> Issue Type: Bug
> Components: Streams/Writers
> Affects Versions: 2.11.0
> Reporter: Marcono1234
> Priority: Major
>
> h3. Description
> The {{available()}} method of
> {{org.apache.commons.io.input.CharSequenceInputStream}} erroneously returns
> values larger than the actual number of available bytes in some cases.
> The underlying issue is that {{CharSequenceInputStream}} makes incorrect
> assumptions about the relation between chars and bytes. The
> {{CodingErrorAction.REPLACE}} can convert 2 chars (1 supplementary code
> point) to one byte (the replacement char {{?}}). Additionally in case
> {{CharSequenceInputStream}} is ever extended to support specifying a
> {{CharsetEncoder}}, the {{CodingErrorAction.IGNORE}} would probably cause
> similar issues. There might also be some uncommon charsets which can encode 2
> chars to 1 byte; though I am not aware of such charset yet.
> This was originally mentioned in pull request
> [#293|https://github.com/apache/commons-io/pull/293]. That PR also proposed
> to replace the underlying {{CharSequenceInputStream}} implementation with
> {{ReaderInputStream}} because in general using {{CharsetEncoder}} is
> error-prone so it might be good to avoid having two classes implementing
> logic on top of it. (Potentially {{CharSequenceInputStream}} is missing a
> call to {{CharsetEncoder.flush}}, see also IO-714)
> h3. Example
> In the example below {{available()}} erroneously returns 2 even though only 1
> byte can be read.
> {code}
> Charset charset = Charset.forName("Big5");
> CharSequenceInputStream in = new CharSequenceInputStream("\uD800\uDC00",
> charset);
> // BUG: available() returns 2 but only 1 byte is read afterwards
> System.out.println("Available: " + in.available());
> // Note: readAllBytes() is a method added in Java 9
> System.out.println("Actually read: " + in.readAllBytes().length);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
