Riddhish1 opened a new pull request, #2116:
URL: https://github.com/apache/stormcrawler/pull/2116
# Fixes #2104
## The problem
`ParserBolt.execute()` checks `parser.mimetype.whitelist` before parsing.
When `parse.Content-Type`
is present in the metadata (written by `JSoupParserBolt` when
`detect.mimetype` is true), it is used
for the whitelist check and also drives Tika's `AutoDetectParser` — the two
are in agreement.
When `parse.Content-Type` is **absent** — because `detect.mimetype` is
false, or because the topology
feeds `ParserBolt` directly without `JSoupParserBolt` upstream — the code
fell back to the
`Content-Type` response header supplied by the fetched server:
```java
// otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
}
```
The whitelist was then evaluated against this server-declared value, while
Tika's
`AutoDetectParser` dispatched on the **raw content bytes**. A server can
claim any MIME type in its
response header, so the two sources can disagree. In the worst case a server
reports a whitelisted
type (e.g.
`application/vnd.openxmlformats-officedocument.wordprocessingml.document`) while
serving an entirely different payload (e.g. HTML). The whitelist gate
opened, and Tika parsed
whatever the bytes actually were.
The practical impact is limited in the common archetype setup because
`JSoupParserBolt` runs ahead
of the Tika bolt with detection enabled, so `parse.Content-Type` is almost
always present.
The gap opens in two real scenarios:
- `detect.mimetype: false` in the crawler configuration.
- A custom topology that wires `FetcherBolt` directly to `ParserBolt`
without a JSoup stage.
In both cases the whitelist was not doing the job its name and the archetype
documentation imply:
controlling **which document types this bolt parses**.
## What this PR changes
### `ParserBolt.execute()` — detect from bytes when `parse.Content-Type` is
absent
When the metadata key `parse.Content-Type` is missing, the bolt now calls
`tika.detect()` on the
content bytes before evaluating the whitelist, rather than trusting the
server header:
```java
if (mimeType == null) {
String httpCTHint =
metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
org.apache.tika.metadata.Metadata detectionMd = new
org.apache.tika.metadata.Metadata();
if (StringUtils.isNotBlank(httpCTHint)) {
// pass the header as a hint only — detect() weighs it but bytes
take precedence
detectionMd.set(org.apache.tika.metadata.Metadata.CONTENT_TYPE,
httpCTHint);
}
mimeType = tika.detect(new ByteArrayInputStream(content), detectionMd);
if (mimeType != null) {
// write back so downstream code sees the same value (avoids a
second detection pass)
metadata.setValue("parse.Content-Type", mimeType);
}
}
```
The HTTP response header is still passed to Tika as a **hint**, which is the
correct and safe use
of a server-declared MIME type. Content bytes take precedence when the two
disagree. The result is
written back into `parse.Content-Type` so that the whitelist check and
Tika's subsequent
`AutoDetectParser` invocation are bound to the same type, and downstream
bolts and metrics see a
consistent value without incurring a second detection pass.
**Behaviour in the common case is unchanged.** When `parse.Content-Type` is
already present (the
normal path with `JSoupParserBolt` upstream), the new block is not entered.
### Possible behaviour change worth noting in release notes
Documents whose server-declared `Content-Type` matched the whitelist but
whose bytes are detected
as a different type will now be **rejected** where they were previously
parsed. This is the
correct outcome — the whitelist was not enforcing what operators expected —
but operators who
relied on the previous behaviour (intentionally or not) should be aware.
Restricting the parser set in `tika-config.xml` is worth documenting
alongside this change as
a complementary defence in depth: it bounds which parsers can be selected at
all, regardless of
what the whitelist or the detection step resolves.
## Tests
### `ParserBoltWhitelistDetectionTest` (new, `external/tika`)
**`whitelistAppliesToTheDetectedType`** — reproduces the original bug:
- Whitelist: `application/.+word.*` (the pattern shipped by the archetypes).
- Server `Content-Type` header:
`application/vnd.openxmlformats-officedocument.wordprocessingml.document`.
- Body bytes: plain HTML — `<html><body><p>not a word
document</p></body></html>`.
- No `parse.Content-Type` in metadata (simulates a topology without
JSoupParserBolt upstream).
Before this fix the bolt parsed the HTML and emitted a document. After this
fix the bolt detects
`text/html` from the bytes, the whitelist does not match, and the tuple is
emitted on the status
stream with `Status.ERROR`.
**`whitelistUsesPreexistingParsedContentType`** — sanity check for the
unchanged common path:
- `parse.Content-Type` is set to `text/html; charset=UTF-8` (as
JSoupParserBolt would write it).
- Whitelist: `text/html.*`.
- Asserts that the document is accepted and no ERROR status is emitted.
## Verification
```
# install core (editorconfig check is a LF/CRLF issue on Windows; bypass
validate phase)
mvn -pl core compiler:compile compiler:testCompile jar:jar install:install
-DskipTests
# run the new tests
mvn -pl external/tika test -Dtest=ParserBoltWhitelistDetectionTest
```
```
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
```
Detected type printed from `parse.Content-Type` after the parse in the
reproduction test:
```
detected type: text/html
emitted documents: 0
```
The bolt now rejects the mismatched document (0 documents emitted) and
correctly identifies the
actual type from the bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
