Riddhish1 commented on code in PR #2116:
URL: https://github.com/apache/stormcrawler/pull/2116#discussion_r3957102046
##########
external/tika/src/main/java/org/apache/stormcrawler/tika/ParserBolt.java:
##########
@@ -162,11 +162,43 @@ public void execute(Tuple tuple) {
// check that the mimetype is in the whitelist
if (!mimeTypeWhiteList.isEmpty()) {
boolean mt_match = false;
- // see if a mimetype was guessed in JSOUPBolt
+ // parse.Content-Type is assumed byte-detected (JSoupParserBolt
uses Tika detection,
+ // not the raw server header). A custom upstream writing a
header-copied value bypasses
+ // this check — that is a caller responsibility.
String mimeType = metadata.getFirstValue("parse.Content-Type");
- // otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
- mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ // parse.Content-Type is absent: detect from content bytes so
that
+ // the whitelist is evaluated against the same type Tika will
use
+ // to select a parser, not the server-declared HTTP header
which is
+ // untrusted and may differ from what the bytes actually are.
+ String httpCTHint =
+ metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
+ org.apache.tika.metadata.Metadata detectionMd =
+ new org.apache.tika.metadata.Metadata();
+ if (StringUtils.isNotBlank(httpCTHint)) {
+ // pass the header as a hint only — detect() weighs it but
+ // content bytes take precedence
Review Comment:
Fixed updated the comment to clarify that the hint narrows the magic result,
specialising application/octet-stream when bytes are unrecognised
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
