jnioche commented on code in PR #2111:
URL: https://github.com/apache/stormcrawler/pull/2111#discussion_r3905010645
##########
core/src/main/java/org/apache/stormcrawler/parse/filter/CommaSeparatedToMultivaluedMetadata.java:
##########
@@ -33,8 +39,20 @@
*/
public class CommaSeparatedToMultivaluedMetadata extends ParseFilter {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(CommaSeparatedToMultivaluedMetadata.class);
+
+ /**
+ * Default upper bound on the number of tokens a single value is split
into. It is above what a
+ * page fetched with the archetype's default 65536 byte http.content.limit
can produce, yet
+ * bounds the metadata a page can generate when the content limit is
raised.
+ */
+ private static final int MAX_TOKENS_DEFAULT = 65536;
Review Comment:
this is ridiculously high. I am not against setting a max number of tokens
if we are paranoid about websites generating a ludicrous amount of data to trip
the crawler but this value is too large to be really useful and also introduces
confusion as it refers to a limit which is expressed by bytes, not token count.
Please add this new behaviour to the documentation + have a default of 64 or
128 but not more.
Also would be good to remove the confusing reference to http.content.limit.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
