exceptionfactory commented on code in PR #11850:
URL: https://github.com/apache/nifi/pull/11850#discussion_r4417320812
##########
nifi-extension-bundles/nifi-jslt-bundle/nifi-jslt-processors/src/test/java/org/apache/nifi/processors/jslt/TestJSLTTransformJSON.java:
##########
@@ -297,6 +300,45 @@ private static Stream<Arguments> multilineArgs() {
);
}
+ @Test
+ public void testJsonLinesSkipsNullTransformationResults() {
+ final String input = """
+ {"id":1,"event":"deposit","amount":10}
+ {"id":2,"event":"payment_updated","amount":10}
+ {"id":3,"event":"deposit","amount":25}
+ {"id":4,"event":"payment_updated","amount":25}
+ {"id":5,"event":"deposit","amount":40}
+ """;
+
+ runner.setProperty(JSLTTransformJSON.INPUT_FORMAT,
JSLTTransformJSON.InputFormat.JSON_LINES);
+ runner.setProperty(JSLTTransformJSON.JSLT_TRANSFORM,
+ """
+ if (.event == "deposit") {"id": .id, "amount":
.amount}""");
+
+ runner.enqueue(input);
+ runner.run();
+
+ runner.assertTransferCount(JSLTTransformJSON.REL_SUCCESS, 1);
+ runner.assertTransferCount(JSLTTransformJSON.REL_FAILURE, 0);
+
+ final MockFlowFile flowFile =
runner.getFlowFilesForRelationship(JSLTTransformJSON.REL_SUCCESS).getFirst();
+ final String expectedJson = """
+ {"id":1,"amount":10}
+ {"id":3,"amount":25}
+ {"id":5,"amount":40}
+ """.trim();
+
+ assertEquals(expectedJson, flowFile.getContent());
+ final MockComponentLog componentLog = runner.getLogger();
+ final long debugMessages = componentLog.getDebugMessages().stream()
+ .filter(logMessage -> logMessage.getMsg().contains("no data"))
+ .count();
+ assertEquals(2, debugMessages);
+ final List<LogMessage> infoMessages = componentLog.getInfoMessages();
+ assertEquals(1, infoMessages.size());
+ assertTrue(infoMessages.getFirst().getMsg().contains("out of"));
Review Comment:
Asserting log messages is generally not very helpful, so I recommend
removing these checks
##########
nifi-extension-bundles/nifi-jslt-bundle/nifi-jslt-processors/src/main/java/org/apache/nifi/processors/jslt/JSLTTransformJSON.java:
##########
@@ -476,14 +476,26 @@ private FlowFile transformJsonLines(final Expression
jsltExpression, final FlowF
try (final BufferedReader reader = new BufferedReader(new
InputStreamReader(inputStream, StandardCharsets.UTF_8));
final JsonGenerator jsonGenerator =
writer.createGenerator(outputStream)) {
String line;
+ int lines = 0;
+ int transforms = 0;
while ((line = reader.readLine()) != null) {
if (line.isBlank()) {
continue;
}
+ lines++;
final JsonNode jsonNode =
JSON_OBJECT_MAPPER.readTree(line);
final JsonNode transformedJson =
jsltExpression.apply(jsonNode);
- jsonGenerator.writeObject(transformedJson);
+ if (transformedJson == null || transformedJson.isNull()) {
+ getLogger().debug("JSLT Transform resulted in no data
on line {}", lines + 1);
Review Comment:
Actually, the current implementation only increments `lines` when the `line`
is not blank. Although slightly duplicative, I recommend adding a separate
`currentLineNumber` which starts with `1` and always increments, to avoid
confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
