andrewmlim commented on code in PR #6044: URL: https://github.com/apache/nifi/pull/6044#discussion_r878450602
########## nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/resources/docs/org.apache.nifi.processors.script.ExecuteScript/additionalDetails.html: ########## @@ -0,0 +1,698 @@ +<!DOCTYPE html> +<html lang="en" xmlns="http://www.w3.org/1999/html";> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + +<head> + <meta charset="utf-8"/> + <title>ExecuteScript</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/> + <style> +h2 {margin-top: 4em} +h3 {margin-top: 3em} +td {text-align: left} + </style> +</head> + +<body> + +<h1>ExecuteScript</h1> + +<h3>Description</h3> +<p> + The ExecuteScript Processor provides the ability to use a scripting language in order to leverage the NiFi API to perform tasks such as the following: +</p> +<ul> + <li>Read content and/or attributes from an incoming Flowfile</li> + <li>Create a new FlowFile (with or without a parent)</li> + <li>Write content and/or attributes to an outgoing FlowFile</li> + <li>Interact with the ProcessSession to transfer FlowFiles to relationships</li> + <li>Read/write to the State Manager to keep track of variables across executions of the processor</li> +</ul> + +<p> + <b>Notes:</b> +<ul> + <li>The engine listed as "python" in the list of available script engines is actually Jython, not Python. When using Jython, you cannot import pure (CPython) modules such as pandas</li> + <li>Lua does not allow for referencing static members of a class, so the REL_SUCCESS and REL_FAILURE relationships are made available via script bindings (aka variables), see the Variable Bindings section for more details</li> + <li>ExecuteScript uses the JSR-223 Script Engine API to evaluate scripts, so the use of idiomatic language structure is sometimes limited. For example, in the case of Groovy, there is a separate ExecuteGroovyScript processor that allows you to do many more idiomatic Groovy tasks. For example it's easier to interact with Controller Services via ExecuteGroovyScript vs. ExecuteScript (see the ExecuteGroovyScript documentation for more details)</li> + <li>JavaScript is provided by the Nashorn engine, which is not available in later Java versions, so it may not show up in the Script Engines list depending on the JRE used</li> +</ul> +</p> +<h3>Variable Bindings</h3> +<p> + The Processor expects a user defined script that is evaluated when the processor is triggered. The following variables are available to the scripts: +</p> +<table> + <tr> + <th>Variable Name</th> + <th>Description</th> + <th>Variable Class</th> + </tr> + <tr> + <td><b>session</b></td> + <td>This is a reference to the ProcessSession assigned to the processor. The session allows you to perform operations on FlowFiles such as create(), putAttribute(), and transfer(), as well as read() and write()</td> + <td><a href="https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/ProcessSession.html";>ProcessSession</a></td> + </tr> + <tr> + <td><b>context</b></td> + <td>This is a reference to the ProcessContext for the processor. It can be used to retrieve processor properties, relationships, Controller Services, and the StateManager.</td> + <td><a href="https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/ProcessContext.html";>ProcessContext</a></td> + </tr> + <tr> + <td><b>log</b></td> + <td>This is a reference to the ComponentLog for the processor. Use it to log messages to NiFi, such as log.info('Hello world!')</td> + <td><a href="https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/logging/ComponentLog.html";>ComponentLog</a></td> + </tr> + <tr> + <td><b>REL_SUCCESS</b></td> + <td>This is a reference to the "success" relationship defined for the processor. It could also be inherited by referencing the static member of the parent class (ExecuteScript), but some engines such as Lua do not allow for referencing static members, so this is a convenience variable. It also saves having to use the fully-qualified name for the relationship.</td> + <td><a href="https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/Relationship.html";>Relationship</a></td> + </tr> + <tr> + <td><b>REL_FAILURE</b></td> + <td>This is a reference to the "failure" relationship defined for the processor. As with REL_SUCCESS, it could also be inherited by referencing the static member of the parent class (ExecuteScript), but some engines such as Lua do not allow for referencing static members, so this is a convenience variable. It also saves having to use the fully-qualified name for the relationship.</td> + <td><a href="https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/processor/Relationship.html";>Relationship</a></td> + </tr> + <tr> + <td><i>Dynamic Properties</i></td> + <td>Any dynamic (user-defined) properties defined in ExecuteScript are passed to the script engine as variables set to the PropertyValue object corresponding to the dynamic property. This allows you to get the String value of the property, but also to evaluate the property with respect to NiFi Expression Language, cast the value as an appropriate data type (such as Boolean, e.g.), etc. Because the dynamic property name becomes the variable name for the script, you must be aware of the variable naming properties for the chosen script engine. For example, Groovy does not allow periods (.) in variable names, so an error will occur if "my.property" was a dynamic property name. + Interaction with these variables is done via the NiFi Java API, the 'Dynamic Properties' section below will discuss the relevant API calls as they are introduced. </td> + <td><a href="https://www.javadoc.io/doc/org.apache.nifi/nifi-api/latest/org/apache/nifi/components/PropertyValue.html";>PropertyValue</a></td> + </tr> +</table> + +<h2>Example Scripts</h2> + +<p><strong>Get an incoming FlowFile from the session</strong></p> +<p><strong>Use Case</strong>: You have incoming connection(s) to ExecuteScript and want to retrieve one FlowFile from the queue(s) for processing.</p> +<p><strong>Approach</strong>: Use the get() method from the session object. This method returns the FlowFile that is next highest priority FlowFile to process. If there is no FlowFile to process, the method will return null. Note that it is possible to have null returned even if there is a steady flow of FlowFiles into the processor. This can happen if there are multiple concurrent tasks for the processor, and the other task(s) have already retrieved the FlowFiles. If the script requires a FlowFile to continue processing, then it should immediately return if null is returned from session.get()</p> +<p><em> Groovy</em></p> +<pre>flowFile = session.get() +if(!flowFile) return +</pre> +<p><em>Jython</em></p> +<pre>flowFile = session.get() +if (flowFile != None): + # All processing code starts at this indent +# implicit return at the end +</pre> +<p><em> Javascript</em></p> +<pre>var flowFile = session.get(); +if (flowFile != null) { + // All processing code goes here +} +</pre> +<p><em> JRuby</em></p> +<pre>flowFile = session.get() +if flowFile != nil + # All processing code goes here +end +</pre> +<p> </p> +<p> </p> +<p><strong>Get multiple incoming FlowFiles from the session</strong>:</p> +<p><strong>Use Case</strong>: You have incoming connection(s) to ExecuteScript and want to retrieve multiple FlowFiles from the queue(s) for processing.</p> +<p><strong>Approach</strong>: Use the get(<em>maxResults</em>) method from the session object. This method returns up to <em>maxResults</em> FlowFiles from the work queue. If no FlowFiles are available, an empty list is returned (the method does not return null). NOTE: If multiple incoming queues are present, the behavior is unspecified in terms of whether all queues or only a single queue will be polled in a single call. Having said that, the observed behavior (for both NiFi 1.1.0+ and before) is described <a href="https://issues.apache.org/jira/browse/NIFI-2751"; target="_blank" rel="noopener">here</a>.</p> +<p><strong>Examples</strong>:</p> +<p><em> Groovy</em></p> +<pre>flowFileList = session.get(100) +if(!flowFileList.isEmpty()) { + flowFileList.each { flowFile -> + // Process each FlowFile here + } +} +</pre> +<p><em>Jython</em></p> +<pre>flowFileList = session.get(100) +if not flowFileList.isEmpty(): + for flowFile in flowFileList: + # Process each FlowFile here +</pre> +<p><em>Javascript</em></p> +<pre>flowFileList = session.get(100) +if(!flowFileList.isEmpty()) { + for each (var flowFile in flowFileList) { + // Process each FlowFile here + } +} +</pre> +<p><em>JRuby</em></p> +<pre>flowFileList = session.get(100) +if !(flowFileList.isEmpty()) + flowFileList.each { |flowFile| + # Process each FlowFile here + } +end +</pre> +<p> </p> +<p> </p> +<p><strong>Create a new FlowFile</strong></p> +<p><strong>Use Case</strong>: You want to generate a new FlowFile to send to the next processor</p> +<p><strong>Approach</strong>: Use the create() method from the session object. This method returns a new FlowFile object, which you can perform further processing on</p> +<p><strong>Examples</strong>:</p> +<p><em>Groovy</em></p> +<pre>flowFile = session.create() +// Additional processing here +</pre> +<p><em>Jython</em></p> +<pre>flowFile = session.create() +# Additional processing here +</pre> +<p><em>Javascript</em></p> +<pre>var flowFile = session.create(); +// Additional processing here +</pre> +<p><em>JRuby</em></p> +<pre>flowFile = session.create() +# Additional processing here +</pre> +<p> </p> +<p> </p> +<p><strong>Create a new FlowFile from a parent FlowFile</strong></p> +<p><strong>Use Case</strong>: You want to generate new FlowFile(s) based on an incoming FlowFile</p> +<p><strong>Approach</strong>: Use the create(<em>parentFlowFile</em>) method from the session object. This method takes a parent FlowFile reference and returns a new child FlowFile object. The newly created FlowFile will inherit all of the parent's attributes except for the UUID. This method will automatically generate a Provenance FORK event or a Provenance JOIN event, depending on whether or not other FlowFiles are generated from the same parent before the ProcessSession is committed.</p> +<p><strong>Examples</strong>:</p> +<p><em>Groovy</em></p> +<pre>flowFile = session.get() +if(!flowFile) return +newFlowFile = session.create(flowFile) +// Additional processing here +</pre> +<p><em>Jython</em></p> +<pre>flowFile = session.get() +if (flowFile != None): + newFlowFile = session.create(flowFile) + # Additional processing here +</pre> +<p><em>Javascript</em></p> +<pre>var flowFile = session.get(); +if (flowFile != null) { + var newFlowFile = session.create(flowFile); + // Additional processing here +} +</pre> +<p><em>JRuby</em></p> +<pre>flowFile = session.get() +if flowFile != nil + newFlowFile = session.create(flowFile) + # Additional processing here +end +</pre> +<p> </p> +<p> </p> +<p><strong>Add an attribute to a FlowFile</strong></p> +<p><strong>Use Case</strong>: You have a FlowFile to which you'd like to add a custom attribute.</p> +<p><strong>Approach</strong>: Use the putAttribute(<em>flowFile</em>, <em>attributeKey</em>, <em>attributeValue</em>) method from the session object. This method updates the given FlowFile's attributes with the given key/value pair. NOTE: The "uuid" attribute is fixed for a FlowFile and cannot be modified; if the key is named "uuid", it will be ignored.</p> +<p>Also this is a good point to mention that FlowFile objects are immutable; this means that if you update a FlowFile's attributes (or otherwise alter it) via the API, you will get a new reference to the new version of the FlowFile. This is very important when it comes to transferring FlowFiles to relationships. You must keep a reference to the latest version of a FlowFile, and you <u>must</u> transfer or remove the latest version of all FlowFiles retrieved from or created by the session, otherwise you will get an error when executing. Most often, the variable used to store a FlowFile reference will be overwritten with the latest version returned from a method that alters the FlowFile (intermediate FlowFile references will be automatically discarded). In these examples you will see this technique of reusing a flowFile reference when adding attributes. Note that the current reference to the FlowFile is passed into the putAttribute() method. The resulting FlowFile has an attribute nam ed 'myAttr' with a value of 'myValue'. Also note that the method takes a String for the value; if you have an Object you will have to serialize it to a String. Finally, please note that if you are adding multiple attributes, it is better to create a Map and use putAllAttributes() instead (see next recipe for details).</p> Review Comment: Change "flowFile" to "FlowFile" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
