ottobackwards commented on a change in pull request #4646: URL: https://github.com/apache/nifi/pull/4646#discussion_r614436894
########## File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/docs/org.apache.nifi.processors.standard.DeduplicateRecords/additionalDetails.html ########## @@ -0,0 +1,53 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8"/> + <title>DeduplicateRecords</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/> +</head> + +<body> +<!-- Processor Documentation ================================================== --> + <h1>Overview</h1> + <p>This processor provides deduplication across either a single record set file or across an entire data lake + using a DistributedMapCacheClient controller service. In the case of the former, it uses either a HashSet or a bloom + filter to provide extremely fast in-memory calculations with a high degree of accuracy. In the latter use case, it + will use the controller service to compare a generated hash against a map cache stored in one of the supported caching + options that Apache NiFi offers.</p> + + <h2>Configuring single file deduplication</h2> + <p>Choose the "single file" option under the configuration property labeled "Deduplication Strategy." Then choose Review comment: The size limits seems important. But the specific property to set isn't mentioned as it is with the Deduplication Strategy. I found myself going to read the code to find the property. I think we should be explicit with the property names here. Also, an example of these settings might be very helpful ( along with some description of the record etc ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
