Lewis John McGibbney created NUTCH-3210:
-------------------------------------------
Summary: Add Atlas IndexWriter to emit crawl/segment lineage to
Apache Atlas
Key: NUTCH-3210
URL: https://issues.apache.org/jira/browse/NUTCH-3210
Project: Nutch
Issue Type: New Feature
Components: indexer, plugin
Affects Versions: 1.24
Reporter: Lewis John McGibbney
Assignee: Lewis John McGibbney
Fix For: 1.24
Add an Atlas IndexWriter plugin that, at IndexingJob time, upserts lineage into
[Apache Atlas|http://atlas.apache.org] via AtlasClientV2 REST.
*Motivation*
Atlas will own Nutch typedefs and a CrawlDb batch bridge (ATLAS-XXXX).
Index-time lineage (crawl/segment to index DataSet) belongs in Nutch so it runs
with the existing indexer plugin path.
*Scope*
# New plugin src/plugin/indexer-atlas implementing IndexWriter
# Configure via conf/index-writers.xml and plugin.includes
# One nutch_index_process per IndexingJob (not per URL)
# Process.inputs: nutch_crawl and/or nutch_segment (must already exist in
Atlas)
# Process.outputs: generic DataSet named by collection/index + clusterName
# Optional updates to existing crawl–host relationship indexedCount /
lastIndexedTime only when the host already exists
# Auth: Atlas REST username/password or token; atlas.cluster.name shared with
the Atlas bridge
# Unit tests with a mocked Atlas client; docs for config keys
*Out of scope*
# Reusing indexer-kafka (payload is NutchDocument JSON, not HookNotification)
# AtlasHook / Kafka ATLAS_HOOK
# Creating crawl, seedlist, domain, or host entities (Atlas CrawlDb bridge)
# URL-level Atlas entities
# Restoring NutchServer / JobManager metadata
Depends on: ATLAS-XXXX (Nutch type model; process/index DataSet types must
exist)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
