[
https://issues.apache.org/jira/browse/ATLAS-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated ATLAS-5399:
----------------------------------------
Summary: Add Apache Nutch metadata model and CrawlDb/HostDB import bridge
(was: Add Apache Nutch metadata model and CrawlDb import bridge)
> Add Apache Nutch metadata model and CrawlDb/HostDB import bridge
> ----------------------------------------------------------------
>
> Key: ATLAS-5399
> URL: https://issues.apache.org/jira/browse/ATLAS-5399
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-intg
> Affects Versions: 3.0.0
> Reporter: Lewis John McGibbney
> Priority: Major
>
> Add first-class [Apache Nutch|http://nutch.apache.org/] support in Atlas: a
> bootstrap type model and a batch bridge that imports crawl catalog metadata
> from an existing Nutch CrawlDb.
> *Motivation*
> Atlas has no crawler/Nutch types. Operators want to govern crawls, seed
> lists, hosts, and registrable domains, and to see lineage from a Nutch
> crawl/segment into a search index. URL-level entities are intentionally out
> of scope: CrawlDb is URL-keyed at web scale and would not fit Atlas's
> table-shaped models.
> *Scope*
> # Bootstrap typedefs under addons/models/7000-Nutch/ (serviceType nutch):
> ## nutch_crawl, nutch_segment extend DataSet
> ## nutch_seedlist, nutch_domain, nutch_host extend Asset
> ## nutch_index_process extends Process (instances created by Nutch, not this
> bridge)
> ## Index sink is a generic DataSet (no Solr/ES typedef set)
> ## COMPOSITION: crawl to seedlist (1:1), crawl to segments
> ## AGGREGATION: domain to hosts
> ## ASSOCIATION: many to many relationships between crawls and hosts with
> relationship attributes (fetchedCount, unfetchedCount, indexedCount,
> lastFetchTime, lastIndexedTime, maxScore)
> ## qualifiedName uses name@clusterName
> ## nutch_domain is eTLD+1 / assigned domain (URLUtil.getDomainName /
> crawler-commons EffectiveTldFinder), NOT TLD
> ## nutch_host is global per cluster (hostname@clusterName)
> # addons/nutch-bridge: CLI importer using AtlasClientV2 (username/password
> or token). Reads \{crawlId}/crawldb, seed dir, segments listing. Creates
> crawl, seedlist, segments, domains, and hosts that have at least one FETCHED
> URL. Does not create nutch_index_process.
> # Distro packaging, docs (HookNutch), and tests with a small CrawlDb fixture
> or mocked CrawlDatum rollup.
> *Out of scope*
> * URL/page entity types or URL sample payloads
> * Nutch REST admin / JobManager types (removed in NUTCH-3165)
> * AtlasHook / ATLAS_HOOK / consuming indexer-kafka JSON
> * Inject/fetch/parse Process types (catalog relationships only in this first
> pass) Creating host/domain/crawl entities from the Nutch IndexWriter (see
> NUTCH)
> See also: NUTCH-3210 (Atlas IndexWriter)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
