My apologies, I missed the link: [1] https://github.com/gbif/pipelines
On Tue, Apr 21, 2020 at 5:58 PM Tim Robertson <[email protected]> wrote: > Hi Jordan > > I don't know if we qualify as a large Beam project but at GBIF.org we > bring together datasets from 1600+ institutions documenting 1,4B > observations of species (museum data, citizen science, environmental > reports etc). > As far as Beam goes though, we aren't using the most advanced > features. It's batch processing of data into Avro files stored on HDFS then > into HBase / Elasticsearch. > > All our data and code [1] are open and I'm happy to discuss any aspect of > it if it is helpful to you. > > Best wishes, > Tim > > On Tue, Apr 21, 2020 at 3:48 PM Jeff Klukas <[email protected]> wrote: > >> Mozilla hosts the code for our data ingestion system publicly on GitHub. >> A good chunk of that architecture consists of Beam pipelines running on >> Dataflow. >> >> See: >> >> https://github.com/mozilla/gcp-ingestion/tree/master/ingestion-beam >> >> and rendered usage documentation at: >> >> https://mozilla.github.io/gcp-ingestion/ingestion-beam/ >> >> On Mon, Apr 20, 2020 at 7:11 PM Jordan Thomas-Green <[email protected]> >> wrote: >> >>> Does anyone have any public repos/examples of larger Beam >>> projects/implementations that they've seen? >>> >>
