Thank you for the response and for directing the conversation to the correct places. I may have misunderstood what ci-hadoop.apache.org actually is. We are looking for a non-production Hadoop cluster which we can use to simulate Nutch jobs. I am not sure if this is what ci-hadoop.apache.org actually is... Instead it looks like lots of compute resources used to perform Jenkins CI/CD tasks for Hadoop and associated projects rather than test things on-top of Hadoop (and associated projects). Any clarity on what ci-hadoop.apache.org actually is would be greatly appreciated.
Let me also clarify my language, rather than have the integration tests run on every PR, we could trigger the integration tests to be run by tagging a Github bot i.e., "@nutchbot integration-test". Similar to what is done with Dependabot or conda-forge for anyon familiar with those mechanisms. Thanks for any advice or comments. lewismc On Wed, Jan 5, 2022 at 9:05 PM Ayush Saxena <ayush...@gmail.com> wrote: > Moved to Dev lists. > > Not sure about this though: > when a PR is submitted to Nutch project it will run some MR job in Hadoop > CI. > > Whatever that PR requires should run as part of Nutch Infra. Why in Hadoop > CI? > Our CI is already loaded with our own workloads. > If by any chance the above assertion gets a pass, then secondly we have > very less number of people managing work related to CI and Infra. I don’t > think most of the people won’t have context or say in the Nutch project, > neither bandwidth to fix stuff if it gets broken. > > Just my thoughts. Looped in the dev lists, if others have any feedback. As > for the process, this would require a consensus from the Hadoop PMC > > -Ayush > > > On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney <lewi...@apache.org> > wrote: > > > > Hi general@, > > > > Not sure if this is the correct mailing list. Please redirect me if there > > is a more suitable location. Thank you > > > > I am PMC over on the Nutch project (https://nutch.apache.org). I would > like > > to investigate whether we can build an integration testing capability for > > the project. This would involve running a Nutch integration test suite > > (collection of MR jobs) in a Hadoop CI environment. For example whenever > a > > pull request is submitted to the Nutch project. This could easily be > > automated through Jenkins. > > > > I’m not sure if this is something the Hadoop PMC would consider. Thank > you > > for the consideration. > > > > lewismc > > -- > > http://home.apache.org/~lewismc/ > > http://people.apache.org/keys/committer/lewismc > -- http://home.apache.org/~lewismc/ http://people.apache.org/keys/committer/lewismc
