Github actions - OWASP check status
Hi all, As you may have noticed the owasp check is failing on the master branch. I sent out multiple pulls to solve the vulnerability checks: - https://github.com/apache/pulsar/pull/14802 - https://github.com/apache/pulsar/pull/14871 - https://github.com/apache/pulsar/pull/14884 Please review them in order to fix the CI status BR, Nicolò Boschi
Re: Jacoco and Codecov.io integration for Code Coverage reports
Thanks Enrico for the feedback. I would like to merge the PR to complete the setup and reach more feedback for other contributors. Once we complete the setup, the Codecov bot will add a comment on every new pull request Nicolò Boschi Il giorno mar 22 mar 2022 alle ore 18:32 Enrico Olivelli < eolive...@gmail.com> ha scritto: > Nicolò, > Great stuff indeed. > Some comments inline below > > Thanks > > > Il Mer 16 Mar 2022, 16:45 Nicolò Boschi ha scritto: > > > Dear community, > > > > Currently we don't have an aggregate code coverage report. There's a > Jacoco > > configuration to generate the report after unit tests but it is not > > easily readable and understandable. > > > > Having a better system would bring a couple of interesting aspects: > > 1. We can block a pull request if the code coverage decreases - that > means > > there are no tests for the new code. > > > > I am afraid that this would lead to some automatic excessive nitpicking. > Let's keep this in the 'future works' basket > > 2. We can have an overview for each modules and discover which features are > > less covered and add new tests > > > > I would like to introduce a new tool that reads, aggregates, stores and > > reports the code coverage for the whole codebase. This tool is Codecov ( > > https://about.codecov.io/) that is one of the most popular tools for > open > > source projects. > > It brings multiple advantages: > > 1. It aggregates all the Jacoco reports in a single report > > 2. It keeps the coverage history > > 3. It is well integrated with Github > > > > The idea is the following: > > - Add this profiling in the CI - for each test suite and upload the > results > > to codecov.io > > - Enable the codecov report that add a comment in the pull > > > > > > Implementation pull: https://github.com/apache/pulsar/pull/14600 > > Codecov comment in the PR: > > https://github.com/apache/pulsar/pull/14600#issuecomment-1061636495 > > Codecov full report: > > > > > https://codecov.io/gh/apache/pulsar/tree/0fa212d40b2bf166b20f3155c40fdb9624f577ed > > > > Initially there were some doubts around whether the jacoco agent would > have > > added a significant overhead (execution time + memory usage). I've seen > > (and you can see the checks times in my pull) that it is not relevant in > > the Pulsar codebase. > > > > The pull only adds Jacoco coverage for unit tests. Integrating jacoco > with > > the integration tests (inside docker) will be more complex but it is > > possible. We can do it in a second moment. > > You basically need to add the jacoco agent inside the docker container, > > enabling it in the java command and downloading the report files before > the > > container stops. > > > > > > Note: Codecov is free for open source projects ( > > https://about.codecov.io/pricing/). It is required to install the app on > > Github Marketplace (https://github.com/marketplace/codecov). > > > > Is it something we should ask to ASF infra probably. > > The PMC can deal with this > > Enrico > > > > BR, > > Nicolò Boschi > > >
[GitHub] [pulsar-site] urfreespace merged pull request #24: Generate add scripts for other clients
urfreespace merged pull request #24: URL: https://github.com/apache/pulsar-site/pull/24 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] urfreespace merged pull request #27: fix: go/nodejs/c# release notes
urfreespace merged pull request #27: URL: https://github.com/apache/pulsar-site/pull/27 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Abount add TDengine Connector to Pulsar.
Hi EveryOne, I would like to contribute a TDengine connector to Pulsar. Let me briefly introduce what a [TDengine](https://tdengine.com/) is. TDengine is a high-performance, distributed time series database that supports SQL. With TDengine, the total cost of ownership of typical IoT, Internet of Vehicles, and Industrial Internet big data platforms can be greatly reduced. About Pulsar IO TDengine Connector, it is composed of TDengine Source Connector and TDengine Sink Connector. It can read data from TDengine and store it in Pulsar. At the same time, it can also write data in Pulsar into TDengine to realize TDengine-based data pipeline. Thanks, JueShan
Re: [Discuss] draft PIP for "Changes to GitHub Actions based Pulsar CI"
The PR https://github.com/apache/pulsar/pull/14819 is now ready for review. The changes in the PR now use GitHub Actions Artifacts for sharing binary files (such as docker images) between the build steps. This saves a lot of GitHub Actions VM resources since the docker images are built once and shared in downstream jobs. GitHub Actions Artifacts are meant to be used for sharing data between the jobs in a GitHub Actions workflow [1]. I'm looking forward to your review and feedback on https://github.com/apache/pulsar/pull/14819 . BR, -Lari References: [1] GitHub Actions: Storing workflow data as artifacts - https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts On 2022/03/23 10:46:10 Lari Hotari wrote: > I have submitted the PR for refactoring the apache/pulsar GitHub Actions > based CI. Please review https://github.com/apache/pulsar/pull/14819 . > > BR, > -Lari > > On 2022/03/22 13:38:36 Enrico Olivelli wrote: > > Lari, > > > > Il Mar 22 Mar 2022, 14:32 Lari Hotari ha scritto: > > > > > I have resumed work to improve our GitHub Actions based Pulsar CI. > > > > > > Last year, I worked on a proof-of-concept which significantly reduced the > > > resource consumption and improved the usability of the build by combining > > > multiple workflows into a single larger workflow. > > > > > > The showstopper a year ago was the lack of being able to re-run a single > > > failed job in a larger workflow. > > > GitHub has since then delivered this feature and no showstoppers are > > > present. > > > > > > I have been posting updates to > > > https://github.com/apache/pulsar/issues/14401 "Speed up CI workflows" > > > about the progress. > > > I have rebased the changes from last year's PoC, and I'm finalizing and > > > testing the changes in my fork under > > > https://github.com/lhotari/pulsar/pull/59 . I'll send a PR to > > > apache/pulsar, when the refactoring is ready. > > > > > > > This is great news ! > > > > Looking forward to your patch > > > > Enrico > > > > > > > > > -Lari > > > > > > On 2021/03/16 01:10:52 Sijie Guo wrote: > > > > > The prototype has demonstrated about 60% reduction in > > > > resource consumption. > > > > > > > > It is hard to quantify. Merging them into one large workflow can result > > > in > > > > more failures. Re-running those failures can consume resources as well. > > > > > > > > > Isn't it urgent to resolve it? > > > > > > > > I think we are in a stage that gives us breathing room to fix flaky > > > > tests > > > > and solve other problems, no? > > > > I don't mean we stop the effort here. I mean we have other enhancements > > > > that we can do to improve the situation. > > > > Once we get into a position where the flakiness is reduced, we can merge > > > > them into one workflow. > > > > > > > > - Sijie > > > > > > > > > > > > On Mon, Mar 15, 2021 at 2:48 AM Lari Hotari wrote: > > > > > > > > > Thanks for the feedback Sijie. > > > > > > > > > > > We are using a lazy consensus approach. Typically if there is no > > > > > objection, > > > > > > please go ahead and not need to wait for approval. > > > > > > If people raise concerns, please address the concerns. > > > > > > > > > > You and Ali have raised concerns about changing the existing GitHub > > > Actions > > > > > workflows in a way where multiple workflows would be combined to a > > > single > > > > > workflow. Before proceeding, there is a need to address the concerns. > > > We > > > > > might end up with a completely different type of solution of what has > > > been > > > > > proposed initially. :) > > > > > > > > > > > Yes. So I am in favor of addressing flaky tests than merging all > > > > > workflows > > > > > > into one giant workflow. > > > > > > > > > > I agree that addressing flaky tests is favorable. The main reason for > > > PIP > > > > > "Changes to GitHub Actions based Pulsar CI" is to > > > > > 1) Reduce GitHub Action Runner resource consumption of Pulsar PR > > > > > builds > > > > > 2) Reduce lead times for Pull Request feedback > > > > > We cannot ignore these problems. If we don't change anything, the > > > problems > > > > > won't get fixed. The prototype has demonstrated about 60% reduction in > > > > > resource consumption. Measuring the lead times hasn't been done in the > > > > > prototype, but since the reason for long lead times has been long > > > > > build > > > > > queues due to excessive resource consumption, it's likely that the > > > > > lead > > > > > times would be reduced. > > > > > > > > > > I know that switching to a single workflow isn't the only solution to > > > the > > > > > above problems. I had a discussion with Ali. He recommended reducing > > > the > > > > > modules in Pulsar repository (PIP-62), reducing the docker container > > > size > > > > > and improving the Pulsar Broker unit test harness so that tests using > > > it > > > > > would be less flaky and that it would be easier to fix the issues in > > > > > failing test wh
[GitHub] [pulsar-site] urfreespace merged pull request #28: fix: restore files that mistaken deleted
urfreespace merged pull request #28: URL: https://github.com/apache/pulsar-site/pull/28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] urfreespace opened a new pull request #29: feat: upgrade docusaurus to 2.0.0-beta.18
urfreespace opened a new pull request #29: URL: https://github.com/apache/pulsar-site/pull/29 Signed-off-by: Li Li -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] urfreespace merged pull request #29: feat: upgrade docusaurus to 2.0.0-beta.18
urfreespace merged pull request #29: URL: https://github.com/apache/pulsar-site/pull/29 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
