you have missed the contributions part, I guess. On Monday, April 1, 2024 at 11:43:05 PM UTC+5:30 samm...@gmail.com wrote:
> I’m working on integrating this into my proposal right now. Feedback > would be greatly appreciated. Thanks! > Link: GSOC 2024 - Google Docs > <https://docs.google.com/document/d/1T6suQad3WgNfjGS9AVD9mHe_0WvPZijVFuVOVvSwIRE/edit> > > > On Monday, April 1, 2024 at 4:37:37 AM UTC-5 asme...@gmail.com wrote: > >> I've updated the ideas page with a link to an issue that discusses some >> ways that benchmarks on GitHub Actions could be improved. >> >> Aaron Meurer >> >> On Mon, Apr 1, 2024 at 2:35 AM Sam Lubelsky <samm...@gmail.com> wrote: >> >>> Yeah, I see no good reason for why the benchmark results show the Master >>> vs previous release. That information does not seem relevant to the PR and >>> I would bet it's causing people to ignore the benchmark when it is actually >>> saying something useful. >>> >>> I think that Master vs previous release section should be moved to a >>> separate program which is run everytime there is a new release, because >>> this information still seems useful just to see the general performance >>> trend and to see if there are any big regressions. >>> >>> It would be nice if this could be run automatically. Does this type of >>> thing seem doable in Github actions? >>> On Monday, April 1, 2024 at 3:11:18 AM UTC-5 asme...@gmail.com wrote: >>> >>>> I agree with this. The usability of the current benchmarking output >>>> needs to be improved a lot. Ideally, it should work in a way that >>>> people are actually alerted to real performance regressions, and not >>>> bothered if there aren't any performance regressions. >>>> >>>> Another issue is that the PR benchmarks comments also list the changes >>>> in master since the previous release. This is almost always completely >>>> irrelevant to the PR in question, so we should remove or demote this >>>> information. >>>> >>>> If the benchmarking system was robust enough, there would never be a >>>> regression in master, because regressions in PRs would be disallowed, >>>> the same as test failures in PRs are currently disallowed. >>>> >>>> Aaron Meurer >>>> >>>> On Mon, Apr 1, 2024 at 1:13 AM Jason Moore <moore...@gmail.com> wrote: >>>> > >>>> > This is my opinion, not sure if it is shared, but I don't think >>>> anyone uses the information that is displayed on the pull request. This >>>> isn't because the information isn't accurate or informative, but because >>>> of >>>> how and when it is presented. I haven't looked at all pull requests, of >>>> course, but I don't recall one where someone noticed the slowdown and it >>>> led to change in the PR. It has probably happened, but it happens rarely. >>>> > >>>> > The current system shows two things: timing differences in the >>>> current commit vs last release and current commit vs master. The current >>>> commit vs last release is most helpful for making the new release, but can >>>> be confusing for the PR because it contains slowdowns/speedups from more >>>> than your own PR work. The current commit vs master should show the PR >>>> author that they have made some good or bad change wrt to the benchmarks. >>>> That's all we really need to tell them (besides which benchmarks are >>>> slower >>>> and by how much). It does this, but it is easy to just not read it. >>>> > >>>> > The old way was that some of us monitored the asv generated websites >>>> and then opened issues about slowdowns and commented on the old PRs. This >>>> isn't automated but it did lead to specific comments on PRs that PR >>>> authors >>>> then were very aware of. >>>> > >>>> > Jason >>>> > moorepants.info >>>> > +01 530-601-9791 <(530)%20601-9791> >>>> > >>>> > >>>> > On Mon, Apr 1, 2024 at 2:57 AM Sam Lubelsky <samm...@gmail.com> >>>> wrote: >>>> >> >>>> >> Is there any specific problems with the current pull request >>>> benchmarking system that this project should address? >>>> >> On Sunday, March 31, 2024 at 1:41:58 PM UTC-5 moore...@gmail.com >>>> wrote: >>>> >>> >>>> >>> HI Sam, >>>> >>> >>>> >>> I think that idea could be a bit outdated. I'm not sure if the text >>>> was updated for this year. If it was, then someone else can speak up about >>>> it. >>>> >>> >>>> >>> I think that improving our sympy_benchmarks repository with more >>>> and better benchmarks and making the benchmarking system that we have >>>> setup >>>> with each pull request to sympy more useful is a better focus. I'm not >>>> sure >>>> we can run the benchmarks on a dedicated machine unless we spend some >>>> sympy >>>> funds to do that. >>>> >>> >>>> >>> We basically want to know if a pull request slows down sympy and >>>> make sure the pull request authors are warned about this in a clear way >>>> before merging. In the past it was helpful to see the historical speed of >>>> various SymPy benchmarks (here is an example I used to maintain: >>>> https://www.moorepants.info/misc/sympy-asv/) but that does require a >>>> dedicated machine so that benchmarks are comparable over time. >>>> >>> >>>> >>> Another thing I thought would be useful in the past, is to run >>>> benchmarks as part of the release process (or just before) so we can see >>>> if >>>> the upcoming release is slower than the prior release. >>>> >>> >>>> >>> Jason >>>> >>> moorepants.info >>>> >>> +01 530-601-9791 <(530)%20601-9791> >>>> >>> >>>> >>> >>>> >>> On Sun, Mar 31, 2024 at 8:13 PM Sam Lubelsky <samm...@gmail.com> >>>> wrote: >>>> >>>> >>>> >>>> Sorry if it is kinda intimidating that I put so many questions. I >>>> really just need the answer to the first one to make my proposal. I know I >>>> am a little late to GSOC, but I've really enjoyed getting to know the >>>> Sympy >>>> community a little bit in this past week and I am committed to putting >>>> together a good project proposal. >>>> >>>> Thanks, >>>> >>>> Sam. >>>> >>>> On Friday, March 29, 2024 at 4:37:55 PM UTC-5 Sam Lubelsky wrote: >>>> >>>>> >>>> >>>>> I put an introduction a few emails down, but to recap my name is >>>> Sam, I'm a college freshman, and I'm very interested in working on >>>> improving Sympy's benchmarking services over this summer through GSOC. >>>> >>>>> >>>> >>>>> While going through the project description I had a few >>>> questions: >>>> >>>>> >>>> >>>>> 1) "It also needs an automated system to run them" >>>> >>>>> What exactly is meant by this. Right now, github actions seems to >>>> be already automatically running benchmarking after each pr. Why is this >>>> not an automated system? Is the meaning of automated system something that >>>> runs weekly/monthly on the whole repo, generates a benchmark report and >>>> sends it somewhere? >>>> >>>>> >>>> >>>>> 2) How to go about hosting benchmarks on a remote, dedicated >>>> machine? What's the general idea of how to go about this in open source >>>> project. Is there money available to pay some cloud provider to host it? >>>> Free hosting options?(doesn't seem reliable enough for benchmarking). >>>> >>>>> >>>> >>>>> 3) SymEngine vs SymPy. I'm not familiar with SymEngine. >>>> Approximately how similar are SymPy and SymEngine? Is making the project >>>> also work with SymEngine more of a quick fix(≈1-2 weeks) or should I >>>> expect >>>> it to take longer? >>>> >>>>> >>>> >>>>> 4) Current Benchmark Suite >>>> >>>>> "We currently have a benchmarking suite and run the benchmarks on >>>> GitHub Actions, but this is limited and is often buggy" >>>> >>>>> >>>> >>>>> What are the limitation(s) to github actions that this project >>>> should address? >>>> >>>>> If we don't use github actions, is there another way to make it >>>> run after every PR like we have now? >>>> >>>>> >>>> >>>>> 5) Where are the tests run now? >>>> >>>>> On the project description it says " the results are run and >>>> hosted Ad Hoc", which I assumes means whatever computer is running all the >>>> other PR tests. Just want to make sure this is correct. >>>> >>>>> >>>> >>>>> >>>> >>>> -- >>>> >>>> You received this message because you are subscribed to the Google >>>> Groups "sympy" group. >>>> >>>> To unsubscribe from this group and stop receiving emails from it, >>>> send an email to sympy+un...@googlegroups.com. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/sympy/9c277927-7ac7-4c91-8c10-9ec63263f307n%40googlegroups.com. >>>> >>>> >>>> >> >>>> >> -- >>>> >> You received this message because you are subscribed to the Google >>>> Groups "sympy" group. >>>> >> To unsubscribe from this group and stop receiving emails from it, >>>> send an email to sympy+un...@googlegroups.com. >>>> >>> >> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/sympy/577753cf-ff9d-4707-a3d3-8695bbf10c77n%40googlegroups.com. >>>> >>>> >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups "sympy" group. >>>> > To unsubscribe from this group and stop receiving emails from it, >>>> send an email to sympy+un...@googlegroups.com. >>>> > To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/sympy/CAP7f1Aip-XDW6oTK%2B7Po3vVtuYT3-tVncktu%3DfEUJgDo6R-zgA%40mail.gmail.com. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "sympy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to sympy+un...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/sympy/72f318c2-0c32-40e8-9720-1539ddcb8174n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/sympy/72f318c2-0c32-40e8-9720-1539ddcb8174n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/5ac08741-407e-4d09-a9de-cc31dfc75711n%40googlegroups.com.