FYI; The first update to summer 2022 edition of CAS integration test is now complete.
<https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/index.htm> Number of integrals remain the same at 85,479. This update was needed for the following: ===================================== 1. Added new check on Fricas result to detected 0 result as failed. Before it was counted as passed. There was 124 such case (one of these is valid as integrand is zero) sqlite> select COUNT(*) from main where fricas_anti=0; 125 This reduced Fricas % pass score by only 0.15 %. Hopefully these integrals will be fixed in the next version of Fricas. 2. re-run all of Fricas/Giac/Maxima tests to make sure there was no false time out by improving the sagemath script. Only few such false timeouts were found and corrected. 3. Reduced the size of the pdf's and web pages by only showing small part of the anti-derivative for a very long result (which used to take many pages before). There is an issue with sagemath interface to Fricas which causes this on some integrals. This should be fixed in sagemath 9.8. 4. Updated the SQL database for current update. CAS tested ================= Mathematica 13.1 (June 29, 2022) on windows 10. Rubi 4.16.1 on Mathematica 13.0.1 on windows 10. Maple 2022.1 on windows 10. Maxima 5.46 using Lisp SBCL 2.1.11.debian via sagemath 9.6. Fricas 1.3.8 based on sbcl 2.1.11.debian via sagemath 9.6. Giac/Xcas 1.9.0-13 via sagemath 9.6. Sympy 1.10.1 Using Python 3.10.4 on Linux. Mupad using Matlab 2021a on windows 10. Percentage solved (compared to summer 2021 build) 2022 2021 ===== ====== Mathematica 97.99 % 98.39 % Rubi 94.21 % 99.52 % Maple 84.53 % 83.58 % Fricas 79.27 % 68.70 % Giac 58.45 % 52.43 % Maxima 56.88 % 52.72 % Mupad 56.10 % 52.55 % Sympy 41.89 % 34.47 % Fricas had the best improvement followed by Sympy and Giac from last year. Maxima can improve the score more if there is a way to eliminate it asking question during the integration which cause it to fail in those cases. Grading % =========== System A B C F ------- ----- ----- ---- ---- Rubi 91.5 1.95 0.75 5.79 Mathematica 78.58 6.21 13.18 2.01 Maple 56.34 18.53 9.66 15.47 Fricas 53.82 19.55 5.9 20.73 Maxima 43.57 11.69 1.62 43.12 Giac 42.41 14.98 1.06 41.55 Sympy 28.49 9.81 3.6 58.11 Mupad* 4.15 51.95 0. 43.9 (*)Mupad is not graded. B is given as placeholder for now. Average Time per second ========================= Rubi 0.26 Maxima 0.53 Giac 1.55 Fricas 1.97 Mathematica 2.38 Mupad 2.74 Maple* 3.33 Sympy 5.24 Maple(*) timing: the option method=_RETURNVERBOSE was used which made Maple run through all integration methods increasing its average time used. The regression, progress and SQL database can be accessed from the main page. Here are the direct links <https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REGRESSION_REPORTS/index.htm> <https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REVERSE_REGRESSION_REPORT/index.htm> <https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/DATA_BASE/index.htm> Hopefully no more updates will be needed unless problems are found. --Nasser ps. I post this here via google web interface. I have to login to google to do this each time and the message seems to lose its formatting for some reason. I wish Fricas will get away from using a google group for communication and use a better platform where one can edit messages after posting and formatting of messages do not get lost. On Thursday, September 15, 2022 at 2:43:25 AM UTC-5 Nasser M. Abbasi wrote: > FYI; > > A small update is made for the summer 2022 edition > for CAS integration tests that only affects only Fricas, Maxima > and Giac to verify some result related to timeouts. > > This below is the result for Fricas as it just finished. The others are > still running. > > The following are the 9 integrals that fail in 1.3.8 vs. 1.3.7 > > In these, The integrals that fail due to time out is because of > the setting of setSimplifyDenomsFlag to true. Otherwise they > complete immediately. > ======================================== > > integrate(x*(e*x^3+d)/(c*x^6+b*x^3+a),x) (time out due to flag) > integrate((e*x^3+d)/(c*x^6+b*x^3+a),x) (time out due to flag) > integrate((F^((-a*x+1)^(1/2)/(a*x+1)^(1/2)))^n/(-a^2*x^2+1),x) (not solved) > integrate(F^(3*(-a*x+1)^(1/2)/(a*x+1)^(1/2))/(-a^2*x^2+1),x) (not solved) > integrate(F^(2*(-a*x+1)^(1/2)/(a*x+1)^(1/2))/(-a^2*x^2+1),x) (not solved) > integrate(F^((-a*x+1)^(1/2)/(a*x+1)^(1/2))/(-a^2*x^2+1),x) (not solved) > integrate(sec(d*x+c)*(a+b*sin(d*x+c))^(5/2),x) (not solved) > integrate(cot(f*x+e)^2*(c+d*tan(f*x+e))^(1/3),x) (time out due to flag) > integrate(((x^2-3*x-4)*log(1+x)+((-x^3+3*x^2+4*x)*exp(x+7)+3*x^2-10*x-8)*log(x)+(-x^2+3*x+4)*exp(x+7)+x^3-4*x^2-5*x)/(x^3-3*x^2-4*x),x) > > (not solved) > > The report is at > > < > https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REGRESSION_REPORTS/fricas/report.htm > > > > There is also now a new _reverse regression report_. This shows > all integrals that were solved in Fricas 1.3.8 but failed in 1.3.7. > > 5,647 such integrals were found for Fricas. Many seem to be solved now > using the new weierstrass, weierstrassZeta, weierstrassPInverse > special functions introduced in 1.3.8. > > The report at > > < > https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REVERSE_REGRESSION_REPORT/fricas/report.htm > > > > It will take another 2-3 weeks to finish building the full > website again and all the individual reports for the other CAS systems > (each will have regression and reverse regression report also) > and the SQL database. Will make a post when done. > > Any bugs/problems found please let me know. > > --Nasser > > > > > On Monday, August 22, 2022 at 6:13:22 AM UTC-5 Nasser M. Abbasi wrote: > >> Thanks Qian for finding this. >> >> I just found the problem and just fixed it. >> >> It is related to running these tests in spawned process using Python's >> multiprocessing. I have to run all these integration tests in a spawned >> process to be able to set a timeout of 3 minutes. >> >> Will rerun all the tests for Fricas/maxima and giac again and rebuild the >> reports just to make sure there are no other integrals that got affected by >> this false timeout. Will take me may be a week. Will post reply here when >> done. >> >> As for using setSimplifyDenomsFlag, yes. I set to true before each call >> from Sagemath using >> >> fricas.setSimplifyDenomsFlag(fricas.true) >> >> So all Fricas integrals are run with this flag set to true. >> >> --Nasser >> On Monday, August 22, 2022 at 5:04:06 AM UTC-5 oldk1331 wrote: >> >>> Hi Nasser, >>> >>> Here is another 2 (minor) problems in your regression report: >>> >>> >>> https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REGRESSION_REPORTS/fricas/reportsection7.htm#x8-190007 >>> >>> >>> integrate((e*x)^m*(a+b*log(c*log(d*x)^p)),x) >>> >>> This returns quickly in fricas-1.3.8, it does not timeout. >>> Can you double check on your side please. >>> >>> >>> https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REGRESSION_REPORTS/fricas/reportsection9.htm#x10-230009 >>> >>> >>> integrate(cot(f*x+e)^2*(c+d*tan(f*x+e))^(1/3),x) >>> >>> For this integral, it returns quickly if you don't set >>> "setSimplifyDenomsFlag true". >>> >>> (P.S. fricas-1.3.7 gave wrong answer.) >>> >>> BTW, do you run all FriCAS tests with "setSimplifyDenomsFlag true"? >>> >>> - Qian >>> >>> On 8/21/22 14:15, 'Nasser M. Abbasi' via FriCAS - computer algebra >>> system wrote: >>> > FYI, >>> > >>> > (Posted at https://groups.google.com/g/fricas-devel and >>> > https://groups.google.com/g/sci.math.symbolic) >>> > >>> > Independent CAS integration tests summer 2022 edition is now complete. >>> > (It took about 5 months since I initially started working on it!) >>> > >>> > < >>> https://www.12000.org/my_notes/CAS_integration_tests/reports/summer_2022/index.htm> >>> >>> >>> > >>> > The following is summary of changes in this version >>> > compared to last year's edition >>> > ======================== >>> > 1. Number of integrals increased to 85,479 from 71,994. >>> > >>> > The integrals are made of the following 210 files: >>> > >>> > a. Rubi test suite. Files 1 to 208. Thanks to Alert Rich. >>> > Downloaded from <https://github.com/RuleBasedIntegration> >>> > b. IntegrateAlgebraic test file. File 209. Thanks to Sam Blake. >>> > c. Fricas test file. File 210. Thanks to Waldek Hebisch. >>> > >>> > 2. CAS version changes: >>> > Maple 2021.1 -> 2022.1 >>> > Mathematica 12.3 -> 13.01 >>> > Giac 1.7.0 -> 1.9.0-13 >>> > Sympy 1.8/python 3.8.8 -> 1.10.1/Python 3.10.4 >>> > Maxima 5.44 -> 5.46 >>> > FriCAS 1.3.7 -> 1.3.8 >>> > Sagemath 9.3 -> 9.6 >>> > >>> > No version change for Mupad at Matlab 2021a and Rubi at 4.16.1 >>> > but Rubi was run again on the current Mathematica 13.1 version. >>> > >>> > 3. General improvements to Latex and other formatting. >>> > >>> > 4. Flow chart of the test program at >>> > < >>> https://www.12000.org/my_notes/CAS_integration_tests/images/design.svg> >>> > >>> > 5. Links to download all the 85,483 integrals in plain text in >>> > Mathematica, Rubi, Maple, Mupad, Sympy, and Sagemath (Fricas, Maxima, >>> Giac) >>> > formats are at introduction page of the main page. >>> > >>> > 6. Regression reports comparing results in 2022 edition to 2021 >>> edition >>> > are also now available. These reports show integrals that failed >>> > in current versions compared to last versions per CAS. >>> > >>> > < >>> https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/REGRESSION_REPORTS/index.htm> >>> >>> >>> > >>> > Regression per CAS >>> > ------------------ >>> > This table below gives the number of integrals that failed in >>> > this test (could be due to timeout, exception or not solved) but >>> > are solved in summer 2021 for each CAS. >>> > >>> > Number of regression per CAS is (less is better) >>> > >>> > Fricas: 16 >>> > Maxima: 17 >>> > Mathematica: 98 >>> > Maple: 114 >>> > Giac: 363 >>> > Sympy: 379 >>> > >>> > Tables give details about each integral that failed in regression. >>> > >>> > 7. Sympy, Sagemath (Fricas, Maxima, Giac) were all run on Linux >>> > inside Oracle Virtual Box. Maple, Rubi, Mupad and Mathematica were >>> > run on windows 10. All on same PC with 128 GB RAM with >>> > Intel Core i9-12900K 3.20 GHz. >>> > >>> > RESULTS >>> > ======= >>> > >>> > Overall solved percentage in summer 2022 edition >>> > ========================================= >>> > 1. Mathematica 13.1 97.99 % >>> > 2. Rubi 4.16.1 94.21 % >>> > 3. Maple 2022.1 84.53 % >>> > 4. Fricas 1.3.8 79.33 % >>> > 5. Giac 1.9.0-13 58.37 % >>> > 6. Maxima 5.46 56.86 % >>> > 7. Mupad 2021a 56.1 % >>> > 8. Sympy 1.10.1 41.89 % >>> > >>> > Grading results >>> > ================= >>> > Note that mupad is not graded. Default grade of B is given for >>> > any passed integral. >>> > >>> > 1. Rubi 91.50 % >>> > 2. Mathematica 78.58 % >>> > 3. Maple 56.34 % >>> > 4. Fricas 53.78 % >>> > 5. Maxima 43.57 % >>> > 6. Giac 42.36 % >>> > 7. Sympy 28.49 % >>> > 8. Mupad* N/A >>> > >>> > Time and leaf size Performance >>> > ============================== >>> > Sorted by time performance in average time used per integrals >>> (seconds). >>> > >>> > Note for Maple(*) timing: the option method=_RETURNVERBOSE was used >>> > which made Maple run through all integration methods increasing >>> > average time used. For example in summer 2021 edition which did >>> > not use this option, Maple had 0.79 seconds average time on >>> > a slower PC also than the one used for the current tests. >>> > >>> > 1. Rubi 0.3 (sec) >>> > 2. Maxima 0.49 (sec) >>> > 3. Giac 0.75 (sec) >>> > 4. Fricas 1.17 (sec) >>> > 5. Mathematica 2.38 (sec) >>> > 6. Mupad 2.75 (sec) >>> > 7. Maple* 3.33 (sec) >>> > 8. Sympy 5.24 (sec) >>> > >>> > General notes >>> > ============== >>> > 1) Maxima, Fricas and Giac were called via Sagemath. Some integrals >>> > could have failed due to interface issues. These will have F(-2) >>> > error code in the section "Detailed conclusion table per each >>> > integral for all CAS systems" per each report and also in the actual >>> > integral page with the actual exception name shown. >>> > >>> > This allows one to see if the cause of the failure is due to >>> > interface issue or internal error from the CAS itself. >>> > >>> > For an example, the integral on >>> > >>> > < >>> https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/test_cases/4_Trig_functions/4.7_Miscellaneous/139_4.7.5_x%5Em_trig-a+b_log-c_x%5En-%5Ep/reportsubsection69.htm#53> >>> >>> >>> > >>> > Has F(-2) on Fricas output, but looking at the exception >>> > >>> > integrate(x*sin(a+b*log(c*x^n))^(1/2),x, algorithm="fricas") >>> > >>> > "Exception raised: TypeError >> Error detected within library code: >>> > integrate: implementation incomplete (has polynomial part)" >>> > >>> > So this is not interface issue with Sagemath. The exception comes >>> > from Fricas itself. >>> > >>> > But this integral >>> > >>> > < >>> https://12000.org/my_notes/CAS_integration_tests/reports/summer_2022/test_cases/4_Trig_functions/4.7_Miscellaneous/139_4.7.5_x%5Em_trig-a+b_log-c_x%5En-%5Ep/reportsubsection57.htm#41> >>> >>> >>> > >>> > Has F(-2) on Giac result, and looking at the exception it says >>> > >>> > integrate(x^2*sin(a+log(c*x^n)*(-1/n^2)^(1/2))^3,x, algorithm="giac") >>> > >>> > "Exception raised: NotImplementedError >> >>> > Unable to parse Giac output: >>> > ((-9*i)*sageVARn^4*sageVARx^3*exp((-3*i) *sageVARa)* >>> > exp((3*sageVARn*abs(sageVARn)*ln(sageVARx)+3*abs(sageVARn)* >>> > ln(sageVARc))/sageVARn^2)+27*i*sageVARn^4*sageVARx^3*exp((-i)" >>> > >>> > Which indicates it is an interface issue this time. This is >>> > verified by running this integral directly in giac and it is >>> > solved with no error. >>> > >>> > CAS integration test program currently does not distinguish >>> > between an exception generated due to a sagemath interface problem >>> > or an exception generated from the CAS itself. I am not >>> > sure if there a robust way to do this for Maxima, Fricas >>> > and Giac. If a way can be found, will add it. >>> > >>> > But there seems to be less of these interface problems in the >>> > current Sagemath version than the last one used. >>> > >>> > 2) The section "List of integrals sorted by grade for each CAS" in >>> each >>> > report shows list of integrals for each grade per CAS. This allows >>> > one to quickly find integrals with specific grade. >>> > >>> > 3) Also as mentioned before there are a number of integrals for Maxima >>> > which fail when it asks a question in order to complete the >>> > integration as this is not supported by Sagemath or by the >>> > CAS integration test program. If an option can be added to >>> > turn this feature off that will be a good thing and will improve >>> > Maxima score. >>> > >>> > 4) Any bugs found or problems please let me know and will try my best >>> > to fix them. >>> > >>> > --Nasser >>> > >>> >> -- You received this message because you are subscribed to the Google Groups "FriCAS - computer algebra system" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/fricas-devel/e1fbf96d-ea83-4bdf-960b-adff82f6c3e8n%40googlegroups.com.
