On 21 October 2013 10:58, Andre Fischer <awf....@gmail.com> wrote: > On 20.10.2013 12:40, janI wrote: > >> On 19 October 2013 19:20, Kay Schenk <kay.sch...@gmail.com> wrote: >> >> On Fri, Oct 18, 2013 at 7:52 AM, Andre Fischer <awf....@gmail.com> >>> wrote: >>> >>> On 18.10.2013 15:58, janI wrote: >>>> >>>> On 18 October 2013 15:00, Andre Fischer <awf....@gmail.com> wrote: >>>>> >>>>> On 18.10.2013 14:02, janI wrote: >>>>> >>>>>> sd >>>>>> >>>>>>> >>>>>>> On 18 October 2013 13:36, Andre Fischer <awf....@gmail.com> wrote: >>>>>>> >>>>>>> On 18.10.2013 11:32, janI wrote: >>>>>>> >>>>>>> Hi. >>>>>>>> >>>>>>>> due to the discussion in thread "Mentor a new build system", I have >>>>>>>>> made a >>>>>>>>> proposal for a central Makefile located in main. >>>>>>>>> >>>>>>>>> Hi Jan, >>>>>>>>> >>>>>>>>> it is great that you are going to improve this part of the build >>>>>>>> system. >>>>>>>> But I think that we need more details about how the proposed >>>>>>>> build >>>>>>>> system >>>>>>>> works. Without them I can not really evaluate the proposal. >>>>>>>> >>>>>>>> First of all, I agree with juergens remarks that this should be >>>>>>>> >>>>>>>> discussed >>>>>>> before implemented, hence the wiki page. >>>>>>> >>>>>>> Secondly this has nothing directly to do with the proposed build >>>>>>> >>>>>> system, >>> >>>> its a simple replacement of build.pl in the current system. >>>>>>> >>>>>>> Yes, that is how I understood it. I just did not know how to call >>>>>>> >>>>>> the >>> >>>> build.pl replacement. >>>>>> >>>>>> >>>>>> >>>>>> I know that build.pl works, but having a Makefile in main, would >>>>>> make >>>>>> >>>>>>> us >>>>>>> one step closer on being compatible with the distros. To me this job >>>>>>> >>>>>> is >>> >>>> a >>>>>>> simple cleanup, not something we deadly need, but nice to have. >>>>>>> >>>>>>> >>>>>>> Some remarks regarding the missing options: >>>>>>> >>>>>>> --from <module> >>>>>>>> This is one of the more important options and one that I use >>>>>>>> frequently >>>>>>>> (also in the form --all:<module>). >>>>>>>> Note that if you are in <moduleA> and call 'make --from >>>>>>>> >>>>>>> <moduleB>' >>> >>>> then >>>>>>>> all modules are built >>>>>>>> a) which <moduleA> depends on >>>>>>>> b) but not those that <moduleB> depends on >>>>>>>> c) Both <moduleA> and <moduleB> are built. >>>>>>>> >>>>>>>> I have changed the documentation. >>>>>>>> >>>>>>>> I use the --all:<module> myself very often, and have changed the >>>>>>> documentation, because it is of course supported. >>>>>>> >>>>>>> The difference is that you do the call in main, but that is a minor >>>>>>> detail >>>>>>> that can be easily corrected (have <module>/Makefile calling >>>>>>> main/Makefile. >>>>>>> >>>>>>> I have also changed documentation on --html due to juergens comments. >>>>>>> >>>>>>> I am not sure that we understand --from and --since in the same way >>>>>>> >>>>>> so >>> >>>> I >>>>>> will try to explain what I think they do. >>>>>> >>>>>> Let's imagine that we have a simple project with modules A, B, C, D >>>>>> and >>>>>> E. >>>>>> where B depends on A, C on B, D on C, and E on D. >>>>>> A ' make all' would mean 'make E'. The dependencies would then lead >>>>>> to >>>>>> building modules A, B, C, D, E in this order. >>>>>> If I am in E and call 'make --from C' then only C, D, and E should be >>>>>> built. A 'make --since C' would only build D and E. >>>>>> >>>>>> If I am in D and call 'make --from B' then modules B, C, and D are >>>>>> >>>>> built. >>> >>>> Call 'make --since B' to build only C and D. >>>>>> Note that 'make --from' accepts more than one module name (while 'make >>>>>> --all:<module>' does not). >>>>>> Note also that in the above case (stand in D, call 'make --from B') >>>>>> module >>>>>> A is not built, regardless of whether there are changes in A or not. >>>>>> Whereas a simple call to make (still standing in D) would build all >>>>>> modules that D depends on, directly or indirectly. Thus the options >>>>>> '--from' and '--since' exist to actively exclude modules from being >>>>>> built. >>>>>> >>>>>> The whole thing becomes a little bit more complicated with multiple >>>>>> options to '--from' (I never use '--since' and also don't know a valid >>>>>> use >>>>>> case so I will ignore it for now) and more complex dependencies then >>>>>> in >>>>>> the >>>>>> simple example above. Let's say that if we stand in instsetoo_native >>>>>> >>>>> and >>> >>>> call 'make --from svx sfx2'. Note that svx depends on sfx2. This >>>>>> >>>>> would >>> >>>> build svx, sfx2 and all modules that depend (directly or indirectly) on >>>>>> svx >>>>>> OR sfx2. >>>>>> >>>>>> got it, now I just have one problem, why would you not build the >>>>>> >>>>> dependent >>>>> modules, if they needed to be built, thats a scenario I dont >>>>> understand. >>>>> With a central makefile, <module>/makefile will not be called so we do >>>>> >>>> not >>> >>>> waste cpu cycles. >>>>> >>>>> With the .done files, we know when a module was last built and all >>>>> >>>> modules >>> >>>> that depend it should be rebuilt which the rule >>>>> <module>.done : <module_depend>.done >>>>> >>>>> will ensure, so If we have A -> B -> C -> D >>>>> >>>>> I go in B, and call make, then when I go in D and make, B,C,D will be >>>>> made. >>>>> >>>>> If we have A -> B -> D C -> D >>>>> and do the same then only D will be made. >>>>> >>>>> So --from is not really saving anything ? >>>>> >>>>> a) In your example you first go into B then, in a second step, into D. >>>> The '--from' option lets you do the same (well, not really the same, >>>> but >>>> see below) just from D. >>>> >>>> b) You go first to B and call make. This makes A, if necessary, then B. >>>> The making of A is exactly the thing that you want to prevent with the >>>> '--from' option. Go into D and call 'make --from B'. A is not built. >>>> >>>> c) After the discussion with you I am not sure if we still need --from >>>> because the two reasons I know for its existence my not be relevant with >>>> the new approach. >>>> >>>> c1) With the '--from' option you can tweak the dependency rules at >>>> >>> runtime >>> >>>> (a bit). This allows you to exclude projects from being built when you >>>> know that that is not necessary. But from experience I know that can >>>> >>> lead >>> >>>> to very subtle errors. Letting the system determine what to built is >>>> usually more reliable. >>>> >>>> c2) With build.pl a 'build --all' still builds every module on which >>>> the >>>> one you are standing in depends on. When those modules have been built >>>> previously, then no compilation takes place. But calling dmake for a >>>> couple of directories for close to 200 modules (when you stand in >>>> instsetoo_native) takes a lot of time (several minutes on Windows), even >>>> when no file has to be compiled. This wasteful way of doing nothing can >>>> >>> be >>> >>>> prevented with the --from option. >>>> However, with the new approach and its .done flag files you can >>>> determine >>>> which modules need to be built much faster. You don't have to call >>>> dmake >>>> on directories that where already built. Hm, but this again, does only >>>> work if your .done files have dependencies on all relevant source files. >>>> That is something that is missing at least from my script. >>>> >>>> >>>> So, reasons for the existence of '--from' are a result of old/slow >>>> computers, slow files systems (still valid on Windows), missing global >>>> dependencies (which we now have for gbuild) and impatient developers. >>>> >>> >>> Looking at what Jan has proposed and this discussion so far, I think we >>> should make some attempt at moving in this direction -- using standard >>> makefiles. >>> >> > Yes, agreed. > > > Given this discussion, I do understand the additional flexibility of >>> build.pl, but I also wonder if the flexibility is worth it considering >>> the >>> IDE tools new developers are likely to be familiar with. >>> >>> A complete build will take a while no matter what is used, depending on >>> your situation. >>> >> > A complete build is the easiest of the tasks of a build system. > The hard part is to handle the development stage where single file changes > might require whole modules to be rebuilt.
this is for sure correct. A complete build would be a simple script. > > > I think this proposal has a lot of merit. >>> >> > I agree. > > > Thx for the kind words. >> >> I have updated the wiki page, to reflect the specific make calls. >> >> and R1533869 shows my first version of the central Makefile >> >> https://svn.apache.org/repos/**asf/openoffice/branches/** >> capstone2013/main/Makefile<https://svn.apache.org/repos/asf/openoffice/branches/capstone2013/main/Makefile> >> >> you will see it contains the intermodule dependencies and nothing more. >> The >> setup in every module is kept in a Makefile in the respective module, so >> we >> have a clear seperation of module and intermodule dependencies. >> > > The gbuild modules already have files called 'Makefile' top-level. Are you > going to reorganize this and the prj/ and util/ directories? > I have the following plan: a) rename all Makefile to Makefile.gb, and correct prj/makefile.mk to the new name b) for each <module> add a new Makefile, that basically call the existing makefile.mk, as per your script (and build.lst) and update build.lst c) add deliver.lst to Makefile and update build.lst once done, both build.pl and main/Makefile will work in the branch. d) have all of you test, and critic the idea. e) remove */prj and build.pl f) integrate in trunk. rgds jan I. > > -Andre > > > >> hope you like it. >> rgds >> jan I. >> >> >> >> >>> >>> >>> >>> >>>> While this is easy to do with eg Perl I am not sure how to handle >>>>> this >>>>> >>>>>> with just a Makefile. The straightforward approach with handling >>>>>> <module>.done files does not work. And that is one of the reasons why >>>>>> >>>>> I >>> >>>> don't think that (GNU) makefiles are a good solution for any problem. >>>>>> Most >>>>>> of us are used to program object oriented/imperative. Makefiles >>>>>> >>>>> require >>> >>>> a >>>>>> declarative approach. Maybe the use of Perl is not such a bad idea. >>>>>> Maybe >>>>>> it would be better to reimplement build.pl with a lot fewer options >>>>>> >>>>> and >>> >>>> with better readable code. >>>>>> >>>>>> I agree that makefiles are nowhere near a good solution to many of >>>>>> >>>>> these >>> >>>> problems, but its like windows, I dont like it, but everybody uses it. >>>>> >>>>> We could easily write a new build.pl, that also took care of the local >>>>> makefiles, but our build system would not be in the mainstream, and >>>>> e.g. >>>>> the distros would not like to integrate AOO. >>>>> >>>>> I have over the last years followed research in building systems, and >>>>> there >>>>> are (sadly enough) nobody that tries a real object oriented aproach. >>>>> >>>> Also >>> >>>> if you look at packages like visual studio, QT, eclipse they all use the >>>>> principle of makefiles with declarative approach. >>>>> >>>>> So my simple question is, do we want to approach the main road >>>>> >>>> (makefiles >>> >>>> for unix, visual studio for windows/mac) or do we want to have better >>>>> >>>> but >>> >>>> non standard system. >>>>> >>>>> Good analysis. Maybe I should answer with Faust: "Zwei Seelen wohnen, >>>> ach! in meiner Brust" (two souls alas! are dwelling in my breast). The >>>> pragmatist says to use the make. It is good enough for others, it is >>>> >>> good >>> >>>> enough for use. >>>> On the other hand, when I start a new project I usually start with the >>>> question of what are the best tools for the problem at hand. And make >>>> >>> does >>> >>>> not seem to be the first or the best answer. Look at our dmake or >>>> gbuild >>>> system. Both don't use make in a standard way. gmake even defines its >>>> >>> own >>> >>>> language, object oriented and imperative, on top of the GNU make macros. >>>> That is, for me, an act of desperation. >>>> I have made experiments with an alternative approach, a domain specific >>>> language somewhat similar to Java. I personally like this approach >>>> >>> because >>> >>>> a) it uses the paradigm that I already use when writing C++ code. That >>>> means that I can apply my existing knowledge to the build process and >>>> >>> that >>> >>>> I don't have to remember all the tricks and pitfalls of makefiles. >>>> b) as expected it was much easier to handle file dependencies and >>>> >>> parallel >>> >>>> processing of build jobs in Java than adding object orientation and >>>> imperative control flow to makefiles. >>>> >>>> If I had the time and if I would be the one working on it then I would >>>> prefer an non-Makefile approach. But maybe I am just suffering too much >>>> from one of the 'three great virtues of a programmer': hubris. >>>> >>>> You are the one who leads the build project changes, so you have to >>>> >>> decide >>> >>>> which approach to use. I am not trying to make your life harder (than >>>> necessary), I am only trying to point out some of the pitfalls that I >>>> >>> have >>> >>>> encountered in the past. And to prevent you from removing features that >>>> >>> I >>> >>>> use :-) >>>> >>>> >>>> >>>> >>>> rgds >>>>> jan I. >>>>> >>>>> Ps. its always refreshing to discuss with you, you often have an >>>>> alternative approach, which is not just a dream but doable. >>>>> >>>>> Thanks. That makes two of us. >>>> >>>> Have a nice weekend, >>>> Andre >>>> >>>> >>>> >>>> -Andre >>>>> >>>>>> >>>>>> --prepare >>>>>> >>>>>>> Also one option that is important for our every day work. Use >>>>>>>> case: >>>>>>>> You make changes in <module> and are not sure if these changes are >>>>>>>> compatible/incompatible. To be on the safe side you discard the >>>>>>>> >>>>>>> output >>> >>>> of >>>>>>>> all depending modules. To save time you keep the output of all >>>>>>>> other >>>>>>>> modules. >>>>>>>> >>>>>>>> Often used together with '--from' like 'make --prepare --from >>>>>>>> svx' to >>>>>>>> prepare a build after making changes in svx. >>>>>>>> >>>>>>>> Documentation changed, funny thing is that svx does not clear >>>>>>>> correctly >>>>>>>> >>>>>>>> on >>>>>>> my ubuntu build. >>>>>>> >>>>>>> >>>>>>> --since <module> >>>>>>> >>>>>>> A variant of '--from'. The only difference is that <module> >>>>>>>> itself >>>>>>>> is >>>>>>>> not built. >>>>>>>> >>>>>>>> If your proposed approach is similar to what my script >>>>>>>> produces >>>>>>>> then >>>>>>>> it >>>>>>>> is not too difficult to support --from/--since. I made some >>>>>>>> experiments >>>>>>>> in >>>>>>>> this direction but was to lazy to finish them. >>>>>>>> >>>>>>>> My approach is very similar, but I failed to see how --since is >>>>>>>> >>>>>>>> supported. >>>>>>> And question is if its real important. >>>>>>> >>>>>>> >>>>>>> --job >>>>>>> >>>>>>> --pre_job >>>>>>>> --post_job >>>>>>>> These are sometimes handy to run a non-standard command for all >>>>>>>> modules. >>>>>>>> >>>>>>>> I have added them, they are by the way a good example why we >>>>>>>> need a >>>>>>>> >>>>>>>> discussion I have never used them. >>>>>>> >>>>>>> However maybe the real discussion is "do we want to replace build and >>>>>>> have >>>>>>> a main/Makefile instead?" >>>>>>> >>>>>>> >>>>>>> >>>>>>> - I have not used the rest of the unsupported options and would >>>>>>> not >>>>>>> miss >>>>>>> >>>>>>> them. Others may have other sets of options that are important to >>>>>>>> them. >>>>>>>> >>>>>>>> >>>>>>>> Some general remarks: >>>>>>>> >>>>>>>> - Why keep one makefile per module? Why not put all the >>>>>>>> inter-module >>>>>>>> dependencies into one file (like my script does)? >>>>>>>> >>>>>>>> Ups, I did not explain that correctly, I propose 1 Makefile >>>>>>>> >>>>>>>> "main/Makefile" >>>>>>> with all inter-module and 1 Makefile "<module>/Makefile" that today >>>>>>> >>>>>> just >>> >>>> will call the old makefiles as described in prj/build.lst >>>>>>> >>>>>>> - Why not use the oportunity to move (a part of) the build >>>>>>> environment >>>>>>> out >>>>>>> >>>>>>> of the way to, say, build/ ? >>>>>>> >>>>>>>> You have guessed my next step. >>>>>>>> >>>>>>>> - How are dependencies between modules handled (just the manual >>>>>>> >>>>>>> dependencies from prj/build.lst or also the file dependencies >>>>>>>> introduced >>>>>>>> by >>>>>>>> gmake). >>>>>>>> >>>>>>>> See doc. on --from. Its done with <module>.done files >>>>>>>> >>>>>>>> - How is the output of the individual calls to dmake or GNU make >>>>>>> >>>>>>> handled/made accessible. Ie. if there is a build error, how can I >>>>>>>> >>>>>>> look >>> >>>> up >>>>>>>> the corresponding build output? >>>>>>>> >>>>>>>> see doc. script make_log >>>>>>>> >>>>>>>> - Are the gmake makefiles included (run in the same process) or >>>>>>> is >>>>>>> >>>>>> GNU >>> >>>> make started for them it its own process? >>>>>>>> >>>>>>>> For a start they would be called (own process), but its something >>>>>>>> where >>>>>>>> >>>>>>>> I >>>>>>> have no strong opinions. >>>>>>> >>>>>>> Please (just to be sure), this proposal has nothing to do with the >>>>>>> students >>>>>>> work, its simply because I saw a positive discussion on removing >>>>>>> build.pl >>>>>>> , >>>>>>> and spent a couple of hours looking at it. If there is a preference >>>>>>> >>>>>> not >>> >>>> to >>>>>>> remove build.pl I will simply forget it. >>>>>>> >>>>>>> rgds >>>>>>> jan I. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>>> Andre >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> It has been roughly tested it, thanks to a clever utility from >>>>>>>> >>>>>>> andre. >>> >>>> As discussed build.pl contains a lot of options, which need to be >>>>>>>>> considered in a makefile. >>>>>>>>> >>>>>>>>> My suggestion is on >>>>>>>>> http://wiki.openoffice.org/********wiki/Build_System_Analysis:****<http://wiki.openoffice.org/******wiki/Build_System_Analysis:**> >>>>>>>>> < >>>>>>>>> >>>>>>>> http://wiki.openoffice.org/******wiki/Build_System_Analysis:**<http://wiki.openoffice.org/****wiki/Build_System_Analysis:**> >>> **> >>> >>>> **<http://wiki.openoffice.org/******wiki/Build_System_**Analysis:**<http://wiki.openoffice.org/****wiki/Build_System_Analysis:**> >>>>>>>>> < >>>>>>>>> >>>>>>>> http://wiki.openoffice.org/****wiki/Build_System_Analysis:**<http://wiki.openoffice.org/**wiki/Build_System_Analysis:**> >>> > >>> >>>> build.pl_versus_makefile<http:******//wiki.openoffice.org/**wiki/****<http://wiki.openoffice.org/wiki/****> >>>>>>>>> < >>>>>>>>> >>>>>>>> http://wiki.openoffice.org/**wiki/**<http://wiki.openoffice.org/wiki/**> >>> > >>> >>>> Build_System_Analysis:build.******pl_versus_makefile<http://** >>>>>>>>> wiki.openoffice.org/wiki/****Build_System_Analysis:build.**<http://wiki.openoffice.org/wiki/**Build_System_Analysis:build.**> >>>>>>>>> pl_versus_makefile< >>>>>>>>> >>>>>>>> http://wiki.openoffice.org/**wiki/Build_System_Analysis:** >>> build.pl_versus_makefile<http://wiki.openoffice.org/wiki/Build_System_Analysis:build.pl_versus_makefile> >>> >>>> Please feel free to edit/comment on the page. I have reduced to >>>>>>>>> options >>>>>>>>> a >>>>>>>>> lot, and some of them might be in use. >>>>>>>>> >>>>>>>>> thanks in advance for your comments. >>>>>>>>> >>>>>>>>> >>>>>>>>> ------------------------------********------------------------* >>>>>>>>> *--** >>>>>>>>> --** >>>>>>>>> >>>>>>>>> --**--------- >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.******a** >>>>>>>> pache.org< >>>>>>>> http://apache.org**> >>>>>>>> <dev-unsubscribe@**openoffice.****apache.org< >>>>>>>> >>>>>>> http://openoffice.apache.org> >>> >>>> <dev-unsubscribe@**openoffice.**apache.org<http://openoffice.apache.org> >>>>>>>> < >>>>>>>> >>>>>>> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >>> > >>> >>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------******--------------------------** >>>>>>>> --** >>>>>>>> >>>>>>> --**--------- >>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.****a**pache.org< >>>>>> >>>>> http://apache.org> >>> >>>> <dev-unsubscribe@**openoffice.**apache.org<http://openoffice.apache.org> >>>>>> < >>>>>> >>>>> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >>> > >>> >>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------****----------------------------** >>>> --**--------- >>>> To unsubscribe, e-mail: >>>> dev-unsubscribe@openoffice.**a**pache.org<http://apache.org> >>>> < >>>> >>> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >>> > >>> >>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>> >>>> >>>> >>> -- >>> >>> ------------------------------**------------------------------** >>> ------------------------------**------- >>> MzK >>> >>> "Truth is stranger than fiction, but it is because Fiction is obliged >>> to stick to possibilities. Truth isn't." >>> -- "Following the Equator", Mark Twain >>> >>> > > ------------------------------**------------------------------**--------- > To unsubscribe, e-mail: > dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> > > For additional commands, e-mail: dev-h...@openoffice.apache.org > >
