On 21 October 2013 10:50, Andre Fischer <awf....@gmail.com> wrote: > On 18.10.2013 19:54, janI wrote: > >> On 18 October 2013 16:52, Andre Fischer <awf....@gmail.com> wrote: >> >> On 18.10.2013 15:58, janI wrote: >>> >>> On 18 October 2013 15:00, Andre Fischer <awf....@gmail.com> wrote: >>>> >>>> On 18.10.2013 14:02, janI wrote: >>>> >>>>> sd >>>>> >>>>>> >>>>>> On 18 October 2013 13:36, Andre Fischer <awf....@gmail.com> wrote: >>>>>> >>>>>> On 18.10.2013 11:32, janI wrote: >>>>>> >>>>>> Hi. >>>>>>> >>>>>>> due to the discussion in thread "Mentor a new build system", I have >>>>>>>> made a >>>>>>>> proposal for a central Makefile located in main. >>>>>>>> >>>>>>>> Hi Jan, >>>>>>>> >>>>>>>> it is great that you are going to improve this part of the build >>>>>>> system. >>>>>>> But I think that we need more details about how the proposed >>>>>>> build >>>>>>> system >>>>>>> works. Without them I can not really evaluate the proposal. >>>>>>> >>>>>>> First of all, I agree with juergens remarks that this should be >>>>>>> >>>>>>> discussed >>>>>> before implemented, hence the wiki page. >>>>>> >>>>>> Secondly this has nothing directly to do with the proposed build >>>>>> system, >>>>>> its a simple replacement of build.pl in the current system. >>>>>> >>>>>> Yes, that is how I understood it. I just did not know how to call >>>>>> the >>>>>> >>>>> build.pl replacement. >>>>> >>>>> >>>>> >>>>> I know that build.pl works, but having a Makefile in main, would >>>>> make >>>>> >>>>>> us >>>>>> one step closer on being compatible with the distros. To me this job >>>>>> is >>>>>> a >>>>>> simple cleanup, not something we deadly need, but nice to have. >>>>>> >>>>>> >>>>>> Some remarks regarding the missing options: >>>>>> >>>>>> --from <module> >>>>>>> This is one of the more important options and one that I use >>>>>>> frequently >>>>>>> (also in the form --all:<module>). >>>>>>> Note that if you are in <moduleA> and call 'make --from >>>>>>> <moduleB>' >>>>>>> then >>>>>>> all modules are built >>>>>>> a) which <moduleA> depends on >>>>>>> b) but not those that <moduleB> depends on >>>>>>> c) Both <moduleA> and <moduleB> are built. >>>>>>> >>>>>>> I have changed the documentation. >>>>>>> >>>>>>> I use the --all:<module> myself very often, and have changed the >>>>>> documentation, because it is of course supported. >>>>>> >>>>>> The difference is that you do the call in main, but that is a minor >>>>>> detail >>>>>> that can be easily corrected (have <module>/Makefile calling >>>>>> main/Makefile. >>>>>> >>>>>> I have also changed documentation on --html due to juergens comments. >>>>>> >>>>>> I am not sure that we understand --from and --since in the same way >>>>>> so >>>>>> >>>>> I >>>>> will try to explain what I think they do. >>>>> >>>>> Let's imagine that we have a simple project with modules A, B, C, D and >>>>> E. >>>>> where B depends on A, C on B, D on C, and E on D. >>>>> A ' make all' would mean 'make E'. The dependencies would then lead to >>>>> building modules A, B, C, D, E in this order. >>>>> If I am in E and call 'make --from C' then only C, D, and E should be >>>>> built. A 'make --since C' would only build D and E. >>>>> >>>>> If I am in D and call 'make --from B' then modules B, C, and D are >>>>> built. >>>>> Call 'make --since B' to build only C and D. >>>>> Note that 'make --from' accepts more than one module name (while 'make >>>>> --all:<module>' does not). >>>>> Note also that in the above case (stand in D, call 'make --from B') >>>>> module >>>>> A is not built, regardless of whether there are changes in A or not. >>>>> Whereas a simple call to make (still standing in D) would build all >>>>> modules that D depends on, directly or indirectly. Thus the options >>>>> '--from' and '--since' exist to actively exclude modules from being >>>>> built. >>>>> >>>>> The whole thing becomes a little bit more complicated with multiple >>>>> options to '--from' (I never use '--since' and also don't know a valid >>>>> use >>>>> case so I will ignore it for now) and more complex dependencies then in >>>>> the >>>>> simple example above. Let's say that if we stand in instsetoo_native >>>>> and >>>>> call 'make --from svx sfx2'. Note that svx depends on sfx2. This >>>>> would >>>>> build svx, sfx2 and all modules that depend (directly or indirectly) on >>>>> svx >>>>> OR sfx2. >>>>> >>>>> got it, now I just have one problem, why would you not build the >>>>> >>>> dependent >>>> modules, if they needed to be built, thats a scenario I dont understand. >>>> With a central makefile, <module>/makefile will not be called so we do >>>> not >>>> waste cpu cycles. >>>> >>>> With the .done files, we know when a module was last built and all >>>> modules >>>> that depend it should be rebuilt which the rule >>>> <module>.done : <module_depend>.done >>>> >>>> will ensure, so If we have A -> B -> C -> D >>>> >>>> I go in B, and call make, then when I go in D and make, B,C,D will be >>>> made. >>>> >>>> If we have A -> B -> D C -> D >>>> and do the same then only D will be made. >>>> >>>> So --from is not really saving anything ? >>>> >>>> a) In your example you first go into B then, in a second step, into D. >>> The '--from' option lets you do the same (well, not really the same, >>> but >>> see below) just from D. >>> >>> b) You go first to B and call make. This makes A, if necessary, then B. >>> The making of A is exactly the thing that you want to prevent with the >>> '--from' option. Go into D and call 'make --from B'. A is not built. >>> >>> Actually I go to main and say "make D", that will cause B but not A to >> be >> built. >> > > I guess that calling 'make D' from main/ would be equivalent to call it > from instsetoo_native/. But the --from option also works from inside other > directories. It can then be used to build a 'range' of modules, ie. go > into module Y/ and call 'make --from X' and all modules from X/ to Y/ will > be built. A real life example for using this method could be: > > 1. Make some changes to documentation but not code in sfx2/. > 2. Make some changes in svx/. > 3. Then go to sw/ and call 'make --from svx' > > This prevents the changes in sfx2/ to trigger recompilation. It builds > only modules svx/ and everything that depends on it until sw/ is reached. > The costly recreation of install sets (especially when there are multiple > languages) is omitted. > > With .done files you could do this in main/ with > make done_before_sfx2 > make sw > The 'done_before_svx' could be implemented similar to the regular svx > target like this: > > done_before_svx : done_l10n ... done_libxslt > # nothing to be done for done_before... > > done_l10n : ... > touch <some-path>/l10n.done
yes this would be piece of cake to implement, more editing than thinking. > > > >> >> c) After the discussion with you I am not sure if we still need --from >>> because the two reasons I know for its existence my not be relevant with >>> the new approach. >>> >>> Thx, I think the discussion is important. I am always afraid (still >> feeling >> newbee) to miss something, and at the same time many other say "we are >> used >> to". I think the right thing to do is somewhere in the middle, and we can >> only reach it by discussing it. >> > > I am now working for about 13 years on OpenOffice but due to its size and > complexity I know only a small part (mostly Impress UI and slideshow). > Everywhere else I am a newbee, too. > And there is always the possibility to miss something. I think it is > important to prepare as good as possible but at some point you just have to > implement your changes/features and handle the consequences. That means > that there will be errors. But those can be fixed. Part of being a > developer in an open source project is to find this point when planning > stops and development starts and when development is finished to 'weather > the storm' by fixing bugs and handling criticism. And keeping the "storm" in a branch, helps against criticism. By the way, when we talk about development (excluding another theme) even though I often talk "loudly" I do not feel critized, I merely feel many of us have strong feelings and sometimes express them a bit bombastic. Actually this discussion is to me a perfect example of how open source developments works when its best. rgds jan I. > > >> I find discussion where e.g. you say "I need option X because" very >> fruitfull, because they are based on actual need, and not just "nice to >> have". >> >> >> c1) With the '--from' option you can tweak the dependency rules at >>> runtime >>> (a bit). This allows you to exclude projects from being built when you >>> know that that is not necessary. But from experience I know that can >>> lead >>> to very subtle errors. Letting the system determine what to built is >>> usually more reliable. >>> >>> you are a pro. and I am getting there, but all newcommers are not, and >> to >> attracts them we need to make it easy and fault tolerant ! >> > > Yes, absolutely. I am convinced that providing a system that does > everything the pros need is more easy then make the same system reliable, > fault tolerant and easy to use. > Therefore focusing on the last three is a good strategy. > > > >> I am sure, I dont need to tell you how to tweek .done file to achieve what >> you want :-) >> >> >> c2) With build.pl a 'build --all' still builds every module on which the >>> one you are standing in depends on. When those modules have been built >>> previously, then no compilation takes place. But calling dmake for a >>> couple of directories for close to 200 modules (when you stand in >>> instsetoo_native) takes a lot of time (several minutes on Windows), even >>> when no file has to be compiled. This wasteful way of doing nothing can >>> be >>> prevented with the --from option. >>> However, with the new approach and its .done flag files you can determine >>> which modules need to be built much faster. You don't have to call dmake >>> on directories that where already built. Hm, but this again, does only >>> work if your .done files have dependencies on all relevant source files. >>> That is something that is missing at least from my script. >>> >>> Hmmm, I had a different idea, let me try that on you. If someone >> changes a >> source file, but does not do a make, its not intented to be public. >> Meaning >> we do not depend on the source files, but the actual run of make. Is that >> a >> wrong idea ? >> > > I am not quite sure that I understand you. Let's look at an (abstract) > example. There is a file code.cxx in module X/. I can build the whole > module by calling 'make X' in main/. The question is, what happens on > 'make all'? Currently, build.pl uses a brute force method and goes into > every module, and in case of dmake modules, into every of their > directories, and calls 'make'. This takes a couple of minutes (on > Windows) when no files has been changed. That is why the --from option, > and others like it, where created. > The alternative is to use dependencies to let 'make' figure out what to > build. These dependencies do not yet exist. There are dependencies > between files, even when they are in different modules. So, when I change > a file in sfx2/ and then build sw/ then some (or many) files in sw/ will be > compiled. But there are no dependencies that tell us that sw/ has to be > built. For this we need dependencies of .done files on the source file. > When a source file changes then these dependencies will trigger the > corresponding .done file to be made. This would then include building the > module and touching the .done file. > > But that would create a lot of dependencies, at least one for each of our > source files. And they all have to be loaded whenever 'make' is called. > Loading the dependencies might take a few seconds. Determining the time > stamps of all files involved might take longer. But gbuild when used > Office wide would introduce more dependencies. But we still have to find > out if the time it takes is acceptable when spend on every call of 'make' > or if we need some shortcuts for the daily work. > > > >> >> >>> So, reasons for the existence of '--from' are a result of old/slow >>> computers, slow files systems (still valid on Windows), missing global >>> dependencies (which we now have for gbuild) and impatient developers. >>> >>> >>> >>> While this is easy to do with eg Perl I am not sure how to handle this >>>> >>>>> with just a Makefile. The straightforward approach with handling >>>>> <module>.done files does not work. And that is one of the reasons why >>>>> I >>>>> don't think that (GNU) makefiles are a good solution for any problem. >>>>> Most >>>>> of us are used to program object oriented/imperative. Makefiles >>>>> require >>>>> a >>>>> declarative approach. Maybe the use of Perl is not such a bad idea. >>>>> Maybe >>>>> it would be better to reimplement build.pl with a lot fewer options >>>>> and >>>>> with better readable code. >>>>> >>>>> I agree that makefiles are nowhere near a good solution to many of >>>>> these >>>>> >>>> problems, but its like windows, I dont like it, but everybody uses it. >>>> >>>> We could easily write a new build.pl, that also took care of the local >>>> makefiles, but our build system would not be in the mainstream, and e.g. >>>> the distros would not like to integrate AOO. >>>> >>>> I have over the last years followed research in building systems, and >>>> there >>>> are (sadly enough) nobody that tries a real object oriented aproach. >>>> Also >>>> if you look at packages like visual studio, QT, eclipse they all use the >>>> principle of makefiles with declarative approach. >>>> >>>> So my simple question is, do we want to approach the main road >>>> (makefiles >>>> for unix, visual studio for windows/mac) or do we want to have better >>>> but >>>> non standard system. >>>> >>>> Good analysis. Maybe I should answer with Faust: "Zwei Seelen wohnen, >>> ach! in meiner Brust" (two souls alas! are dwelling in my breast). The >>> pragmatist says to use the make. It is good enough for others, it is >>> good >>> enough for use. >>> On the other hand, when I start a new project I usually start with the >>> question of what are the best tools for the problem at hand. And make >>> does >>> not seem to be the first or the best answer. Look at our dmake or gbuild >>> system. Both don't use make in a standard way. gmake even defines its >>> own >>> language, object oriented and imperative, on top of the GNU make macros. >>> That is, for me, an act of desperation. >>> >>> I just wonder did you see "gütterdämmerungen" (I hope I spelled it >> right, >> > almost, it is Götterdämmerung > > its 2 years since I lived in Vienna so its a bit rusty), I saw it on >> danube, 2 nights in a row. >> > I have not yet have much exposure to Wagner. But seeing this in Vienna on > the Danube must have been an unforgettable spectacle. > > > >> >> I have made experiments with an alternative approach, a domain specific >>> language somewhat similar to Java. I personally like this approach >>> because >>> a) it uses the paradigm that I already use when writing C++ code. That >>> means that I can apply my existing knowledge to the build process and >>> that >>> I don't have to remember all the tricks and pitfalls of makefiles. >>> b) as expected it was much easier to handle file dependencies and >>> parallel >>> processing of build jobs in Java than adding object orientation and >>> imperative control flow to makefiles. >>> >>> I like your approach, and would like to help on that, BUT I think we >> benefit as a community to stay on the main road. >> > > Good point. > > > >> >> If I had the time and if I would be the one working on it then I would >>> prefer an non-Makefile approach. But maybe I am just suffering too much >>> from one of the 'three great virtues of a programmer': hubris. >>> >>> No you are like me, just not understaning why the world isnt perfect :-) >> > > :-) > > >> >> You are the one who leads the build project changes, so you have to >>> decide >>> which approach to use. I am not trying to make your life harder (than >>> necessary), I am only trying to point out some of the pitfalls that I >>> have >>> encountered in the past. And to prevent you from removing features that >>> I >>> use :-) >>> >> >> I do not decide things, I think about solutions, throw them at the >> community, and see if the float. If they float I help implement them. >> >> I am a doer, but also a listener. Implementing what I think is right, >> would >> be dead wrong, but I pave the way and make us discuss/think sometimes by >> provoking a bit, which I think is good. >> > > A good approach in general, but I don't think that it would be wrong when > you implemented what you think is right, if you do it after a discussion. > > -Andre > > >> >> >>> >>> >>> rgds >>>> jan I. >>>> >>>> Ps. its always refreshing to discuss with you, you often have an >>>> alternative approach, which is not just a dream but doable. >>>> >>>> Thanks. That makes two of us. >>> >>> Have a nice weekend, >>> >>> same to you ! >> >> rgds >> jan I. >> >> >> Andre >>> >>> >>> -Andre >>>> >>>>> >>>>> --prepare >>>>> >>>>>> Also one option that is important for our every day work. Use >>>>>>> case: >>>>>>> You make changes in <module> and are not sure if these changes are >>>>>>> compatible/incompatible. To be on the safe side you discard the >>>>>>> output >>>>>>> of >>>>>>> all depending modules. To save time you keep the output of all other >>>>>>> modules. >>>>>>> >>>>>>> Often used together with '--from' like 'make --prepare --from >>>>>>> svx' to >>>>>>> prepare a build after making changes in svx. >>>>>>> >>>>>>> Documentation changed, funny thing is that svx does not clear >>>>>>> correctly >>>>>>> >>>>>>> on >>>>>> my ubuntu build. >>>>>> >>>>>> >>>>>> --since <module> >>>>>> >>>>>> A variant of '--from'. The only difference is that <module> >>>>>>> itself >>>>>>> is >>>>>>> not built. >>>>>>> >>>>>>> If your proposed approach is similar to what my script produces >>>>>>> then >>>>>>> it >>>>>>> is not too difficult to support --from/--since. I made some >>>>>>> experiments >>>>>>> in >>>>>>> this direction but was to lazy to finish them. >>>>>>> >>>>>>> My approach is very similar, but I failed to see how --since is >>>>>>> >>>>>>> supported. >>>>>> And question is if its real important. >>>>>> >>>>>> >>>>>> --job >>>>>> >>>>>> --pre_job >>>>>>> --post_job >>>>>>> These are sometimes handy to run a non-standard command for all >>>>>>> modules. >>>>>>> >>>>>>> I have added them, they are by the way a good example why we need >>>>>>> a >>>>>>> >>>>>>> discussion I have never used them. >>>>>> >>>>>> However maybe the real discussion is "do we want to replace build and >>>>>> have >>>>>> a main/Makefile instead?" >>>>>> >>>>>> >>>>>> >>>>>> - I have not used the rest of the unsupported options and would not >>>>>> miss >>>>>> >>>>>> them. Others may have other sets of options that are important to >>>>>>> them. >>>>>>> >>>>>>> >>>>>>> Some general remarks: >>>>>>> >>>>>>> - Why keep one makefile per module? Why not put all the inter-module >>>>>>> dependencies into one file (like my script does)? >>>>>>> >>>>>>> Ups, I did not explain that correctly, I propose 1 Makefile >>>>>>> >>>>>>> "main/Makefile" >>>>>> with all inter-module and 1 Makefile "<module>/Makefile" that today >>>>>> just >>>>>> will call the old makefiles as described in prj/build.lst >>>>>> >>>>>> - Why not use the oportunity to move (a part of) the build environment >>>>>> out >>>>>> >>>>>> of the way to, say, build/ ? >>>>>> >>>>>>> You have guessed my next step. >>>>>>> >>>>>>> - How are dependencies between modules handled (just the manual >>>>>> >>>>>> dependencies from prj/build.lst or also the file dependencies >>>>>>> introduced >>>>>>> by >>>>>>> gmake). >>>>>>> >>>>>>> See doc. on --from. Its done with <module>.done files >>>>>>> >>>>>>> - How is the output of the individual calls to dmake or GNU make >>>>>> >>>>>> handled/made accessible. Ie. if there is a build error, how can I >>>>>>> look >>>>>>> up >>>>>>> the corresponding build output? >>>>>>> >>>>>>> see doc. script make_log >>>>>>> >>>>>>> - Are the gmake makefiles included (run in the same process) or >>>>>> is GNU >>>>>> >>>>>> make started for them it its own process? >>>>>>> >>>>>>> For a start they would be called (own process), but its something >>>>>>> where >>>>>>> >>>>>>> I >>>>>> have no strong opinions. >>>>>> >>>>>> Please (just to be sure), this proposal has nothing to do with the >>>>>> students >>>>>> work, its simply because I saw a positive discussion on removing >>>>>> build.pl >>>>>> , >>>>>> and spent a couple of hours looking at it. If there is a preference >>>>>> not >>>>>> to >>>>>> remove build.pl I will simply forget it. >>>>>> >>>>>> rgds >>>>>> jan I. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>>> Andre >>>>>>> >>>>>>> >>>>>>> >>>>>>> It has been roughly tested it, thanks to a clever utility from >>>>>>> andre. >>>>>>> >>>>>>> As discussed build.pl contains a lot of options, which need to be >>>>>>>> considered in a makefile. >>>>>>>> >>>>>>>> My suggestion is on >>>>>>>> http://wiki.openoffice.org/********wiki/Build_System_Analysis:****<http://wiki.openoffice.org/******wiki/Build_System_Analysis:**> >>>>>>>> <http://wiki.openoffice.org/******wiki/Build_System_**Analysis:**<http://wiki.openoffice.org/****wiki/Build_System_Analysis:**> >>>>>>>> > >>>>>>>> **<http://wiki.openoffice.org/******wiki/Build_System_**Analysis:**<http://wiki.openoffice.org/****wiki/Build_System_Analysis:**> >>>>>>>> <http://wiki.**openoffice.org/**wiki/Build_**System_Analysis:**<http://wiki.openoffice.org/**wiki/Build_System_Analysis:**> >>>>>>>> > >>>>>>>> build.pl_versus_makefile<http:******//wiki.openoffice.org/** >>>>>>>> wiki/**** <http://wiki.openoffice.org/wiki/****><http://wiki.** >>>>>>>> openoffice.org/wiki/** <http://wiki.openoffice.org/wiki/**>> >>>>>>>> Build_System_Analysis:build.******pl_versus_makefile<http://** >>>>>>>> wiki.openoffice.org/wiki/****Build_System_Analysis:build.**<http://wiki.openoffice.org/wiki/**Build_System_Analysis:build.**> >>>>>>>> pl_versus_makefile<http://**wiki.openoffice.org/wiki/** >>>>>>>> Build_System_Analysis:build.**pl_versus_makefile<http://wiki.openoffice.org/wiki/Build_System_Analysis:build.pl_versus_makefile> >>>>>>>> > >>>>>>>> >>>>>>>> Please feel free to edit/comment on the page. I have reduced to >>>>>>>> options >>>>>>>> a >>>>>>>> lot, and some of them might be in use. >>>>>>>> >>>>>>>> thanks in advance for your comments. >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------********------------------------** >>>>>>>> --** >>>>>>>> --** >>>>>>>> >>>>>>>> --**--------- >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.******a** >>>>>>> pache.org< >>>>>>> http://apache.org**> >>>>>>> <dev-unsubscribe@**openoffice.****apache.org<http://** >>>>>>> openoffice.apache.org <http://openoffice.apache.org>> >>>>>>> <dev-unsubscribe@**openoffice.**apache.org<http://openoffice.apache.org> >>>>>>> <dev-unsubscribe@**openoffice.apache.org<dev-unsubscr...@openoffice.apache.org> >>>>>>> > >>>>>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------******--------------------------** >>>>>>> --** >>>>>>> >>>>>> --**--------- >>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.****a**pache.org< >>>>> http://apache.org**> >>>>> <dev-unsubscribe@**openoffice.**apache.org<http://openoffice.apache.org> >>>>> <dev-unsubscribe@**openoffice.apache.org<dev-unsubscr...@openoffice.apache.org> >>>>> > >>>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>>> >>>>> >>>>> >>>>> ------------------------------****----------------------------** >>> --**--------- >>> To unsubscribe, e-mail: >>> dev-unsubscribe@openoffice.**a**pache.org<http://apache.org> >>> <dev-unsubscribe@**openoffice.apache.org<dev-unsubscr...@openoffice.apache.org> >>> > >>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>> >>> >>> > > ------------------------------**------------------------------**--------- > To unsubscribe, e-mail: > dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> > For additional commands, e-mail: dev-h...@openoffice.apache.org > >
