Ilya, Thanks for your inputs. The reason why we decided to split Ignite into several packages mimics the reason why Java community introduced modular subsystem for JDK. That's all about size. Ignite distribution is too big, and we're trying to separate it into several components so that people can install only the features they need.
The point of a package is to ship something into root file system that can > be used from root file system. If cpp files require compilation we should > not ship them, or ship them to 'examples'. Ditto with benchmarks. If > there's no mechanism to add optional libs to Ignite classpath, we should > not ship optional libs. Moreover, some of 'optional' modules such as yarn > don't make sense here because they're not supposed to be used with > standalone Ignite. Agree that we need to ship the code that is ready to be run. As for the classpath thing, if an optional package is installed into the root (core) package directory, then its jars have to be added to "ignite/libs" folder. After that, the one needs to restart a cluster node, nd it will add the just installed optional libs to the classpath. *Petr*, does it work this way or can be implemented this way to address Ilya's concerns? -- Denis On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev <ilya.kasnach...@gmail.com> wrote: > 2018-04-13 7:42 GMT+03:00 Peter Ivanov <mr.wei...@gmail.com>: > > > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev <ilya.kasnach...@gmail.com > > > > wrote: > > > > > > > > Moreover I did not find a way to start service if default installed JVM > > is > > > Java 7 :( I understand it's EOL, still this is something that hit me. > > > > > > Apache Ignite >=2.4 does not support Java 7 - it is said in > documentation, > > DEVNOTES and even in startup scripts. > > > > I have Java 8 too, but I could not get service from package to start > Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that > matter). Is it possible to specify it while running packaged Ignite? > > > > > > > > > > apache-ignite-libs is a totally unexpected package name. apache-ignite > > core > > > doesn't depend on it. It doesn't enable anything out of the box. The > > > package is huge. > > > > ‘apache-ignite-libs’ is an aggregation package (for now) for all optional > > libs we are delivering. Possibly later they will be split more granular > or > > even package per lib (like php, perl, python, etc. do for their libs). > > This package dependency on ‘apache-ignite-core’ may seem confusing > though, > > I will try to explain it in IEP at least for current iteration. > > > > Okay, but how do you add optional libs to be included into Ignite classpath > while being launched by service? Is it even possible? If it isn't, I think > it doesn't make sense to ship apache-ignite-libs at all. > > > > > > Further naming may become clear when we’ll start initiative on including > > packages to popular Linux distributions and theirs community will join > > naming discussions. > > > Renaming packages once they're deployed widely will be a pain point to out > users. Some things should probably be thought out in advance. > > > > > > > > > > > > > > Frankly speaking, I'm not sure that improvements over Stage I are > enough > > as > > > of now. For demo-like activity, we can probably go with one package > fits > > > all. > > > > > > > The process of finding the best package architecture is iterative, but > > previously community agreed in split design proposed for 2.5 release. > > > > Also, split architecture is half of proposed improvements. The other > half - > > new process for deploying packages to Bintray (with virtually indefinite > > storage capabilities). > > > I think we could drop the split for now, or at least drop > apache-ignite-libs package at all. Probably also drop apache-ignite-cpp > package and maybe apache-ignite-benchmarks. > > The point of a package is to ship something into root file system that can > be used from root file system. If cpp files require compilation we should > not ship them, or ship them to 'examples'. Ditto with benchmarks. If > there's no mechanism to add optional libs to Ignite classpath, we should > not ship optional libs. Moreover, some of 'optional' modules such as yarn > don't make sense here because they're not supposed to be used with > standalone Ignite. > > IMO it is not right to try and shove every file from Ignite distribution > into some package. We should only put in packages things that can be used. > If something can't be used without copying it to a different FS location, > it should be in examples or not packaged at all. > > In my opinion, it doesn't make sense to implement an underwhelming package > split right now just because we have agreed to have *some* package split in > 2.5. Let's aim for happiness. > > > > > > > > > > > > > > -- > > > Ilya Kasnacheev > > > > > > > > > > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>: > > > > > > > If someone from PMCы or Committers still sees necessity about > including > > > > these tasks into Apache Ignite 2.5 release, this is the last chance > to > > do > > > > so. > > > > Otherwise this task will be moved to at 2.6 release at least, or even > > > > moved to backlog indefinitely. > > > > > > > > > > > > > > > > > On 9 Apr 2018, at 19:08, Petr Ivanov <mr.wei...@gmail.com> wrote: > > > > > > > > > > To top new RPM architecture off, update to release process is > > > introduced > > > > — [1] [2]. > > > > > > > > > > Both tasks (this one and IGNITE-7647) are ready for review and > should > > > be > > > > merged simultaneously. > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172 > > > > > [2] https://github.com/apache/ignite-release/pull/1 > > > > > > > > > > > > > > > > > > > > > > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev < > ilya.kasnach...@gmail.com > > > > > > > wrote: > > > > >> > > > > >> Hello! > > > > >> > > > > >> Let me share my idea of how this shoud work. Splitting package > into > > > > >> sub-packages should be dependency-driven. > > > > >> > > > > >> It means that all Ignite modules without dependencies or with > small > > > > >> dependencies (such as ignite-log4j) should be included in > > ignite-core. > > > > It > > > > >> doesn't make sense to make a zillion RPM packages. > > > > >> > > > > >> Critical things like ignite-spring and ignite-indexing should be > in > > > > >> ignite-core of course, even if they have dependencies. Ignite-core > > > > should > > > > >> be fully self-sufficient and feature-complete. > > > > >> > > > > >> However, e.g. .net API should probably be in a separate package, > > > > because it > > > > >> should depend on mono | net-core. We may also have ignite-devel > > > package > > > > >> which should include all modules which only make sense for > > developers > > > > who > > > > >> write code. Such as hibernate integration. > > > > >> > > > > >> I'm not sure about MR modules. The main question should be, does > it > > > have > > > > >> dependencies? Can it run stand-alone without writing code? > > > > >> > > > > >> Hope this helps, > > > > >> > > > > >> -- > > > > >> Ilya Kasnacheev > > > > >> > > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>: > > > > >> > > > > >>> Hi, Igniters! > > > > >>> > > > > >>> > > > > >>> Here are some news on our RPM packages initiative. > > > > >>> > > > > >>> 1. I’ve finished preliminary developing of Stage II version of > RPM > > > > >>> packages [1]. Main “new feature” is — split design. Also I’ve > added > > > > >>> package.sh script for automating package building process which > > will > > > > help > > > > >>> organise corresponding builds in TC as well as simplify process > for > > > > >>> developers who wishes to have custom packages. > > > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up with > > > > Apache > > > > >>> Ignite 2.5 release, I’d greatly appreciate your help in finding > > > > reviewer. > > > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB > [4] > > > > >>> repositories on Apache Bintray. Though they are already prepared > > for > > > > >>> hosting RPM and DEB packages respectively, and there is a way of > > > > linking > > > > >>> them to apache.org/dist/ignite page, there is possible > alternative > > > in > > > > >>> storing there only plain directory layout corresponding to each > > > > repository > > > > >>> type (RPM and DEB) and manage this layout (repodata, > distributions, > > > > >>> versions, etc.) by ourselves, having more control over > repositories > > > but > > > > >>> lacking some simplicity of deploying new releases. WDYT? Should > we > > > try > > > > >>> Cassandra approach? They are storing their DEB packages as I > > > described > > > > >>> above [5]. > > > > >>> > > > > >>> Also — a question arose while I was working on this issue: which > > OSes > > > > (and > > > > >>> which versions of each) are we going to support (if we are going) > > in > > > > terms > > > > >>> of step-by-step list? Currently RPM packages are tested only with > > > > latest > > > > >>> CentOS (and, respectively — RHEL), but there are a lot more > > RPM-based > > > > >>> distributives [6] some of which are more o less popular among OS > > > > community > > > > >>> (ALT, Fedora, openSUSE, etc.). > > > > >>> > > > > >>> > > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647 > > > > >>> [2] https://github.com/apache/ignite/pull/3703 > > > > >>> [3] https://bintray.com/apache/ignite-rpm > > > > >>> [4] https://bintray.com/apache/ignite-deb > > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/ > > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_ > > > > distributions > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <mr.wei...@gmail.com> > > wrote: > > > > >>>> > > > > >>>> I suppose that most everything if not all from libs/options will > > go > > > to > > > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs'). > > > > >>>> More precise lib selection (if something from optional would > > better > > > to > > > > >>> have in core package) will be discussed right after preliminary > > split > > > > >>> architecture agreement. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov <dpavlov....@gmail.com > > > > > > wrote: > > > > >>>>> > > > > >>>>> I like idea of keeping simple system of modules, so +1 from me. > > > > >>>>> > > > > >>>>> Where optional libs (e.g Direct IO plugin) would be included, > > would > > > > it > > > > >>> be > > > > >>>>> core or optional? > > > > >>>>> > > > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda <dma...@apache.org>: > > > > >>>>> > > > > >>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> How big would be a final core module? > > > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and > > create > > > > it’s > > > > >>> own > > > > >>>>>>> package. > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree with > > Petr > > > > and > > > > >>>>>> propose the simplest modular system: > > > > >>>>>> > > > > >>>>>> - core module that includes basic Ignite capabilities > including > > > SQL, > > > > >>>>>> compute grid, service grid, k/v > > > > >>>>>> - optional module hosts the rest - ML, streamers integration > > > (kafka, > > > > >>>>>> flink), kubernetes, etc. > > > > >>>>>> > > > > >>>>>> What do you think? > > > > >>>>>> > > > > >>>>>> -- > > > > >>>>>> Denis > > > > >>>>>> > > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov < > > > mr.wei...@gmail.com> > > > > >>> wrote: > > > > >>>>>> > > > > >>>>>>> *DEB package > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov <mr.wei...@gmail.com> > > > > wrote: > > > > >>>>>>>> > > > > >>>>>>>> Considering that DEV package for now is almost platform > > > > independent > > > > >>>>>> (its > > > > >>>>>>> a java application more or less), that package will work > almost > > > on > > > > any > > > > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian, > > > etc. > > > > >>>>>>>> The only restriction is existence of systemctl (systemd) > > service > > > > >>>>>> manager > > > > >>>>>>> — we are dependent on it. > > > > >>>>>>>> > > > > >>>>>>>> Thats why, for instance, our RPM repository is called simply > > > ‘rpm’ > > > > >>> and > > > > >>>>>>> package has no arch or dist suffix — it will work on CentOS, > > > RHEL, > > > > >>>>>> Fedora, > > > > >>>>>>> etc. with presence of aforementioned systemd. > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan < > > > > dsetrak...@apache.org> > > > > >>>>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>> Will Debian package work for Ubuntu? > > > > >>>>>>>>> > > > > >>>>>>>>> D. > > > > >>>>>>>>> > > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov < > > > > mr.wei...@gmail.com> > > > > >>>>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will move to > > > > official > > > > >>>>>>>>>> repositories, there can be a problem from OS community. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base for > DEB > > > > package > > > > >>>>>>> build > > > > >>>>>>>>>> (package layout / install scripts) for speeding up things > > and > > > > >>>>>> excluding > > > > >>>>>>>>>> possible duplication and desynchronisation, so its a > matter > > of > > > > ’sit > > > > >>>>>>> and do’ > > > > >>>>>>>>>> rather then some technical research. Thats why I rose > > > discussion > > > > >>>>>> about > > > > >>>>>>>>>> future package architecture, so that after agreement I'm > be > > > > able to > > > > >>>>>>> pack > > > > >>>>>>>>>> both RPM and DEB identically. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Yet, if you insist, I can create DEB package according to > > > > current > > > > >>> RPM > > > > >>>>>>>>>> layout in no time. > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan < > > > > >>> dsetrak...@apache.org> > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Peter, > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> I don't think the package size of 280M is going to be a > > > > problem at > > > > >>>>>>> all, > > > > >>>>>>>>>> but > > > > >>>>>>>>>>> what you are suggesting can be an improvement down the > > road. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> In the mean time, I think our top priority should be to > > > provide > > > > >>>>>>> packages > > > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly > > enough. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Agree? > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> D. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider < > > > mr.wei...@gmail.com> > > > > >>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> Hi, Igniters! > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of it, > > so > > > > I'd > > > > >>>>>> like > > > > >>>>>>> to > > > > >>>>>>>>>>>> move > > > > >>>>>>>>>>>> forward to further improve and widen AI delivery through > > > > >>> packages. > > > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package weighing > > about > > > > >>> 280M+ > > > > >>>>>>> and, > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>> improve usability and significantly reduce required > > download > > > > >>>>>> sizes, I > > > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted > delivery > > > as > > > > >>>>>>> follows: > > > > >>>>>>>>>>>> - CORE > > > > >>>>>>>>>>>> - bin > > > > >>>>>>>>>>>> - config > > > > >>>>>>>>>>>> - libs (!optional) > > > > >>>>>>>>>>>> - OPTIONAL LIBS > > > > >>>>>>>>>>>> - BENCHMARKS > > > > >>>>>>>>>>>> - DOCS (?) > > > > >>>>>>>>>>>> - EXAMPLES > > > > >>>>>>>>>>>> - .NET PLATFORM FILES > > > > >>>>>>>>>>>> - C++ PLATFORM FILES > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility (no > > > > reason > > > > >>> to > > > > >>>>>>>>>> download > > > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core > node > > > > >>>>>>> functionality) > > > > >>>>>>>>>>>> and > > > > >>>>>>>>>>>> maintainability (you are in full control of what is > > > installed > > > > on > > > > >>>>>> your > > > > >>>>>>>>>>>> system). > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> After successful architecture choice, same scheme are > > > planned > > > > to > > > > >>> be > > > > >>>>>>>>>> used in > > > > >>>>>>>>>>>> DEB packages as well. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> WDYT? > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> -- > > > > >>>>>>>>>>>> Sent from: > > > http://apache-ignite-developers.2346864.n4.nabble. > > > > >>> com/ > > > > >>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >>> > > > > > > > > > > > > > > > > > > >
