Hi Nikolay, Thanks for your feedback! Absolutely, the IEP is currently in the draft mode - we will be adding more details going forward. Also, Petr mentioned that he wants to create a prototype which I think will make things clearer for all of us.
As for "download jars from the Internet on a production server", that's not necessarily what we propose. In case of Maven, artifacts are downloaded from a specified repository, which doesn't have to be on the Internet. Companies that have security concerns typically have a mirror/proxy containing approved artifacts only, already checked for viruses, etc. Production servers use the proxy rather than public repositories. The concerns that you brought up are certainly valid. However, I believe that using Maven actually addresses them, because Maven (as any other package/dependency manager for that matter) provides the corresponding functionality out of the box. On the contrary, users who currently download our ZIP package are *forced* to download modules and dependencies that they do not even need and will not use. In my experience, this is actually a bigger source of concern. Anyway, let us add more details to the IEP, and we will go from there. P.S. I'm also not sure I agree with your assessment of the embedded mode and thick clients, but that's probably a different discussion. Please feel free to create a new thread if you would like to discuss this. -Val On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <nizhi...@apache.org> wrote: > Also, guys. > > Can we make one step back and add some requirements to the IEP. > How we want to install and upgrade Ignite? > What specific issues we want to solve? > > With this input we can be on the same page during solution discussion. > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <nizhikov....@gmail.com> > написал(а): > > > > Hello, Val. > > > > Sorry, but I sill don’t understand how «download jars from the internet > on production server» approach helps us to solve the issues you describe. > > > > Moreover, with this enhancement you want to add more dependencies to > Ignite and create some security vulnerabilities: > > * Maven tool - new dependency. I doubt that and of the production > server have it > > * Internet maven repository - many production servers restrict > outgoing internet connection to any addresses. > > It’s hard to understand why distributed DB should have the > ability to connect to some kind of address on the Internet. > > * All Ignite distributions can be compromised with the hack of > some random maven repo or with the malicious proxy. > > * Note, that Ignite installation or upgrade procedure can become > dead just because of some Maven repo down. > > > > > > I have 2 concerns: > > > > 1. We shouldn’t download any jars from the internet during production > deployment or upgrade procedure. > > 2. The IEP description, for now, is quite small for such important > things as a way we distribute and upgrade Ignite. > > Petr, can you, please, make IEP more specific. > > I think we should add the following details: > > * Description of the typical Ignite deployment procedure > including folder structure(config, persistence, .m2 and other folders) > > * This also should include examples of the bash > commands if any required. > > * Description of the typical upgrade(downgrade?) procedure > > * This also should include examples of the bash > commands if any required. > > * Description of «single tool responsible for all > operations». > > * all commands > > * all operations > > * all parameters > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively > forces (or at least motivates) users to put everything related to Ignite > under IGNITE_HOME (configs, additional JARs, etc.). > > > > AFAIU this issue doesn’t relate to the way Ignite distributed. > > We can and should fix this outside of this IEP. > > > >> 2. Modules are enabled/disabled manually by moving folders around… this > is error-prone, > > > > Makes sense. > > Let’s fix this without «download jar from the internet»? > > We can have some kind of config with the enabled modules. > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if > you look at how this ZIP is built, you will notice that we don't actually > include all transitive dependencies, > > > > We need to fix this. > > Let’s include all the required JARs to the distribution. > > > >> which means that an embedded node that uses Maven can have a different > classpath. > > > > Embedded Ignite node is an error-prone design decision in the first > place. > > We can’t fix it with the «download jars from the internet» approach. > > We should eliminate the client node as a thing and use a thin clients > instead of it. > > > >> 4. ignite.sh allows manual modifications (there are several "Uncomment > this to..." lines). This complicates upgrades as well. > > > > Makes sense. > > Let’s fix this. > > > >> The whole purpose of the IEP is to fix the above, automate and simplify > operations for the users. The general idea is to create a single tool > responsible for all those operations and use Maven under the hood to > include/exclude modules. > > > > For now, IEP lacks the description of this tool. > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko < > valentin.kuliche...@gmail.com> написал(а): > >> > >> Hi Nikolay, > >> > >> Here are some of the issues that I see with the current binary ZIP > package > >> that we create for every release. > >> > >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively > >> forces (or at least motivates) users to put everything related to Ignite > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user > >> downloads and unzips a new version, they have to somehow merge the > content > >> together - that's poor upgradability. > >> 2. Modules are enabled/disabled manually by moving folders around. Not > only > >> this is error-prone, but it also adds more issues to upgradability - you > >> have to apply these steps for every version you download. > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if > you > >> look at how this ZIP is built, you will notice that we don't actually > >> include all transitive dependencies, which means that an embedded node > that > >> uses Maven can have a different classpath. It seems that we've been > lucky > >> so far as this hasn't caused any issues, but I still think the approach > is > >> wrong. The behavior should be consistent. > >> 4. ignite.sh allows manual modifications (there are several "Uncomment > this > >> to..." lines). This complicates upgrades as well. > >> > >> The whole purpose of the IEP is to fix the above, automate and > >> simplify operations for the users. The general idea is to create a > single > >> tool responsible for all those operations and use Maven under the hood > to > >> include/exclude modules. > >> > >> Exact commands and steps are still under discussion though. It looks > like > >> Petr is going to create a prototype that can be used as a starting > point, > >> but if you have any suggestions or ideas in the meantime, please share > them > >> with us. > >> > >> -Val > >> > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <nizhi...@apache.org> > >> wrote: > >> > >>> Hello, Igniters. > >>> > >>> Not sure if I understand the essence of the proposal at the moment. > >>> Can you, please, describe the advantages of the proposed way from the > user > >>> perspective? > >>> > >>> How the typical DevOps pipeline should look like with this enhancement? > >>> > >>> How I as a user can create a fully functional installation package of > the > >>> Ignite? > >>> AFAIK downloading some artifacts from the internet straight to the > >>> production server is a security anti-pattern. > >>> > >>> > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dma...@apache.org> написал(а): > >>>> > >>>> Petr, thanks, > >>>> > >>>> There is also a collection of modules located in our extensions > >>> repository: > >>>> https://github.com/apache/ignite-extensions > >>>> > >>>> @Saikat Maitra <saikat.mai...@gmail.com> is migrating all our > existing > >>>> integrations to that repository and, once this is done, the extensions > >>> will > >>>> be released to Maven separately. Is it reasonable to expand the scope > of > >>>> the IEP-52 and contemplate on how to install those extensions? > >>>> > >>>> - > >>>> Denis > >>>> > >>>> > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko < > >>>> valentin.kuliche...@gmail.com> wrote: > >>>> > >>>>> Hi Petr, > >>>>> > >>>>> The proposal makes sense to me - thanks for starting the discussion. > >>>>> Current installation and upgrade procedures involve a lot of manual > >>> steps > >>>>> and quite error-prone, we need to automate and simplify them as much > as > >>>>> possible. > >>>>> > >>>>> I believe we eventually should end up with a unified command-line > tool > >>>>> (ignitectl?) that will incorporate all the operations (enable/disable > >>>>> modules, start/stop, configuration updates, activation, baseline, > etc.). > >>>>> This IEP is a step in this direction. > >>>>> > >>>>> Looking forward to testing a prototype :) > >>>>> > >>>>> -Val > >>>>> > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr.wei...@gmail.com> > >>> wrote: > >>>>> > >>>>>> Hi, Igniters! > >>>>>> > >>>>>> > >>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced > >>> delivery > >>>>>> and upgrade processes [1]. > >>>>>> The key idea is to make main binary as slim as possible (delivering > >>> every > >>>>>> optional component by demand only) while providing way to run each > new > >>>>>> version seamlessly without much of the efforts migrating data and > >>>>>> configuration between upgrades. > >>>>>> > >>>>>> I plan to create prototype based on current release of Apache Ignite > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September. > >>>>>> > >>>>>> > >>>>>> > >>>>>> [1] > >>>>>> > >>>>> > >>> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958 > >>>>>> > >>>>>> > >>>>> > >>> > >>> > > > >
