Guys, I had a quick chat with Nikita earlier today and it seems that we have came across of a blocker of some kind. With my member's hat on I would love to help to find a resolution that let us (and potentially other communities) to move forward.
Let's craft the message and circulate it with Infra and Greg. Denis, could you find me tomorrow at the IMCS and give me an insight into the technical side of it? I feel a bit at loss after going through this thread... -- Regards, Cos On October 2, 2017 7:31:48 AM PDT, Denis Magda <dma...@apache.org> wrote: >Dmitriy, > >That’s the rule. See replies in the ticket [1] > >*Background: the TLP server is already pretty darned busy just serving >*static* sites. Dynamic operation for near-200 PMCs would bury the >machine. Our policy of "static-only websites" has been in place since >the Foundation started* > >Download scripts seem to be the only exception and we as PMC don’t even >have access to them. > >If you want to keep pushing this direction let’s craft a message to >Greg and Daniel directly. I don’t know what else to ask for here rather >than a virtual machine that’s conceivably to much for a single script >like that. > >[1] https://issues.apache.org/jira/browse/INFRA-15182 ><https://issues.apache.org/jira/browse/INFRA-15182> > >— >Denis > >> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrak...@apache.org> >wrote: >> >> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov ><voze...@gridgain.com> >> wrote: >> >>> I am not sure it is good idea to send requests to 3rd-party >addresses from >>> Ignite node. Let's do not make the same mistakes again. >>> >> >> Agree with Vladimir. >> >> We obviously have CGI support on the website. Can someone explain why >CGI >> is not possible to use? >> >> >>> >>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov ><anovi...@gridgain.com> >>> wrote: >>> >>>> We may directly send request to GA from Ignite node: >>>> https://developers.google.com/analytics/devguides/ >>> collection/protocol/v1/ >>>> <https://developers.google.com/analytics/devguides/ >>> collection/protocol/v1/ >>>>> >>>> Latest version can be received from maven central: >>>> https://repo1.maven.org/maven2/org/apache/ignite/ >>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/ >>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml> >>>> >>>> >>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrak...@apache.org> >>>> написал(а): >>>>> >>>>> Denis, >>>>> >>>>> I am not sure I understand. We already do have CGI enabled for >>>>> download.cgi. Is there something else we need? >>>>> >>>>> D. >>>>> >>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dma...@gridgain.com> >>> wrote: >>>>> >>>>>> There is an obstacle. There is no way to execute the script using >PHP >>> or >>>>>> similar sever side language and trigger GA as discussed earlier: >>>>>> https://issues.apache.org/jira/browse/INFRA-15182 >>>>>> >>>>>> How else can we tackle this? >>>>>> >>>>>> Denis >>>>>> >>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan < >>>> dsetrak...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> I think it is safe to assume at this point that everyone is in >>> general >>>>>>> agreement, since there are no active objections. >>>>>>> >>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it >>> happen: >>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305 >>>>>>> >>>>>>> D. >>>>>>> >>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan < >>>>>> dsetrak...@apache.org >>>>>>> <javascript:;>> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani < >>> raul....@evosent.com >>>>>>> <javascript:;>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yeah, I guess that's doable as well and requires less >management >>>>>> effort >>>>>>>>> than my suggestion. We could use events [1] to store payload >data >>>>>> (e.g. >>>>>>>>> IP, >>>>>>>>> version, etc.) >>>>>>>> >>>>>>>> >>>>>>>> Yes, we could use events or some other similar API provided by >GA. >>>>>>>> >>>>>>>> >>>>>>>>> What the download page CGI developed in? PHP? >>>>>>>>> >>>>>>>> >>>>>>>> To be honest, no clue. I guess someone in the community can >figure >>> it >>>>>>> out: >>>>>>>> >https://svn.apache.org/repos/asf/ignite/site/trunk/download.html >>>>>>>> >>>>>>>> >>>>>>>>> However, I'm not sure whether storing this data in a 3rd party >>>>>> (Google) >>>>>>> is >>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if >>> there's >>>>>>>>> doubt >>>>>>>>> in the PMC, it's better to ask legal@. >>>>>>>> >>>>>>>> >>>>>>>> I am not sure there is anything to ask about. The whole Ignite >>> website >>>>>> is >>>>>>>> GA enabled, and all we are doing is accessing a standard web >page >>> from >>>>>>> the >>>>>>>> Ignite web site. The information gathered from GA is available >only >>> to >>>>>>> the >>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by >this >>>>>>>> question. >>>>>>>> >>>>>>>> Even ASF website itself uses GA: https://www.apache.org/ >>>>>>>> foundation/policies/privacy.html >>>>>>>> >>>>>>>> >>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in. >>>>>>>>> >>>>>>>> >>>>>>>> Sure, would be nice to hear from Roman as well. >>>>>>>> >>>>>>>> >>>>>>>>> Cheers. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://developers.google.com/analytics/devguides/collection >>>>>>>>> /analyticsjs/events >>>>>>>>> >>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan < >>>>>>>>> dsetrak...@apache.org <javascript:;>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Raul, >>>>>>>>>> >>>>>>>>>> Could point about Javascript, it will not work because it >executes >>>>>> in >>>>>>>>> the >>>>>>>>>> browser. This means we need a server-side script, like CGI we >are >>>>>>> using >>>>>>>>> on >>>>>>>>>> our download page. >>>>>>>>>> >>>>>>>>>> How about this approach. We create something like >>> ignite-version.cgi >>>>>>>>> script >>>>>>>>>> which will invoke a call to GA and then return the latest >version. >>>>>>>>>> >>>>>>>>>> This should work, right? >>>>>>>>>> >>>>>>>>>> D. >>>>>>>>>> >>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani < >>>>>> raul....@evosent.com >>>>>>> <javascript:;>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Dmitriy and all >>>>>>>>>>> >>>>>>>>>>> Also, since we have GA enabled for the website, we can track >how >>>>>>> many >>>>>>>>>> times >>>>>>>>>>>> this page was accessed, which will be equal to the number >of >>>>>>> starts. >>>>>>>>>> This >>>>>>>>>>>> way, the counter information is tracked and monitored by >the >>>>>>> Ignite >>>>>>>>>> PMC. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Unfortunately this won't work because GA is loaded via >Javascript >>>>>> on >>>>>>>>> the >>>>>>>>>>> browser, so Google will never receive the page hit. >>>>>>>>>>> >>>>>>>>>>> Given the constraints, the most viable solution is an HTTPS >>>>>> endpoint >>>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST >>>>>> request. >>>>>>>>> The >>>>>>>>>>> simplest thing is to write each request in a log file, along >with >>>>>>> the >>>>>>>>>>> timestamp, the version reported by the client, maybe the IP >(not >>>>>>> sure >>>>>>>>>> about >>>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK >if >>>>>> you >>>>>>>>> make >>>>>>>>>> it >>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the >node >>>>>>> creates >>>>>>>>> on >>>>>>>>>>> first startup or something. >>>>>>>>>>> >>>>>>>>>>> This endpoint would need some basic DDoS protection and >>>>>> blacklisting >>>>>>>>> to >>>>>>>>>>> prevent data spoofing. >>>>>>>>>>> >>>>>>>>>>> If we'll be implementing this endpoint anyway, then there's >no >>>>>> point >>>>>>>>>>> placing another file on Git or elsewhere for reporting the >latest >>>>>>>>>> versions: >>>>>>>>>>> the endpoint itself can return them. >>>>>>>>>>> >>>>>>>>>>> WDYT? >>>>>>>>>>> >>>>>>>>>>> Cheers. >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan < >>>>>>>>>> dsetrak...@apache.org <javascript:;>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Cos, Raul, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the feedback. I completely agree about Maven >Central >>>>>>>>> being a >>>>>>>>>>> 3rd >>>>>>>>>>>> party repo (did not think about that initially). All your >>>>>>>>> suggestions >>>>>>>>>>> make >>>>>>>>>>>> sense, but I would like to keep it as simple as possible, >and so >>>>>>> far >>>>>>>>>>>> everything suggested required GIT dependencies and extra >work. >>>>>>>>>>>> >>>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the >Ignite >>>>>>>>>> website >>>>>>>>>>>> which will have only the latest version. Every time a node >>>>>> starts, >>>>>>>>> it >>>>>>>>>>>> receives the latest version from the page and suggests that >>>>>> users >>>>>>>>>> upgrade >>>>>>>>>>>> if needed. >>>>>>>>>>>> >>>>>>>>>>>> Also, since we have GA enabled for the website, we can >track how >>>>>>>>> many >>>>>>>>>>> times >>>>>>>>>>>> this page was accessed, which will be equal to the number >of >>>>>>> starts. >>>>>>>>>> This >>>>>>>>>>>> way, the counter information is tracked and monitored by >the >>>>>>> Ignite >>>>>>>>>> PMC. >>>>>>>>>>>> >>>>>>>>>>>> This approach looks pretty innocent to me and everything is >kept >>>>>>> and >>>>>>>>>>>> managed within Apache. >>>>>>>>>>>> >>>>>>>>>>>> Thoughts? >>>>>>>>>>>> >>>>>>>>>>>> D. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik < >>>>>>>>> c...@apache.org <javascript:;>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I agree with Raul. >>>>>>>>>>>>> - providing a ping-back address to a 3rd party might be >frown >>>>>>>>> upon by >>>>>>>>>>>> some. >>>>>>>>>>>>> And might have a consequences like stats collection about >>>>>>> users' >>>>>>>>>>>>> infrastructure. >>>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any >>>>>> binary >>>>>>>>>> data: >>>>>>>>>>>>> everything is clear text and could be easily monitored by >>>>>> any >>>>>>> of >>>>>>>>>> the >>>>>>>>>>>>> network >>>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a >>>>>> bit >>>>>>> of >>>>>>>>>> the >>>>>>>>>>>>> release >>>>>>>>>>>>> discipline. >>>>>>>>>>>>> - the binary data download in the runtime is my main >concern. >>>>>>>>> This is >>>>>>>>>>> the >>>>>>>>>>>>> vector of MMA. What if someone gains the control over the >>>>>>>>>> repository >>>>>>>>>>>> and >>>>>>>>>>>>> replaces the file with some malicious content. >>>>>>>>>>>>> >>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make >a >>>>>> call >>>>>>>>> to >>>>>>>>>> an >>>>>>>>>>>>> external script to check upon the atest software version >>>>>>> available >>>>>>>>>> for >>>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd >party >>>>>>>>>> server, >>>>>>>>>>> I >>>>>>>>>>>>> believe the best way would be to put this script on ASF >infra >>>>>>> and >>>>>>>>>> have >>>>>>>>>>>> the >>>>>>>>>>>>> "update checker" running in a completely controlled >>>>>> environment. >>>>>>>>>>>> Actually, >>>>>>>>>>>>> I >>>>>>>>>>>>> recall we had this very discussion during the Incubation; >I >>>>>> can >>>>>>>>>>> probably >>>>>>>>>>>>> dig >>>>>>>>>>>>> out the corresponding thread. >>>>>>>>>>>>> >>>>>>>>>>>>> Thoughts? >>>>>>>>>>>>> Cok >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote: >>>>>>>>>>>>>> Hey guys >>>>>>>>>>>>>> >>>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party >>>>>>>>>> (Sonatype). >>>>>>>>>>>> We >>>>>>>>>>>>>> don't know what kind of data analysis or intelligence >>>>>>> extraction >>>>>>>>>> they >>>>>>>>>>>>> run. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If Ignite servers all over the world were hitting >maven.org >>>>>>>>>>>> periodically >>>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a >>>>>> clear >>>>>>>>>>>> indication >>>>>>>>>>>>>> of who is running an Ignite server. >>>>>>>>>>>>>> >>>>>>>>>>>>>> They could reverse lookup the IP address and find out >what >>>>>>>>>>> corporation >>>>>>>>>>>> it >>>>>>>>>>>>>> is. >>>>>>>>>>>>>> >>>>>>>>>>>>>> How about having Ignite check the ASF Git directly? >>>>>>>>>>>>>> >>>>>>>>>>>>>> We could use the Git ssh API, but that would require a >new >>>>>>>>>>> dependency, >>>>>>>>>>>>>> which I advise against. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new >Git >>>>>>>>> tags: >>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git >>>>>>>>>>>>>> >>>>>>>>>>>>>> Another option is to place a txt file in the root of the >>>>>>> master >>>>>>>>>>> branch >>>>>>>>>>>>> (e.g >>>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for >>>>>> each >>>>>>>>> major >>>>>>>>>>>>> version >>>>>>>>>>>>>> line (1.x, 2.x). >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would advocate this last option, but it requires >somebody >>>>>>>>>>> remembering >>>>>>>>>>>>> to >>>>>>>>>>>>>> update the file with every release, unless we automate it >>>>>>> with a >>>>>>>>>>> Maven >>>>>>>>>>>>>> plugin. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hope that helps! >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dma...@apache.org >>>>>>> <javascript:;>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I see nothing wrong with this approach. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? >Is >>>>>> it >>>>>>>>> good >>>>>>>>>>> to >>>>>>>>>>>>> go? >>>>>>>>>>>>>> >>>>>>>>>>>>>> — >>>>>>>>>>>>>> Denis >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan < >>>>>>>>>>>> dsetrak...@apache.org <javascript:;> >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a >ticket >>>>>> on >>>>>>>>> it? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan < >>>>>>>>>>>>> dsetrak...@apache.org <javascript:;>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Igniters, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There has been lots of talk of proposals about various >>>>>>> usage >>>>>>>>>>> metrics >>>>>>>>>>>>> for >>>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to >resurrect >>>>>>> the >>>>>>>>>> topic >>>>>>>>>>>> and >>>>>>>>>>>>>>>> propose something very simple and non-intrusive. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. Update Checker >>>>>>>>>>>>>>>> The main purpose of the update checker is not to >collect >>>>>>>>>> metrics, >>>>>>>>>>>> but >>>>>>>>>>>>> to >>>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing >>>>>>>>>> maven.org >>>>>>>>>>>> and >>>>>>>>>>>>>>>> getting the version out of the metadata file: >>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/ >>>>>>> org/apache/ignite/ignite-core/ >>>>>>>>>>>>>>>> maven-metadata.xml >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This way we do not send any information anywhere and, >at >>>>>>> the >>>>>>>>>> same >>>>>>>>>>>>> time, >>>>>>>>>>>>>>>> urge our users to download and start using the latest >>>>>>>>> version of >>>>>>>>>>>>> Ignite. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Startup Counter >>>>>>>>>>>>>>>> This piece is optional, but we can also get an insight >in >>>>>>> how >>>>>>>>>> many >>>>>>>>>>>>> times >>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> certain Ignite release gets started. This is just a >cool >>>>>>>>> metric >>>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> community to gauge the project popularity. You can >think >>>>>> of >>>>>>>>> it >>>>>>>>>> as >>>>>>>>>>>> of a >>>>>>>>>>>>>> page >>>>>>>>>>>>>>>> visit counter shown on many websites. We can even >decide >>>>>> to >>>>>>>>>>> display >>>>>>>>>>>>> this >>>>>>>>>>>>>>>> counter on the Ignite website as well. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every >>>>>>>>> release, >>>>>>>>>>> e.g. >>>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 >byte. >>>>>>>>> Every >>>>>>>>>>> time >>>>>>>>>>>>> an >>>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the >>>>>>>>> background. >>>>>>>>>>>> Then >>>>>>>>>>>>> we >>>>>>>>>>>>>>>> will be able to view the number of the total downloads >>>>>> for >>>>>>>>> this >>>>>>>>>>> JAR >>>>>>>>>>>> in >>>>>>>>>>>>>>>> Maven Central, which is essentially the number of >starts >>>>>> of >>>>>>>>>> Ignite >>>>>>>>>>>>> nodes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Note that neither of the above suggestions require >>>>>> Ignite >>>>>>> to >>>>>>>>>> send >>>>>>>>>>>> or >>>>>>>>>>>>>>>> track any user information whatsoever.* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this >>>>>>>>> approach. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> D. >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>
