Dmitriy, That’s the rule. See replies in the ticket [1]
*Background: the TLP server is already pretty darned busy just serving *static* sites. Dynamic operation for near-200 PMCs would bury the machine. Our policy of "static-only websites" has been in place since the Foundation started* Download scripts seem to be the only exception and we as PMC don’t even have access to them. If you want to keep pushing this direction let’s craft a message to Greg and Daniel directly. I don’t know what else to ask for here rather than a virtual machine that’s conceivably to much for a single script like that. [1] https://issues.apache.org/jira/browse/INFRA-15182 <https://issues.apache.org/jira/browse/INFRA-15182> — Denis > On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrak...@apache.org> wrote: > > On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <voze...@gridgain.com> > wrote: > >> I am not sure it is good idea to send requests to 3rd-party addresses from >> Ignite node. Let's do not make the same mistakes again. >> > > Agree with Vladimir. > > We obviously have CGI support on the website. Can someone explain why CGI > is not possible to use? > > >> >> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovi...@gridgain.com> >> wrote: >> >>> We may directly send request to GA from Ignite node: >>> https://developers.google.com/analytics/devguides/ >> collection/protocol/v1/ >>> <https://developers.google.com/analytics/devguides/ >> collection/protocol/v1/ >>>> >>> Latest version can be received from maven central: >>> https://repo1.maven.org/maven2/org/apache/ignite/ >>> ignite-core/maven-metadata.xml <https://repo1.maven.org/ >>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml> >>> >>> >>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrak...@apache.org> >>> написал(а): >>>> >>>> Denis, >>>> >>>> I am not sure I understand. We already do have CGI enabled for >>>> download.cgi. Is there something else we need? >>>> >>>> D. >>>> >>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dma...@gridgain.com> >> wrote: >>>> >>>>> There is an obstacle. There is no way to execute the script using PHP >> or >>>>> similar sever side language and trigger GA as discussed earlier: >>>>> https://issues.apache.org/jira/browse/INFRA-15182 >>>>> >>>>> How else can we tackle this? >>>>> >>>>> Denis >>>>> >>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan < >>> dsetrak...@apache.org> >>>>> wrote: >>>>> >>>>>> I think it is safe to assume at this point that everyone is in >> general >>>>>> agreement, since there are no active objections. >>>>>> >>>>>> I have filed a ticket for the 2.3 release. Let's try to make it >> happen: >>>>>> https://issues.apache.org/jira/browse/IGNITE-6305 >>>>>> >>>>>> D. >>>>>> >>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan < >>>>> dsetrak...@apache.org >>>>>> <javascript:;>> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani < >> raul....@evosent.com >>>>>> <javascript:;>> >>>>>>> wrote: >>>>>>> >>>>>>>> Yeah, I guess that's doable as well and requires less management >>>>> effort >>>>>>>> than my suggestion. We could use events [1] to store payload data >>>>> (e.g. >>>>>>>> IP, >>>>>>>> version, etc.) >>>>>>> >>>>>>> >>>>>>> Yes, we could use events or some other similar API provided by GA. >>>>>>> >>>>>>> >>>>>>>> What the download page CGI developed in? PHP? >>>>>>>> >>>>>>> >>>>>>> To be honest, no clue. I guess someone in the community can figure >> it >>>>>> out: >>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html >>>>>>> >>>>>>> >>>>>>>> However, I'm not sure whether storing this data in a 3rd party >>>>> (Google) >>>>>> is >>>>>>>> compliant with the ASF policy. I guess it's no biggie, but if >> there's >>>>>>>> doubt >>>>>>>> in the PMC, it's better to ask legal@. >>>>>>> >>>>>>> >>>>>>> I am not sure there is anything to ask about. The whole Ignite >> website >>>>> is >>>>>>> GA enabled, and all we are doing is accessing a standard web page >> from >>>>>> the >>>>>>> Ignite web site. The information gathered from GA is available only >> to >>>>>> the >>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by this >>>>>>> question. >>>>>>> >>>>>>> Even ASF website itself uses GA: https://www.apache.org/ >>>>>>> foundation/policies/privacy.html >>>>>>> >>>>>>> >>>>>>>> I think Cos said it's OK; maybe Roman can pitch in. >>>>>>>> >>>>>>> >>>>>>> Sure, would be nice to hear from Roman as well. >>>>>>> >>>>>>> >>>>>>>> Cheers. >>>>>>>> >>>>>>>> [1] >>>>>>>> https://developers.google.com/analytics/devguides/collection >>>>>>>> /analyticsjs/events >>>>>>>> >>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan < >>>>>>>> dsetrak...@apache.org <javascript:;>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Raul, >>>>>>>>> >>>>>>>>> Could point about Javascript, it will not work because it executes >>>>> in >>>>>>>> the >>>>>>>>> browser. This means we need a server-side script, like CGI we are >>>>>> using >>>>>>>> on >>>>>>>>> our download page. >>>>>>>>> >>>>>>>>> How about this approach. We create something like >> ignite-version.cgi >>>>>>>> script >>>>>>>>> which will invoke a call to GA and then return the latest version. >>>>>>>>> >>>>>>>>> This should work, right? >>>>>>>>> >>>>>>>>> D. >>>>>>>>> >>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani < >>>>> raul....@evosent.com >>>>>> <javascript:;>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hey Dmitriy and all >>>>>>>>>> >>>>>>>>>> Also, since we have GA enabled for the website, we can track how >>>>>> many >>>>>>>>> times >>>>>>>>>>> this page was accessed, which will be equal to the number of >>>>>> starts. >>>>>>>>> This >>>>>>>>>>> way, the counter information is tracked and monitored by the >>>>>> Ignite >>>>>>>>> PMC. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Unfortunately this won't work because GA is loaded via Javascript >>>>> on >>>>>>>> the >>>>>>>>>> browser, so Google will never receive the page hit. >>>>>>>>>> >>>>>>>>>> Given the constraints, the most viable solution is an HTTPS >>>>> endpoint >>>>>>>>>> running on ASF infra that Ignite invokes via a GET or POST >>>>> request. >>>>>>>> The >>>>>>>>>> simplest thing is to write each request in a log file, along with >>>>>> the >>>>>>>>>> timestamp, the version reported by the client, maybe the IP (not >>>>>> sure >>>>>>>>> about >>>>>>>>>> the ASF rules about this concerning privacy, I guess it's OK if >>>>> you >>>>>>>> make >>>>>>>>> it >>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node >>>>>> creates >>>>>>>> on >>>>>>>>>> first startup or something. >>>>>>>>>> >>>>>>>>>> This endpoint would need some basic DDoS protection and >>>>> blacklisting >>>>>>>> to >>>>>>>>>> prevent data spoofing. >>>>>>>>>> >>>>>>>>>> If we'll be implementing this endpoint anyway, then there's no >>>>> point >>>>>>>>>> placing another file on Git or elsewhere for reporting the latest >>>>>>>>> versions: >>>>>>>>>> the endpoint itself can return them. >>>>>>>>>> >>>>>>>>>> WDYT? >>>>>>>>>> >>>>>>>>>> Cheers. >>>>>>>>>> >>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan < >>>>>>>>> dsetrak...@apache.org <javascript:;>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Cos, Raul, >>>>>>>>>>> >>>>>>>>>>> Thanks for the feedback. I completely agree about Maven Central >>>>>>>> being a >>>>>>>>>> 3rd >>>>>>>>>>> party repo (did not think about that initially). All your >>>>>>>> suggestions >>>>>>>>>> make >>>>>>>>>>> sense, but I would like to keep it as simple as possible, and so >>>>>> far >>>>>>>>>>> everything suggested required GIT dependencies and extra work. >>>>>>>>>>> >>>>>>>>>>> How about Yakov's suggestion. We simply add a page to the Ignite >>>>>>>>> website >>>>>>>>>>> which will have only the latest version. Every time a node >>>>> starts, >>>>>>>> it >>>>>>>>>>> receives the latest version from the page and suggests that >>>>> users >>>>>>>>> upgrade >>>>>>>>>>> if needed. >>>>>>>>>>> >>>>>>>>>>> Also, since we have GA enabled for the website, we can track how >>>>>>>> many >>>>>>>>>> times >>>>>>>>>>> this page was accessed, which will be equal to the number of >>>>>> starts. >>>>>>>>> This >>>>>>>>>>> way, the counter information is tracked and monitored by the >>>>>> Ignite >>>>>>>>> PMC. >>>>>>>>>>> >>>>>>>>>>> This approach looks pretty innocent to me and everything is kept >>>>>> and >>>>>>>>>>> managed within Apache. >>>>>>>>>>> >>>>>>>>>>> Thoughts? >>>>>>>>>>> >>>>>>>>>>> D. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik < >>>>>>>> c...@apache.org <javascript:;>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I agree with Raul. >>>>>>>>>>>> - providing a ping-back address to a 3rd party might be frown >>>>>>>> upon by >>>>>>>>>>> some. >>>>>>>>>>>> And might have a consequences like stats collection about >>>>>> users' >>>>>>>>>>>> infrastructure. >>>>>>>>>>>> - checking an ASF git-repo is easy and won't download any >>>>> binary >>>>>>>>> data: >>>>>>>>>>>> everything is clear text and could be easily monitored by >>>>> any >>>>>> of >>>>>>>>> the >>>>>>>>>>>> network >>>>>>>>>>>> diagnostic tools, shall it be required. But it involves a >>>>> bit >>>>>> of >>>>>>>>> the >>>>>>>>>>>> release >>>>>>>>>>>> discipline. >>>>>>>>>>>> - the binary data download in the runtime is my main concern. >>>>>>>> This is >>>>>>>>>> the >>>>>>>>>>>> vector of MMA. What if someone gains the control over the >>>>>>>>> repository >>>>>>>>>>> and >>>>>>>>>>>> replaces the file with some malicious content. >>>>>>>>>>>> >>>>>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a >>>>> call >>>>>>>> to >>>>>>>>> an >>>>>>>>>>>> external script to check upon the atest software version >>>>>> available >>>>>>>>> for >>>>>>>>>>>> download. In the past, the endpoint was running on a 3rd party >>>>>>>>> server, >>>>>>>>>> I >>>>>>>>>>>> believe the best way would be to put this script on ASF infra >>>>>> and >>>>>>>>> have >>>>>>>>>>> the >>>>>>>>>>>> "update checker" running in a completely controlled >>>>> environment. >>>>>>>>>>> Actually, >>>>>>>>>>>> I >>>>>>>>>>>> recall we had this very discussion during the Incubation; I >>>>> can >>>>>>>>>> probably >>>>>>>>>>>> dig >>>>>>>>>>>> out the corresponding thread. >>>>>>>>>>>> >>>>>>>>>>>> Thoughts? >>>>>>>>>>>> Cok >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote: >>>>>>>>>>>>> Hey guys >>>>>>>>>>>>> >>>>>>>>>>>>> In my opinion, maven.org is still owned by a third party >>>>>>>>> (Sonatype). >>>>>>>>>>> We >>>>>>>>>>>>> don't know what kind of data analysis or intelligence >>>>>> extraction >>>>>>>>> they >>>>>>>>>>>> run. >>>>>>>>>>>>> >>>>>>>>>>>>> If Ignite servers all over the world were hitting maven.org >>>>>>>>>>> periodically >>>>>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a >>>>> clear >>>>>>>>>>> indication >>>>>>>>>>>>> of who is running an Ignite server. >>>>>>>>>>>>> >>>>>>>>>>>>> They could reverse lookup the IP address and find out what >>>>>>>>>> corporation >>>>>>>>>>> it >>>>>>>>>>>>> is. >>>>>>>>>>>>> >>>>>>>>>>>>> How about having Ignite check the ASF Git directly? >>>>>>>>>>>>> >>>>>>>>>>>>> We could use the Git ssh API, but that would require a new >>>>>>>>>> dependency, >>>>>>>>>>>>> which I advise against. >>>>>>>>>>>>> >>>>>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git >>>>>>>> tags: >>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git >>>>>>>>>>>>> >>>>>>>>>>>>> Another option is to place a txt file in the root of the >>>>>> master >>>>>>>>>> branch >>>>>>>>>>>> (e.g >>>>>>>>>>>>> LATEST), containing a list of the latest GA versions for >>>>> each >>>>>>>> major >>>>>>>>>>>> version >>>>>>>>>>>>> line (1.x, 2.x). >>>>>>>>>>>>> >>>>>>>>>>>>> I would advocate this last option, but it requires somebody >>>>>>>>>> remembering >>>>>>>>>>>> to >>>>>>>>>>>>> update the file with every release, unless we automate it >>>>>> with a >>>>>>>>>> Maven >>>>>>>>>>>>> plugin. >>>>>>>>>>>>> >>>>>>>>>>>>> Hope that helps! >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dma...@apache.org >>>>>> <javascript:;>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I see nothing wrong with this approach. >>>>>>>>>>>>> >>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is >>>>> it >>>>>>>> good >>>>>>>>>> to >>>>>>>>>>>> go? >>>>>>>>>>>>> >>>>>>>>>>>>> — >>>>>>>>>>>>> Denis >>>>>>>>>>>>> >>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan < >>>>>>>>>>> dsetrak...@apache.org <javascript:;> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket >>>>> on >>>>>>>> it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan < >>>>>>>>>>>> dsetrak...@apache.org <javascript:;>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Igniters, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There has been lots of talk of proposals about various >>>>>> usage >>>>>>>>>> metrics >>>>>>>>>>>> for >>>>>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect >>>>>> the >>>>>>>>> topic >>>>>>>>>>> and >>>>>>>>>>>>>>> propose something very simple and non-intrusive. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Update Checker >>>>>>>>>>>>>>> The main purpose of the update checker is not to collect >>>>>>>>> metrics, >>>>>>>>>>> but >>>>>>>>>>>> to >>>>>>>>>>>>>>> notify users about a new version of Ignite by accessing >>>>>>>>> maven.org >>>>>>>>>>> and >>>>>>>>>>>>>>> getting the version out of the metadata file: >>>>>>>>>>>>>>> http://repo2.maven.org/maven2/ >>>>>> org/apache/ignite/ignite-core/ >>>>>>>>>>>>>>> maven-metadata.xml >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This way we do not send any information anywhere and, at >>>>>> the >>>>>>>>> same >>>>>>>>>>>> time, >>>>>>>>>>>>>>> urge our users to download and start using the latest >>>>>>>> version of >>>>>>>>>>>> Ignite. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. Startup Counter >>>>>>>>>>>>>>> This piece is optional, but we can also get an insight in >>>>>> how >>>>>>>>> many >>>>>>>>>>>> times >>>>>>>>>>>>> a >>>>>>>>>>>>>>> certain Ignite release gets started. This is just a cool >>>>>>>> metric >>>>>>>>>> for >>>>>>>>>>>> the >>>>>>>>>>>>>>> community to gauge the project popularity. You can think >>>>> of >>>>>>>> it >>>>>>>>> as >>>>>>>>>>> of a >>>>>>>>>>>>> page >>>>>>>>>>>>>>> visit counter shown on many websites. We can even decide >>>>> to >>>>>>>>>> display >>>>>>>>>>>> this >>>>>>>>>>>>>>> counter on the Ignite website as well. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To do this, we can simply add a JAR in maven for every >>>>>>>> release, >>>>>>>>>> e.g. >>>>>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte. >>>>>>>> Every >>>>>>>>>> time >>>>>>>>>>>> an >>>>>>>>>>>>>>> Ignite node starts, it will download this JAR in the >>>>>>>> background. >>>>>>>>>>> Then >>>>>>>>>>>> we >>>>>>>>>>>>>>> will be able to view the number of the total downloads >>>>> for >>>>>>>> this >>>>>>>>>> JAR >>>>>>>>>>> in >>>>>>>>>>>>>>> Maven Central, which is essentially the number of starts >>>>> of >>>>>>>>> Ignite >>>>>>>>>>>> nodes. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Note that neither of the above suggestions require >>>>> Ignite >>>>>> to >>>>>>>>> send >>>>>>>>>>> or >>>>>>>>>>>>>>> track any user information whatsoever.* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please reply suggesting weather you are OK with this >>>>>>>> approach. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> D. >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>> >>> >>
