On 13 June 2017 at 05:07, Sam Ruby <ru...@intertwingly.net> wrote: > On Mon, Jun 12, 2017 at 9:54 PM, Sam Ruby <ru...@intertwingly.net> wrote: >> On Mon, Jun 12, 2017 at 9:44 PM, John D. Ament <johndam...@apache.org> wrote: >>> On Mon, Jun 12, 2017 at 9:24 PM Sam Ruby <ru...@intertwingly.net> wrote: >>> >>>> On Mon, Jun 12, 2017 at 9:06 PM, Sam Ruby <ru...@intertwingly.net> wrote: >>>> > On Mon, Jun 12, 2017 at 7:59 PM, John D. Ament <johndam...@apache.org> >>>> wrote: >>>> >> On Mon, Jun 12, 2017 at 7:55 PM Sam Ruby <ru...@intertwingly.net> >>>> wrote: >>>> >> >>>> >>> On Mon, Jun 12, 2017 at 7:44 PM, <johndam...@apache.org> wrote: >>>> >>> > --- >>>> >>> > lib/whimsy/asf/svn.rb | 11 +++++++++++ >>>> >>> > www/roster/public_podlings.rb | 7 ++++++- >>>> >>> > 2 files changed, 17 insertions(+), 1 deletion(-) >>>> >>> > >>>> >>> > diff --git a/lib/whimsy/asf/svn.rb b/lib/whimsy/asf/svn.rb >>>> >>> > index 134609c..64a596e 100644 >>>> >>> > --- a/lib/whimsy/asf/svn.rb >>>> >>> > +++ b/lib/whimsy/asf/svn.rb >>>> >>> > @@ -141,6 +141,17 @@ module ASF >>>> >>> > return revision, content >>>> >>> > end >>>> >>> > >>>> >>> > + def self.updateSimple(path) >>>> >>> > + cmd = ['svn', 'update', path, '--non-interactive'] >>>> >>> >>>> >>> This will undoubtedly fail as the $apache::user (www-data) does not >>>> >>> have write access to those directories. >>>> >> >>>> >> Err so should we run cron as whimsysvn ? >>>> > >>>> > That's indeed possible, but then it probably can't write to the web >>>> directory. >>>> > >>>> > Also from reading, bad things can happen if two processes are updating >>>> > the same directory at the same time. This can be fixed via file >>>> > locking. My gitpubsub logic solves this by running the puppet agent >>>> > itself, and puppet ensures that there is only one agent running at one >>>> > time. >>>> > >>>> > I learned all this the hard way on the original whimsy_vm where >>>> > directories often got 'wedged' and needed manual intervention for >>>> > cleanup. That's why I instituted a hard separation between what can >>>> > be updated in each process. >>>> >>>> Adding to my answer: this decision (which can be changed if that what >>>> we collectively want to do) was to prefer slightly stale data over >>>> data that (at best) might occasionally stop updating, and (at worst) >>>> can become corrupt. >>>> >>>> The /srv/svn files update every 10 minutes. For most purposes, that >>>> is fast enough. >>>> >>>> Programs like the board agenda tool, the secretary mail tool, and now >>>> the roster take great care to update svn in separate tmp directories. >>>> >>> This is a very valuable piece of information. My main concern isn't roster >>> but instead the podlings information. >>> >>> Shane and I were jokingly talking about this on hipchat - we should switch >>> all of this to be pubsub. I'm more convinced that this is correct. >> >> You would still need to use flock(*) or equivalent, but definitely doable. >> >> The code for pubsub is basically the same for svn as it is for git. >> The only real difference is that the notification is 'commit' instead >> of 'push'. >> >> https://github.com/apache/whimsy/blob/master/tools/pubsub.rb >> >> The other thing to be aware of is that pubsub is only available for >> publicly readable sources. So things like foundation and documents >> can't be done this way.
I wondered about that. >> >>> Where's the logic that clones/svn's in a tmp directory? >> >> Plenty of places. Here is one: >> >> https://github.com/apache/whimsy/blob/master/www/roster/views/actions/ppmc.json.rb#L71 >> >> "git grep tmpdir" to find more. > > Another thought that should at least work for the podlings.xml case: > > podlings_xml = `svn cat > https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings.xml` > > No flock. No temp dirs. No chance of wedging/corrupting existing > directories. Or just read it as a URL ... However, SVN does sometimes hiccup, so it might be worth including a retry. As has been done for rake svn:update in the top level dir. >>>> - Sam Ruby >> >> (*) https://ruby-doc.org/core-2.4.0/File.html#method-i-flock > > - Sam Ruby
