Hey guys,
We did some work in the past to make this transition begin to happen
automatically without user actions. I think there was an outline in
here a year or so back. Basically, the path forward was:
1) to standardize a location in a SWORD repository for .zip package.
This has many benefits beyond helping us get to HTTPS everywhere. It
conceptually allows reading directly from the zip file by the engine.
It could facilitate the engine itself building the zip packages from an
exploded tree. All transports, including the dominant FTP transport
right now could look to see if there is an available single .zip package
for download instead of downloading each individual file for a module
during a remote install, and I am sure more ideas people will imagine in
the future... but simply this step was to standardize the location of
.zip package which each represent a zip of a single package based at the
root of a sword module repo, e.g.,
KJV.zip:
mods.d/kjv.conf
modules/texts/ztext/kjv/nt.bzs
modules/texts/ztext/kjv/ot.bzs
modules/texts/ztext/kjv/ot.bzv
modules/texts/ztext/kjv/nt.bzv
modules/texts/ztext/kjv/ot.bzz
modules/texts/ztext/kjv/nt.bzz
This location decided on was pretty straighforward, at the root of the
sword module repo, packages/
You'll notice this if you have a look at the crosswire repo here:
https://crosswire.org/ftpmirror/pub/sword/raw/
The latest version of the engine does use this now and tries to downoad
a package if available, before it falls back to the previous behavior of
downloading each individual file.
2) Support 'chained' transport providers in the engine, allowing
multiple transports to the same repository. This has been completed and
the first use of this concept is the recognition of a new
HTTPSPackagePreference option in installmgr's configuration file.
The value of the property is: existing repo name|hostname|repository_path
You can see this added for the CrossWire repository in the master repo
list registry at:
https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf
HTTPSPackagePreference=CrossWire|crosswire.org|/ftpmirror/pub/sword/raw
FTPSource=CrossWire|ftp.crosswire.org|/pub/sword/raw
This tells installmgr that if a user requests a package install from
"CrossWire" then chain 2 transport provider attempts together and prefer
the HTTPS transport first.
The hope is that this path forward will see the vast amount of SWORD
frontends begin to start using HTTPS instead of using FTP, to facilitate
a smooth transition from FTP to HTTPS. The idea is that if a repo owner
sets everything up correctly, and a user has the latest software and
latest configuration information from our registry of known SWORD repos
( https://crosswire.org/ftpmirror/pub/sword/masterRepoList.conf ),
everything will flow over HTTPS instead of FTP and once we see FTP
traffic drop to very low levels, a repo owner could choose to
discontinue support for FTP if they desire.
Michael, I am glad you have your server back up and running well again.
Thank you for all that you do to distribute the Word of God to a needing
world.
Troy
On 2/27/25 9:45 PM, Michael Johnson wrote:
Hi David,
Yes, under eBible.org, where it says "HTTP access", it should say
"HTTPS and HTTP access". Also, any repository that also supports HTTPS
and/or HTTP access should explicitly say so. As Greg mentioned, mobile
often blocks FTP. FTP may not be dead, yet, but it is in hospice care.
HTTP likewise can cause problems with mobile platforms, where the
major app stores now require use of HTTPS instead of insecure HTTP in
most cases. I still support insecure HTTP connections to eBible.org
except for at shop.eBible.org and
https://eBible.org/cgi-bin/contact.cgi but there may come a day when
HTTP goes away, too.
On 2/26/25 22:03, David Haslam wrote:
Hi Michael,
/Does anything need to be changed here?/
Official and Affiliated Module Repositories - CrossWire Bible Society
<https://wiki.crosswire.org/Official_and_Affiliated_Module_Repositories#eBible.org>
_Everyone_: Except for the note under *AndBibleExtra*, there's no
mention of *https* !
/Do we need to make any wider changes to be future proof?/
Best regards,
David
Sent with Proton Mail <https://proton.me/mail/home> secure email.
On Thursday, February 27th, 2025 at 7:13 AM, Michael Johnson
<kahunap...@ebible.org> wrote:
On 2/26/25 17:12, Greg Hellings wrote:
On Wed, Feb 26, 2025, 7:33 PM Kahunapule Michael Johnson
<kahunap...@ebible.org> wrote:
Greetings from Maui!
tldr: upgrade your Sword apps to always use https instead of
http or ftp to access repositories ASAP.
While technically any network acess other than anonymous FTP
support is optionally supported only with a build dep, in reality
there is no need to support anything other than HTTPS. Every Linux
distribution, and Windows build of note has libcurl, the Brew
version is also built against it, and the HTTP(S) support was added
because mobile often blocks FTP.
So you're basically completely safe.
Awesome!
As many of you are probably aware, the last week was not a
model of reliability for the eBible.org repository, or for the
rest of the eBible.org site. On the 19th of February, the
eBible.org server hardware failed. Exactly what failure, I
don't know, because it was in a data center over 4,000 miles
from my house. I just knew that it wouldn't talk to me in any
of the 3 ways I can normally access the leased dedicated
server. No worries, because I have a fast backup, right? I
allocated a new dedicated server
from the same company (Ionos) and attempted to restore from a
backup. That failed with about 80 error messages. Next plan:
restore from a mirror image of the server in my home office.
That actually worked, but it took more than 3 days to get all
of the data there (about 300 GBytes), plus time to get all of
the configuration right. In the mean time, my other leased
server (the one that didn't crash, hosting 24 other sites) gave
early warning signs that it was not going to be in service much
longer. Then
everything worked except that I forgot a couple of tweaks I had
to do to make the ftp server compatible with Sword. I fixed
that, and things were still not OK. EBible.org availability
kept going up and down like a yo-yo, mostly because the remote
control software I was using was not designed to handle
multiple IP addresses per server and anonymous ftp sites. Also,
the cost of allocating multiple IP v4 addresses has gone up.
Anonymous ftp is pretty much obsolete. I will be dropping it,
but slowly.
A Herculean effort, but I'm glad for you that your recovery was
successful! I'm curious why you need 4 separate addresses? What is
the need, there?
So far, I have been using Plesk to set up virtual hosts. I have 25
sites (and some aliases for those), some of which are much more
important than others. Plesk lets me share one IP address with all
sites except any site that has an anonymous ftp service associated
with it. The only site I have that has an anonymous ftp service
associated with it, of course, is the ftp.eBible.org Sword
repository. So I had to assign 2 IP version 4 addresses to the
server. For a long time, I was running 2 servers with every site on
them for redundancy. I had stopped doing that because the sites grew
too large for one of the servers I was renting, and I thought I had
a workable fast backup/restore plan, unlike when I had extremely
slow and expensive Internet in Papua New Guinea. (I have some
serious space in audio and video Bibles.) So that is 2 servers x 2
IP addresses = 4 IP addresses. But that configuration was unstable,
so I went to just one IP address per server by fighting my old ally,
Plesk, using manual ProFTP configuration (and a cron job to slap my
configuration back whenever Plesk rewrites it). That is not a really
good long-term solution, though.
...
Would you like a hand building up some DR or deployment automation
so you can avoid needing to remember settings? IT automation is one
of my primary skillsets, so if you'd like any sort of help setting
it up, let me know. For instance, it's not too hard to put together
automation scripts to run on a provisioned box to stand up the web
server, ftp server, etc so that you don't need to manually edit
files and the like.
That would be useful. That could be a way to escape my dependence on
and fight with Plesk.
Alternatively, have you considered an alternative way to host the
data? You could probably build a Container image with all the files
in it and host that on something like Amazon Container Service or
any of the many cloud Kubernetes hosts around. A container image
would also make it easy for someone to grab the whole collection
and make it available in an offline context the way they can with
the old CD images Troy used to distribute.
I have looked at alternatives in the past, but it may be worth
looking again. When I last looked, AWS was more expensive at my
traffic levels and site counts than using a rented dedicated server.
Another alternative might be hosting at my house when (if?) Hawaiian
Telephone makes good on its promise to bring fiber Internet to my
neighborhood. (It is actually available about a half mile away,
right now, but I haven't seen them working on it around here.)
Or even put the files into an object storage container if you're
dedicated to eliminating FTP access eventually. With just a small
shell script you can push the needed files and their indexes into
an S3, Ceph, etc object storage service and then you wouldn't need
to run a dedicated server with them to manage uptime. All of those
services offer ways to expose the files over HTTPS.
As I said on Facebook, I'm happy to lend a hand if there's anything
I can do to help smooth your infrastructure! I can even host an
emergency mirror if need be, as I have pretty reliable Internet and
electric when my neighbors don't drive into the electric poles.
This year I'm dedicating some of my time to working on home
electric backups!
Thank you, Greg. I may take you up on that...
--
Peace,
*/Michael Johnson/**
26 HIWALANI LOOP • MAKAWAO HI 96768-8747*• USA
mljohnson.org <https://mljohnson.org/> • eBible.org
<https://eBible.org> • WorldEnglish.Bible
<https://WorldEnglish.Bible> • PNG.Bible <https://PNG.Bible>
Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921
Skype: kahunapule • Telegram: @kahunapule • Facebook:
fb.me/kahunapule <https://www.facebook.com/kahunapule>
_______________________________________________
sword-devel mailing list:sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
--
signature
Peace,
*/Michael Johnson/**
26 HIWALANI LOOP • MAKAWAO HI 96768-8747*• USA
mljohnson.org <https://mljohnson.org/> • eBible.org
<https://eBible.org> • WorldEnglish.Bible <https://WorldEnglish.Bible>
• PNG.Bible <https://PNG.Bible>
Signal/Telegram/WhatsApp/Telephone: +1 808-333-6921
Skype: kahunapule • Telegram: @kahunapule • Facebook: fb.me/kahunapule
<https://www.facebook.com/kahunapule>
_______________________________________________
sword-devel mailing list:sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel@crosswire.org
http://crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
