Hello,
I've investigated the issue some more and I found the issue.
Before that, a bit of history to understand the context:
For the past few months I've been on-and-off developing a suite of
tools based on the FS subsystem; these tools are almost ready for
release and I was planning to announce it here in the future.
I had met the reported issue (downloads freezing after a few minutes)
some time ago but I wanted to finish the core features of the programs
before delving on the finer details and especially since I had found a
temporary workaround, I didn't really bother about it until recently.
After making my report, as I was dealing with other bugs I discovered
that the issue was triggered by a slightly incorrect use of the FS
API.
The suite can, optionally, do some file path processing to help with
file indexing under certain circumstances (more details on
release). At some point, before the publishing happens, the
program "owns" two different paths pointing to the same file and only
one of them is the "true" one.
To publish a file the API needs a 'GNUNET_FS_FileInformation' struct
and the most reliable way to create one from a local file is to do
what gnunet-publish does: use a 'GNUNET_FS_DirScanner' on the path and
create the file information struct from the resulting
'GNUNET_FS_ShareTreeItem'.
Due to a typo in the code, even though the
'GNUNET_FS_file_information_create_from_file' function was receiving
the correct "true" path, the share tree was created on the _other_
path resulting in some kind of inconsistent state in which half the
data come from one path and half come from the other.
The FS index file uses the path given while creating the file
information struct, so it appeared correct when examining FS's blocks
and since both paths were pointing to the same file, the published
data was also correct.
I managed to find the error only because I decided to delete the local
FS index and start from scratch using gnunet-publish itself, rather
than my programs, and found out the issue did not manifest itself. I
think I had also managed to corrupt my previous index file or
something due to the mismatching data.
I don't know if this is a failure of the user (me), a design issue of
the APIs allowing this behaviour or if there are already plans to
revisit the entire FS system, but in the meantime would it be possible
to add the following
'GNUNET_FS_file_information_create_from_share_tree' function? The
share tree has all the required values and it would be less error
prone.
The function:
struct GNUNET_FS_FileInformation *
GNUNET_FS_file_information_create_from_share_tree (
struct GNUNET_FS_Handle *h,
void *client_info,
const struct GNUNET_FS_ShareTreeItem *item,
int do_index,
const struct GNUNET_FS_BlockOptions *bo)
{
return GNUNET_FS_file_information_create_from_file (
h,
client_info,
item->filename,
item->ksk_uri,
item->meta,
do_index,
bo);
}
(This is obviously a minimal wrapper, it can be expanded to add some
checks as necessary. As I had said previously, my knowledge of this
API comes from reading the source code for gnunet-publish and
gnunet-download, so I don't know if this wrapper breaks some
assumptions or misses others.)
Thank you,
A.V.
