changeset: 7047:27bb55faa024 user: Kevin McCarthy <ke...@8t8.us> date: Sat May 20 18:52:12 2017 -0700 link: http://dev.mutt.org/hg/mutt/rev/27bb55faa024
Move the IMAP msn field to IMAP_HEADER_DATA. (see #3942) Ticket 3942 revealed that it is possible for a FETCH to have gaps in the MSN numbers of the results. The code makes many assumptions that equate context index counts and MSN. This is the first in a series of commits fixing that assumption. The header->index field is supposed to hold the SORT_ORDER index number of the message. If there are gaps in the MSN, than the highext MSN can in fact be out of the range 0..ctx->msgcount-1. After studying the code, I believe curs_main.c would actually work with header->index values out of the range, at least for IMAP. But some other parts of the code, such as mutt_reopen_mailbox(), do rely on the values being a valid index to ctx->hdrs[]. And the intertwining of menu->oldcurrent with header->index values makes me nervous about future changes. So, to be safe, move the MSN to its own field in IMAP_HEADER_DATA. The next commit will fix the EXPUNGE behavior. changeset: 7048:ab3595fbb698 user: Kevin McCarthy <ke...@8t8.us> date: Sat May 20 18:52:13 2017 -0700 link: http://dev.mutt.org/hg/mutt/rev/ab3595fbb698 Fix imap expunge to match msn and fix index. (see #3942) The expunge needs to match against the MSN now. Since cmd_parse_expunge() does not automatically fix h->index anymore, change imap_expunge_mailbox() to fix up the h->index values. changeset: 7049:b85bf6466c79 user: Kevin McCarthy <ke...@8t8.us> date: Sat May 20 18:52:14 2017 -0700 link: http://dev.mutt.org/hg/mutt/rev/b85bf6466c79 Fix cmd_parse_fetch() to match against MSN. (see #3942) changeset: 7050:5289d2caadbb user: Kevin McCarthy <ke...@8t8.us> date: Sat May 20 18:52:16 2017 -0700 link: http://dev.mutt.org/hg/mutt/rev/5289d2caadbb Start fixing imap_read_headers() to account for MSN gaps. (see #3942) Change the parameters to pass MSN instead of index, to make a bit simpler. Change header cache evaluation to look at the largest MSN retrieved instead of ctx->msgcount for the next fetch start point. This still depends on the assumption that MSNs are retrieved in ascending order, which needs to be fixed. Simplify the header cache inner loop termination and memory cleanup logic. Fix a memory leak if a hole in the header cache occured. Fix the header retrieval logic to take into account MSN gaps in the results. Loop only as long as we get IMAP_CMD_CONTINUE instead of over a fixed count. Simplify the inner loop termination and memory cleanup logic too. Simplify the "new mail while fetching" logic by creating a third outer loop to handle re-fetches. Fix msg_fetch_header() to return -2 if msg_parse_fetch() encounters a corrupt FETCH response. Previously it would pass on the rc of msg_parse_fetch(), meaning it would return -1 even though the response was corrupt. changeset: 7051:77d3173aecff user: Kevin McCarthy <ke...@8t8.us> date: Sat May 20 18:52:18 2017 -0700 link: http://dev.mutt.org/hg/mutt/rev/77d3173aecff Add msn_index and max_msn to find and check boundaries by MSN. (see #3942) Since there can be gaps in MSNs, the largest MSN in the context is not necessarily ctx->msgcount. Use max_msn instead of ctx->msgcount for: - the starting MSN of new mail header fetching - boundary checking in fetch, expunge, and other places Use msn_index to efficiently look up headers by MSN. This makes the expunge code slightly more efficient. It also makes FETCH handling, and duplicate FETCH FLAG handling efficient. diffs (748 lines): diff -r 4bffaa6d189a -r 77d3173aecff imap/command.c --- a/imap/command.c Sat May 13 09:48:28 2017 -0700 +++ b/imap/command.c Sat May 20 18:52:18 2017 -0700 @@ -279,18 +279,18 @@ if (idata->reopen & IMAP_REOPEN_ALLOW) { - int count = idata->newMailCount; + unsigned int count = idata->newMailCount; if (!(idata->reopen & IMAP_EXPUNGE_PENDING) && (idata->reopen & IMAP_NEWMAIL_PENDING) - && count > idata->ctx->msgcount) + && count > idata->max_msn) { /* read new mail messages */ dprint (2, (debugfile, "imap_cmd_finish: Fetching new mail\n")); /* check_status: curs_main uses imap_check_mailbox to detect * whether the index needs updating */ idata->check_status = IMAP_NEWMAIL_PENDING; - imap_read_headers (idata, idata->ctx->msgcount, count-1); + imap_read_headers (idata, idata->max_msn+1, count); } else if (idata->reopen & IMAP_EXPUNGE_PENDING) { @@ -458,7 +458,7 @@ { char* s; char* pn; - int count; + unsigned int count; s = imap_next_word (idata->buf); pn = imap_next_word (s); @@ -479,7 +479,7 @@ count = atoi (pn); if ( !(idata->reopen & IMAP_EXPUNGE_PENDING) && - count < idata->ctx->msgcount) + count < idata->max_msn) { /* Notes 6.0.3 has a tendency to report fewer messages exist than * it should. */ @@ -488,7 +488,7 @@ } /* at least the InterChange server sends EXISTS messages freely, * even when there is no new mail */ - else if (count == idata->ctx->msgcount) + else if (count == idata->max_msn) dprint (3, (debugfile, "cmd_handle_untagged: superfluous EXISTS message.\n")); else @@ -589,27 +589,37 @@ * be reopened at our earliest convenience */ static void cmd_parse_expunge (IMAP_DATA* idata, const char* s) { - int expno, cur; + unsigned int exp_msn, cur; HEADER* h; dprint (2, (debugfile, "Handling EXPUNGE\n")); - expno = atoi (s); + exp_msn = atoi (s); + if (exp_msn < 1 || exp_msn > idata->max_msn) + return; - /* walk headers, zero seqno of expunged message, decrement seqno of those - * above. Possibly we could avoid walking the whole list by resorting - * and guessing a good starting point, but I'm guessing the resort would - * nullify the gains */ - for (cur = 0; cur < idata->ctx->msgcount; cur++) + h = idata->msn_index[exp_msn - 1]; + if (h) { - h = idata->ctx->hdrs[cur]; + /* imap_expunge_mailbox() will rewrite h->index. + * It needs to resort using SORT_ORDER anyway, so setting to INT_MAX + * makes the code simpler and possibly more efficient. */ + h->index = INT_MAX; + HEADER_DATA(h)->msn = 0; + } - if (h->index+1 == expno) - h->index = -1; - else if (h->index+1 > expno) - h->index--; + /* decrement seqno of those above. */ + for (cur = exp_msn; cur < idata->max_msn; cur++) + { + h = idata->msn_index[cur]; + if (h) + HEADER_DATA(h)->msn--; + idata->msn_index[cur - 1] = h; } + idata->msn_index[idata->max_msn - 1] = NULL; + idata->max_msn--; + idata->reopen |= IMAP_EXPUNGE_PENDING; } @@ -619,34 +629,26 @@ * Of course, a lot of code here duplicates code in message.c. */ static void cmd_parse_fetch (IMAP_DATA* idata, char* s) { - int msgno, cur; - HEADER* h = NULL; + unsigned int msn; + HEADER *h; dprint (3, (debugfile, "Handling FETCH\n")); - msgno = atoi (s); - - if (msgno <= idata->ctx->msgcount) - /* see cmd_parse_expunge */ - for (cur = 0; cur < idata->ctx->msgcount; cur++) - { - h = idata->ctx->hdrs[cur]; - - if (h && h->active && h->index+1 == msgno) - { - dprint (2, (debugfile, "Message UID %d updated\n", HEADER_DATA(h)->uid)); - break; - } - - h = NULL; - } - - if (!h) + msn = atoi (s); + if (msn < 1 || msn > idata->max_msn) { dprint (3, (debugfile, "FETCH response ignored for this message\n")); return; } - + + h = idata->msn_index[msn - 1]; + if (!h || !h->active) + { + dprint (3, (debugfile, "FETCH response ignored for this message\n")); + return; + } + + dprint (2, (debugfile, "Message UID %d updated\n", HEADER_DATA(h)->uid)); /* skip FETCH */ s = imap_next_word (s); s = imap_next_word (s); diff -r 4bffaa6d189a -r 77d3173aecff imap/imap.c --- a/imap/imap.c Sat May 13 09:48:28 2017 -0700 +++ b/imap/imap.c Sat May 20 18:52:18 2017 -0700 @@ -250,16 +250,21 @@ { HEADER* h; int i, cacheno; + short old_sort; #ifdef USE_HCACHE idata->hcache = imap_hcache_open (idata, NULL); #endif + old_sort = Sort; + Sort = SORT_ORDER; + mutt_sort_headers (idata->ctx, 0); + for (i = 0; i < idata->ctx->msgcount; i++) { h = idata->ctx->hdrs[i]; - if (h->index == -1) + if (h->index == INT_MAX) { dprint (2, (debugfile, "Expunging message UID %d.\n", HEADER_DATA (h)->uid)); @@ -284,6 +289,8 @@ imap_free_header_data ((IMAP_HEADER_DATA**)&h->data); } + else + h->index = i; } #if USE_HCACHE @@ -293,6 +300,7 @@ /* We may be called on to expunge at any time. We can't rely on the caller * to always know to rethread */ mx_update_tables (idata->ctx, 0); + Sort = old_sort; mutt_sort_headers (idata->ctx, 1); } @@ -600,6 +608,7 @@ idata->status = 0; memset (idata->ctx->rights, 0, sizeof (idata->ctx->rights)); idata->newMailCount = 0; + idata->max_msn = 0; mutt_message (_("Selecting %s..."), idata->mailbox); imap_munge_mbox_name (idata, buf, sizeof(buf), idata->mailbox); @@ -754,7 +763,7 @@ ctx->v2r = safe_calloc (count, sizeof (int)); ctx->msgcount = 0; - if (count && (imap_read_headers (idata, 0, count-1) < 0)) + if (count && (imap_read_headers (idata, 1, count) < 0)) { mutt_error _("Error opening mailbox"); mutt_sleep (1); @@ -1398,6 +1407,8 @@ if (ctx->hdrs[i] && ctx->hdrs[i]->data) imap_free_header_data ((IMAP_HEADER_DATA**)&(ctx->hdrs[i]->data)); hash_destroy (&idata->uid_hash, NULL); + FREE (&idata->msn_index); + idata->msn_index_size = 0; for (i = 0; i < IMAP_CACHE_LEN; i++) { diff -r 4bffaa6d189a -r 77d3173aecff imap/imap_private.h --- a/imap/imap_private.h Sat May 13 09:48:28 2017 -0700 +++ b/imap/imap_private.h Sat May 20 18:52:18 2017 -0700 @@ -212,11 +212,14 @@ char *mailbox; unsigned short check_status; unsigned char reopen; - unsigned int newMailCount; + unsigned int newMailCount; /* Set when EXISTS notifies of new mail */ IMAP_CACHE cache[IMAP_CACHE_LEN]; HASH *uid_hash; unsigned int uid_validity; unsigned int uidnext; + HEADER **msn_index; /* look up headers by (MSN-1) */ + unsigned int msn_index_size; /* allocation size */ + unsigned int max_msn; /* the largest MSN fetched so far */ body_cache_t *bcache; /* all folder flags - system flags AND keywords */ @@ -264,7 +267,7 @@ /* message.c */ void imap_add_keywords (char* s, HEADER* keywords, LIST* mailbox_flags, size_t slen); void imap_free_header_data (IMAP_HEADER_DATA** data); -int imap_read_headers (IMAP_DATA* idata, int msgbegin, int msgend); +int imap_read_headers (IMAP_DATA* idata, unsigned int msn_begin, unsigned int msn_end); char* imap_set_flags (IMAP_DATA* idata, HEADER* h, char* s); int imap_cache_del (IMAP_DATA* idata, HEADER* h); int imap_cache_clean (IMAP_DATA* idata); diff -r 4bffaa6d189a -r 77d3173aecff imap/message.c --- a/imap/message.c Sat May 13 09:48:28 2017 -0700 +++ b/imap/message.c Sat May 20 18:52:18 2017 -0700 @@ -69,23 +69,45 @@ } } +static void imap_alloc_msn_index (IMAP_DATA *idata, unsigned int msn_count) +{ + unsigned int new_size; + + if (msn_count <= idata->msn_index_size) + return; + + /* Add a little padding, like mx_allloc_memory() */ + new_size = msn_count + 25; + + if (!idata->msn_index) + idata->msn_index = safe_calloc (new_size, sizeof (HEADER *)); + else + { + safe_realloc (&idata->msn_index, sizeof (HEADER *) * new_size); + memset (idata->msn_index + idata->msn_index_size, 0, + sizeof (HEADER *) * (new_size - idata->msn_index_size)); + } + + idata->msn_index_size = new_size; +} + /* imap_read_headers: * Changed to read many headers instead of just one. It will return the - * msgno of the last message read. It will return a value other than - * msgend if mail comes in while downloading headers (in theory). + * msn of the last message read. It will return a value other than + * msn_end if mail comes in while downloading headers (in theory). */ -int imap_read_headers (IMAP_DATA* idata, int msgbegin, int msgend) +int imap_read_headers (IMAP_DATA* idata, unsigned int msn_begin, unsigned int msn_end) { CONTEXT* ctx; char *hdrreq = NULL; FILE *fp; char tempfile[_POSIX_PATH_MAX]; - int msgno, idx = msgbegin - 1; + int msgno, idx; IMAP_HEADER h; IMAP_STATUS* status; int rc, mfhrc, oldmsgcount; - int fetchlast = 0; - int maxuid = 0; + int fetch_msn_end = 0; + unsigned int maxuid = 0; static const char * const want_headers = "DATE FROM SUBJECT TO CC MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO LINES LIST-POST X-LABEL"; progress_t progress; int retval = -1; @@ -129,9 +151,11 @@ unlink (tempfile); /* make sure context has room to hold the mailbox */ - while ((msgend) >= idata->ctx->hdrmax) - mx_alloc_memory (idata->ctx); + while (msn_end > ctx->hdrmax) + mx_alloc_memory (ctx); + imap_alloc_msn_index (idata, msn_end); + idx = ctx->msgcount; oldmsgcount = ctx->msgcount; idata->reopen &= ~(IMAP_REOPEN_ALLOW|IMAP_NEWMAIL_PENDING); idata->newMailCount = 0; @@ -139,7 +163,7 @@ #if USE_HCACHE idata->hcache = imap_hcache_open (idata, NULL); - if (idata->hcache && !msgbegin) + if (idata->hcache && (msn_begin == 1)) { uid_validity = mutt_hcache_fetch_raw (idata->hcache, "/UIDVALIDITY", imap_hcache_keylen); puidnext = mutt_hcache_fetch_raw (idata->hcache, "/UIDNEXT", imap_hcache_keylen); @@ -157,7 +181,7 @@ /* L10N: Comparing the cached data with the IMAP server's data */ mutt_progress_init (&progress, _("Evaluating cache..."), - MUTT_PROGRESS_MSG, ReadInc, msgend + 1); + MUTT_PROGRESS_MSG, ReadInc, msn_end); snprintf (buf, sizeof (buf), "UID FETCH 1:%u (UID FLAGS)", uidnext - 1); @@ -165,47 +189,59 @@ imap_cmd_start (idata, buf); rc = IMAP_CMD_CONTINUE; - for (msgno = msgbegin; rc == IMAP_CMD_CONTINUE; msgno++) + for (msgno = 1; rc == IMAP_CMD_CONTINUE; msgno++) { - mutt_progress_update (&progress, msgno + 1, -1); + mutt_progress_update (&progress, msgno, -1); memset (&h, 0, sizeof (h)); h.data = safe_calloc (1, sizeof (IMAP_HEADER_DATA)); do { - mfhrc = 0; + mfhrc = -1; rc = imap_cmd_step (idata); if (rc != IMAP_CMD_CONTINUE) - { - imap_free_header_data (&h.data); break; - } /* hole in the header cache */ if (!evalhc) continue; - if ((mfhrc = msg_fetch_header (ctx, &h, idata->buf, NULL)) == -1) + if ((mfhrc = msg_fetch_header (ctx, &h, idata->buf, NULL)) < 0) continue; - else if (mfhrc < 0) - { - imap_free_header_data (&h.data); - break; - } if (!h.data->uid) { dprint (2, (debugfile, "imap_read_headers: skipping hcache FETCH " - "response for unknown message number %d\n", h.sid)); - mfhrc = -1; + "response for message number %d missing a UID\n", h.data->msn)); continue; } - idx++; + if (h.data->msn < 1 || h.data->msn > msn_end) + { + dprint (1, (debugfile, "imap_read_headers: skipping hcache FETCH " + "response for unknown message number %d\n", h.data->msn)); + continue; + } + + if (idata->msn_index[h.data->msn - 1]) + { + dprint (2, (debugfile, "imap_read_headers: skipping hcache FETCH " + "for duplicate message %d\n", h.data->msn)); + continue; + } + ctx->hdrs[idx] = imap_hcache_get (idata, h.data->uid); if (ctx->hdrs[idx]) { + /* TODO: This assumes the results arrive in ascending MSN order. + * That is not guaranteed, but the code already has that + * assumption. */ + msn_begin = MAX (msn_begin, h.data->msn + 1); + + idata->max_msn = MAX (idata->max_msn, h.data->msn); + idata->msn_index[h.data->msn - 1] = ctx->hdrs[idx]; + ctx->hdrs[idx]->index = idx; /* messages which have not been expunged are ACTIVE (borrowed from mh * folders) */ @@ -221,151 +257,149 @@ ctx->msgcount++; ctx->size += ctx->hdrs[idx]->content->length; + + h.data = NULL; + idx++; } else { /* bad header in the cache, we'll have to refetch. */ - dprint (3, (debugfile, "bad cache entry at %d, giving up\n", h.sid - 1)); - imap_free_header_data(&h.data); + dprint (3, (debugfile, "bad cache entry at MSN %d, giving up\n", h.data->msn)); evalhc = 0; - idx--; } } - while (rc != IMAP_CMD_OK && mfhrc == -1); - if (rc == IMAP_CMD_OK) - break; + while (mfhrc == -1); + + imap_free_header_data (&h.data); + if ((mfhrc < -1) || ((rc != IMAP_CMD_CONTINUE) && (rc != IMAP_CMD_OK))) { - imap_free_header_data (&h.data); imap_hcache_close (idata); goto error_out_1; } } - /* could also look for first null header in case hcache is holey */ - msgbegin = ctx->msgcount; } #endif /* USE_HCACHE */ mutt_progress_init (&progress, _("Fetching message headers..."), - MUTT_PROGRESS_MSG, ReadInc, msgend + 1); + MUTT_PROGRESS_MSG, ReadInc, msn_end); - for (msgno = msgbegin; msgno <= msgend ; msgno++) + while (msn_begin <= msn_end && fetch_msn_end < msn_end) { - mutt_progress_update (&progress, msgno + 1, -1); + char *cmd; - /* we may get notification of new mail while fetching headers */ - if (msgno + 1 > fetchlast) + fetch_msn_end = msn_end; + safe_asprintf (&cmd, "FETCH %d:%d (UID FLAGS INTERNALDATE RFC822.SIZE %s)", + msn_begin, fetch_msn_end, hdrreq); + imap_cmd_start (idata, cmd); + FREE (&cmd); + + rc = IMAP_CMD_CONTINUE; + for (msgno = msn_begin; rc == IMAP_CMD_CONTINUE; msgno++) { - char *cmd; - - fetchlast = msgend + 1; - safe_asprintf (&cmd, "FETCH %d:%d (UID FLAGS INTERNALDATE RFC822.SIZE %s)", - msgno + 1, fetchlast, hdrreq); - imap_cmd_start (idata, cmd); - FREE (&cmd); - } - - rewind (fp); - memset (&h, 0, sizeof (h)); - h.data = safe_calloc (1, sizeof (IMAP_HEADER_DATA)); - - /* this DO loop does two things: - * 1. handles untagged messages, so we can try again on the same msg - * 2. fetches the tagged response at the end of the last message. - */ - do - { - mfhrc = 0; - - rc = imap_cmd_step (idata); - if (rc != IMAP_CMD_CONTINUE) - break; - - if ((mfhrc = msg_fetch_header (ctx, &h, idata->buf, fp)) == -1) - continue; - else if (mfhrc < 0) - break; - - if (!ftello (fp)) - { - dprint (2, (debugfile, "msg_fetch_header: ignoring fetch response with no body\n")); - mfhrc = -1; - msgend--; - continue; - } - - /* make sure we don't get remnants from older larger message headers */ - fputs ("\n\n", fp); - - idx++; - if (idx > msgend) - { - dprint (1, (debugfile, "imap_read_headers: skipping FETCH response for " - "unknown message number %d\n", h.sid)); - mfhrc = -1; - idx--; - continue; - } - /* May receive FLAGS updates in a separate untagged response (#2935) */ - if (idx < ctx->msgcount) - { - dprint (2, (debugfile, "imap_read_headers: message %d is not new\n", - h.sid)); - idx--; - continue; - } - - ctx->hdrs[idx] = mutt_new_header (); - - ctx->hdrs[idx]->index = h.sid - 1; - /* messages which have not been expunged are ACTIVE (borrowed from mh - * folders) */ - ctx->hdrs[idx]->active = 1; - ctx->hdrs[idx]->read = h.data->read; - ctx->hdrs[idx]->old = h.data->old; - ctx->hdrs[idx]->deleted = h.data->deleted; - ctx->hdrs[idx]->flagged = h.data->flagged; - ctx->hdrs[idx]->replied = h.data->replied; - ctx->hdrs[idx]->changed = h.data->changed; - ctx->hdrs[idx]->received = h.received; - ctx->hdrs[idx]->data = (void *) (h.data); - - if (maxuid < h.data->uid) - maxuid = h.data->uid; + mutt_progress_update (&progress, msgno, -1); rewind (fp); - /* NOTE: if Date: header is missing, mutt_read_rfc822_header depends - * on h.received being set */ - ctx->hdrs[idx]->env = mutt_read_rfc822_header (fp, ctx->hdrs[idx], - 0, 0); - /* content built as a side-effect of mutt_read_rfc822_header */ - ctx->hdrs[idx]->content->length = h.content_length; - ctx->size += h.content_length; + memset (&h, 0, sizeof (h)); + h.data = safe_calloc (1, sizeof (IMAP_HEADER_DATA)); + + /* this DO loop does two things: + * 1. handles untagged messages, so we can try again on the same msg + * 2. fetches the tagged response at the end of the last message. + */ + do + { + rc = imap_cmd_step (idata); + if (rc != IMAP_CMD_CONTINUE) + break; + + if ((mfhrc = msg_fetch_header (ctx, &h, idata->buf, fp)) < 0) + continue; + + if (!ftello (fp)) + { + dprint (2, (debugfile, "msg_fetch_header: ignoring fetch response with no body\n")); + continue; + } + + /* make sure we don't get remnants from older larger message headers */ + fputs ("\n\n", fp); + + if (h.data->msn < msn_begin || h.data->msn > fetch_msn_end) + { + dprint (1, (debugfile, "imap_read_headers: skipping FETCH response for " + "unknown message number %d\n", h.data->msn)); + continue; + } + + /* May receive FLAGS updates in a separate untagged response (#2935) */ + if (idata->msn_index[h.data->msn - 1]) + { + dprint (2, (debugfile, "imap_read_headers: skipping FETCH response for " + "duplicate message %d\n", h.data->msn)); + continue; + } + + ctx->hdrs[idx] = mutt_new_header (); + + idata->max_msn = MAX (idata->max_msn, h.data->msn); + idata->msn_index[h.data->msn - 1] = ctx->hdrs[idx]; + + ctx->hdrs[idx]->index = idx; + /* messages which have not been expunged are ACTIVE (borrowed from mh + * folders) */ + ctx->hdrs[idx]->active = 1; + ctx->hdrs[idx]->read = h.data->read; + ctx->hdrs[idx]->old = h.data->old; + ctx->hdrs[idx]->deleted = h.data->deleted; + ctx->hdrs[idx]->flagged = h.data->flagged; + ctx->hdrs[idx]->replied = h.data->replied; + ctx->hdrs[idx]->changed = h.data->changed; + ctx->hdrs[idx]->received = h.received; + ctx->hdrs[idx]->data = (void *) (h.data); + + if (maxuid < h.data->uid) + maxuid = h.data->uid; + + rewind (fp); + /* NOTE: if Date: header is missing, mutt_read_rfc822_header depends + * on h.received being set */ + ctx->hdrs[idx]->env = mutt_read_rfc822_header (fp, ctx->hdrs[idx], + 0, 0); + /* content built as a side-effect of mutt_read_rfc822_header */ + ctx->hdrs[idx]->content->length = h.content_length; + ctx->size += h.content_length; #if USE_HCACHE - imap_hcache_put (idata, ctx->hdrs[idx]); + imap_hcache_put (idata, ctx->hdrs[idx]); #endif /* USE_HCACHE */ - ctx->msgcount++; - } - while ((rc != IMAP_CMD_OK) && ((mfhrc == -1) || - ((msgno + 1) >= fetchlast))); + ctx->msgcount++; - if ((mfhrc < -1) || ((rc != IMAP_CMD_CONTINUE) && (rc != IMAP_CMD_OK))) - { + h.data = NULL; + idx++; + } + while (mfhrc == -1); + imap_free_header_data (&h.data); + + if ((mfhrc < -1) || ((rc != IMAP_CMD_CONTINUE) && (rc != IMAP_CMD_OK))) + { #if USE_HCACHE - imap_hcache_close (idata); + imap_hcache_close (idata); #endif - goto error_out_1; + goto error_out_1; + } } /* in case we get new mail while fetching the headers */ if (idata->reopen & IMAP_NEWMAIL_PENDING) { - msgend = idata->newMailCount - 1; - while ((msgend) >= ctx->hdrmax) - mx_alloc_memory (ctx); + msn_begin = fetch_msn_end + 1; + msn_end = idata->newMailCount; + while (msn_end > ctx->hdrmax) + mx_alloc_memory (ctx); + imap_alloc_msn_index (idata, msn_end); idata->reopen &= ~IMAP_NEWMAIL_PENDING; idata->newMailCount = 0; } @@ -392,6 +426,8 @@ if (ctx->msgcount > oldmsgcount) { + /* TODO: it's not clear to me why we are calling mx_alloc_memory + * yet again. */ mx_alloc_memory(ctx); mx_update_context (ctx, ctx->msgcount - oldmsgcount); imap_update_context (idata, oldmsgcount); @@ -399,7 +435,7 @@ idata->reopen |= IMAP_REOPEN_ALLOW; - retval = msgend; + retval = msn_end; error_out_1: safe_fclose (&fp); @@ -1131,6 +1167,7 @@ IMAP_DATA* idata; long bytes; int rc = -1; /* default now is that string isn't FETCH response*/ + int parse_rc; idata = (IMAP_DATA*) ctx->data; @@ -1139,7 +1176,7 @@ /* skip to message number */ buf = imap_next_word (buf); - h->sid = atoi (buf); + h->data->msn = atoi (buf); /* find FETCH tag */ buf = imap_next_word (buf); @@ -1153,7 +1190,10 @@ /* FIXME: current implementation - call msg_parse_fetch - if it returns -2, * read header lines and call it again. Silly. */ - if ((rc = msg_parse_fetch (h, buf)) != -2 || !fp) + parse_rc = msg_parse_fetch (h, buf); + if (!parse_rc) + return 0; + if (parse_rc != -2 || !fp) return rc; if (imap_get_literal_count (buf, &bytes) == 0) @@ -1180,7 +1220,13 @@ return rc; } -/* msg_parse_fetch: handle headers returned from header fetch */ +/* msg_parse_fetch: handle headers returned from header fetch. + * Returns: + * 0 on success + * -1 if the string is corrupted + * -2 if the fetch contains a body or header lines + * that still need to be parsed. + */ static int msg_parse_fetch (IMAP_HEADER *h, char *s) { char tmp[SHORT_STRING]; diff -r 4bffaa6d189a -r 77d3173aecff imap/message.h --- a/imap/message.h Sat May 13 09:48:28 2017 -0700 +++ b/imap/message.h Sat May 20 18:52:18 2017 -0700 @@ -37,13 +37,12 @@ unsigned int parsed : 1; unsigned int uid; /* 32-bit Message UID */ + unsigned int msn; /* Message Sequence Number */ LIST *keywords; } IMAP_HEADER_DATA; typedef struct { - unsigned int sid; - IMAP_HEADER_DATA* data; time_t received;
