Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Hubro
Posfix keeps mails in a binary format in folders under /var/spool/postfix, at least by default. I want to write some tools for searching and filtering by the meta data of a large number (hundreds of thousands) of emails under /var/spool/postfix/deferred. Among other things, I want to find all queu

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Christian Recktenwald
About three years ago I had a pretty similar question: <20130924132623.ga23...@citecs.de> . The answer I got: On Tue, Sep 24, 2013 at 09:37:11AM -0400, Wietse Venema wrote: > > If by chance someone could provide a pointer to the documentation > > of the queue file format this would be appreciate

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Wietse Venema
Queue files are part of the Postfix internal API. Non-Postfix programs that depend on internal details are NOT SUPPORTED and WILL CAUSE LOST MAIL. That saud, there are ways to do this that don't require unsupported usage. Hubro: > I want to write some tools for searching and filtering by the meta

Re: Restrictions for smtp/submission

2017-05-28 Thread Noel Jones
On 5/27/2017 8:05 AM, John Ankarström wrote: > To clarify my questions: > > - Am I correct in my assumptions about `smtp' and `submission'? Yes, submission and smtpd both accept mail from the network using the same protocol and same executable, but with different settings. The expectation is that

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Wietse Venema
Wietse Venema: > $ grep client= /var/log/maillog > May 28 01:22:27 spike postfix/smtpd[74407]: 3wb7Xz3wSczJrP0: > client=molamola.ripe.net[2001:67c:2e8:11::c100:1371] This example assumes that you have "enable_long_queue_ids = yes", so that a queue ID is used only once. Short queue IDs ar

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Hubro
Wietse Venema wrote > That information is already available from the Postfix maillog file. My first attempt at solving this actually relied on going through the log file to find the client IP, but I found out that the line containing "client=..." was frequently missing. If I grep the log file for

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Hubro
I have already made similar scripts, but the issue is that it runs "postcat" and "postsuper" once for every queue ID, so it becomes absolutely unusable when needing to delete tens- or hundreds of thousands of emails. So far I have been lucky in that most of spam scripts send mail with only a few d

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Hubro
A postqueue option that listed all queued mails in JSON with some envelope information like sender IP would be amazing... If I had more free time I would consider trying to patch it in as you suggested. But - I think it will be much less work to write a script that feeds large batches of queue IDs

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Bill Cole
On 28 May 2017, at 19:07, Hubro wrote: > I really, really wish "postcat -e" had a "-" option, like postsuper, that > allowed me to stream queue IDs in through stdin... xargs postcat -e < listofqueuefiles.txt OR { some procedure that spits out target queue filenames } | xargs postcat -e OR { s

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Peter
On 29/05/17 11:07, Hubro wrote: > I have already made similar scripts, but the issue is that it runs "postcat" > and "postsuper" once for every queue ID, so it becomes absolutely unusable > when needing to delete tens- or hundreds of thousands of emails. postcat -e "$(postconf -h queue_directory)/

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Hubro
The problem with that is that you're passing all the mail file paths right in the command line. Say one path is 41 bytes (which they are on my system), filtering 100 000 mails results in 4,1 MB of paths passed to postcat as command line arguments, which is double the limit of my home desktop. That

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Hubro
The problem with that is that you're passing all the mail file paths right in the command line. Say one path is 41 bytes (which they are on my system), filtering 100 000 mails results in 4,1 MB of paths passed to postcat as command line arguments, which is double the limit of my home desktop. Tha

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Peter
On 29/05/17 15:59, Hubro wrote: > The problem with that is that you're passing all the mail file paths right in > the command line. Say one path is 41 bytes (which they are on my system), > filtering 100 000 mails results in 4,1 MB of paths passed to postcat as > command line arguments, which is do

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Peter
On 29/05/17 16:57, Peter wrote: > find "$(postconf -h queue_directory)/deferred/)" -type f -exec postcat > -e {} + | your_program | postsuper -d - Oops, typo there, should be: find "$(postconf -h queue_directory)/deferred/" -type f -exec postcat -e {} + | your_program | postsuper -d - Peter

Re: Is there any documentation on the binary format of the mail files under /var/spool/postfix/ ?

2017-05-28 Thread Peter
On 29/05/17 16:00, Hubro wrote: > The problem with that is that you're passing all the mail file paths right in > the command line. No, he's not, go look up the xargs man page and see what it does. It's basically a variation on the find solution I just gave you. Peter