Re: [DISCUSS][FORMAT] Data Integrity

2019-07-15 Thread Antoine Pitrou
Le 15/07/2019 à 16:15, Wes McKinney a écrit : > If we adopt the position (as we already are in practice, I think) that > the encapsulated IPC message format is the main way that we expose > data from one process to another, then having digests at the message > level seems like the simplest and mo

Re: [DISCUSS][FORMAT] Data Integrity

2019-07-15 Thread Wes McKinney
If we adopt the position (as we already are in practice, I think) that the encapsulated IPC message format is the main way that we expose data from one process to another, then having digests at the message level seems like the simplest and most useful thing. FWIW, the Parquet format technically p

Re: [DISCUSS][FORMAT] Data Integrity

2019-07-12 Thread Antoine Pitrou
Le 12/07/2019 à 09:56, Micah Kornfield a écrit : > Per Antoine's recommendation. I'm splitting off the discussion about data > integrity from the previous e-mail thread about the format additions [1]. > To re-cap I made a proposal including data integrity [2] by adding a new > message type to t