Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
On Thu, Jul 31, 2014 at 4:18 PM, Gregory Maxwell wrote: > On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley wrote: >>> the FEC still lets you fill in the missing transactions without knowing in >>> advance all that will be missing. >> >> I don't see why we need to solve that problem, since the protoco

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Gregory Maxwell
On Thu, Jul 31, 2014 at 3:27 PM, Kaz Wesley wrote: >> the FEC still lets you fill in the missing transactions without knowing in >> advance all that will be missing. > > I don't see why we need to solve that problem, since the protocol > already involves exchanging the information necessary to de

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
> the FEC still lets you fill in the missing transactions without knowing in > advance all that will be missing. I don't see why we need to solve that problem, since the protocol already involves exchanging the information necessary to determine (with some false positives) what a peer is missing,

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Gregory Maxwell
On Thu, Jul 31, 2014 at 2:41 PM, Kaz Wesley wrote: >> the need to have transmitted the transaction list [..] first > > 32 bits per transaction is at least double the communication overhead > of the simple approach, and only offers a bound on the probability of > needing a round trip. "(e.g. from

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
> the need to have transmitted the transaction list [..] first 32 bits per transaction is at least double the communication overhead of the simple approach, and only offers a bound on the probability of needing a round trip. On Thu, Jul 31, 2014 at 2:29 PM, Gregory Maxwell wrote: > On Thu, Jul 3

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Gregory Maxwell
On Thu, Jul 31, 2014 at 1:47 PM, Kaz Wesley wrote: > trip to request the missing tx; if we could somehow get the "What's > the Difference" approach to effectively operate on full transactions > instead I explain how to do this on the network block coding page. Given that you know the sizes and o

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-31 Thread Kaz Wesley
I don't see how set reconciliation alone would be practical for condensed block exchange -- if the keys are txids it'd require a round trip to request the missing tx; if we could somehow get the "What's the Difference" approach to effectively operate on full transactions instead, the log(keysize) f

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-19 Thread Wladimir
On Fri, Jul 18, 2014 at 4:53 PM, Gavin Andresen wrote: > Two more half-baked thoughts: > > We should be able to assume that the majority of transaction data (except > for coinbase) has already been propagated. As Jeff said, incentivizing nodes > to propagate transactions is a very good thing (the

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Gregory Maxwell
On Fri, Jul 18, 2014 at 8:06 PM, Emin Gün Sirer wrote: > >> Most things I've seen working in this space are attempting to minimize >> the data transfered. At least for the miner-interested case the round >> complexity is much more important because a single RTT is enough to >> basically send the w

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Emin Gün Sirer
> Most things I've seen working in this space are attempting to minimize > the data transfered. At least for the miner-interested case the round > complexity is much more important because a single RTT is enough to > basically send the whole block on a lot of very relevant paths. Agreed. Yaron's s

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Gregory Maxwell
On Fri, Jul 18, 2014 at 5:54 PM, Emin Gün Sirer wrote: > The problem being tackled here is very similar to "set reconciliation," > where > peer A thinks that the set of transactions that should be in the block is > S_A, Most things I've seen working in this space are attempting to minimize the da

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Emin Gün Sirer
I thought I'd chime in and point out some research results that might help. Even if they don't, there is a cool underlying technique that some of you might find interesting. The problem being tackled here is very similar to "set reconciliation," where peer A thinks that the set of transactions tha

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
Related: We must handle some legitimate miner-privately-mined cases, such as miner payout TXs (outside coinbase) or side chain conditional TXs[1]. [1] https://bitcointalk.org/index.php?topic=676703.msg7682680#msg7682680 On Fri, Jul 18, 2014 at 3:51 PM, Kaz Wesley wrote: > I've updated the gist,

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Kaz Wesley
I've updated the gist, and added an additional proposal that I think meshes well: https://gist.github.com/kazcw/43c97d3924326beca87d#ultra-fast-block-validation sparseblocks + UFBV would tighten the new-block process to this (when txes have been received in advance): - receive block (~2kB for 1000

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Kaz Wesley
That's true, but I think it can be balanced with the usefulness of knowing what messages a node has received. An invack would be sent if N invs have been received without any resulting getdata; since we're keeping track of peer inv order, one message can cover an arbitrarily large series of invs.

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
On a flood-fill network, you don't want to create a storm of "I already have this" replies. On Fri, Jul 18, 2014 at 1:39 PM, Kaz Wesley wrote: > Peers exchanging mempool priority policies is great; that accomplishes > the flexibility in what txes to remember that I was going for with the > forget

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Kaz Wesley
Peers exchanging mempool priority policies is great; that accomplishes the flexibility in what txes to remember that I was going for with the forget-filters, but much more neatly, with less overhead and some side benefits. Here's what I'm picturing now: - exchange priority policies in peer introdu

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
On Fri, Jul 18, 2014 at 10:53 AM, Gavin Andresen wrote: > But if there was some agreed-upon canonical ordering, then it should > theoretically be possible to take shortcuts in the "what order". > > You'd start with setof(transactions I think everybody knows about) > Select some subset, based on mi

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Gavin Andresen
Two more half-baked thoughts: We should be able to assume that the majority of transaction data (except for coinbase) has already been propagated. As Jeff said, incentivizing nodes to propagate transactions is a very good thing (the signature cache already gives a small incentive to miners to prop

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-18 Thread Jeff Garzik
On Thu, Jul 17, 2014 at 6:46 PM, Gavin Andresen wrote: > I'd encourage you to code up a prototype first (or at the same time), in > whatever programming language / networking library you're most familiar > with. +1 > Maybe not even using the existing p2p protocol; there could be a mining-only >

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Gregory Maxwell
On Thu, Jul 17, 2014 at 2:35 PM, Kaz Wesley wrote: > A node should be able to forget invs it has seen without invalidating what > peers > know about its known txes. To allow for this, a node assembles a bloom > filter of Another option would be to just guarantee to keep at least the last N sent i

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Kaz Wesley
I'm moving this design document to a gist so that I can integrate changes as they come up: https://gist.github.com/kazcw/43c97d3924326beca87d One thing that I think is an important improvement over my initial idea is that the bloom filters don't need to be kept around and built up, they can just be

Re: [Bitcoin-development] Squashing redundant tx data in blocks on the wire

2014-07-17 Thread Gavin Andresen
A couple of half-baked thoughts: On Thu, Jul 17, 2014 at 5:35 PM, Kaz Wesley wrote: > If there's support for this proposal, I can begin working on the specific > implementation details, such as the bloom filters, message format, and > capability advertisment, and draft a BIP once I have a concre