Hello Guix!
Hi Everyone, I am Vijaya Anand, a mathematics undergraduate studying in IITK, India. I am interested in the topics distributed systems and cryptography and have been self-reading on these topics a lot recently. I wanted to participate in GSoC this summer and while browsing some interesting organizations, I came across guix and the project 'Decentralized Substitute Distribution' sparked my interest a lot. I have recently been interacting with pukkamustard, who is mentoring this project, and got to know a handful about this project and how we can distribute substitutes using any p2p network using ERIS. This project matched my interests and I'm very excited to work on this. Hoping to interact with the community and contribute to guix! Thank you Vijaya Anand
Re: [GSoC 23] distributed substitutes, cost of storage
Hi Attila Thanks for the welcome! I agree that the responsibility of re-uploading the blocks back to the network should be with the clients rather than the substitute server. Also I didn't really think about the point about having to pay for the p2p services at some point of time. In this case we will have to pay for the storage of substitutes both on the p2p storage backend as well as for storage in the substitute server am I right? So ideally we will want to eliminate the usage of these substitute servers and shift totally to p2p services and in this case we will have to shift the responsibility of re-uploading the blocks to the clients itself. Also if we dont keep the re-uploading blocks option as default for the users, won't users usually not choose to enable it? Maybe we can keep it on as default and resource conscious users can choose to turn it off? Please let me know your thoughts on these points and I will change the implementation point of my proposal accordingly. Thank you Vijaya Anand On Sun, 26 Mar 2023 at 00:30, Attila Lendvai wrote: > welcome on board Anand! > > > > In case a user requests for a substitute and there is a missing > > block in the decoding process, a HTTP request for block would sent > > to the substitute server and the server will encode the > > corresponding block in real time and push it back into the > > network. The block will be searched again and retrieved. > > > something to consider here: whose responsibility should it be that a > block, that is missing from a p2p network, is (re-)uploaded there? the > clients? or the current substitute server? > > my gut instinct says that it's better if the clients do the (re-)upload of > the blocks. > > in this architecture the substitute server is just another storage > mechanism along the other storage backends (although with a different > reliability characteristics), and it's the clients that are doing the > mirroring/spreading/distribution of the blocks among the various backends. > the clients of course will/should keep the current substitute servers at > the bottom of their list of backends in their configuration. > > this way the load is distributed, and we don't need to add (too much) > extra complexity to the substitute server codebase, and the actors are less > tightly coupled. > > it's another question whether this mirroring should be enabled by default > in the clients. probably it shouldn't, and the project infrastructure > should be running clients where it is turned on. altruistic third parties > could also enable this mirroring feature, and donate their > bandwidth/resources. > > there's an issue with this, though: > > some p2p storage backends will require some form of payment/credentials to > use their resources. arguably, all p2p storage networks that will survive > into the future will have some mechanism to limit the infinite abuse of > their resources. it is to be researched how these payment mechanisms work > on the various p2p networks, and whether it is possible that the Guix > project pays for the storage globally, and then the random clients will > have the necessary credentials to (re-)upload the missing blocks. > > this architecture shouldn't be impossible, because the content is > authenticated by its hash, and if the payment/authorization mechanism is > based on the hashes of the blocks (probably), then any client could > (re-)upload a missing block that was already paid for. > > i'll look into this, especially in the context of Swarm. > > meta: i think such specific discussions should be kept off-list, but the > financing of the storage fees is probably something that should be known > about more widely. > > -- > • attila lendvai > • PGP: 963F 5D5F 45C7 DFCD 0A39 > -- > Every lie is a debt to the truth. > >
Re: [GSoC 23] distributed substitutes, cost of storage
Hi, Sorry for the late reply. So in the case we are running swarm nodes that serves the network and hence help fund the substitute server, we can also use these to also upload eris encoded substitute blocks onto the network am I right? The total cost will thus be cost to run the swarm nodes + storage cost of substitute blocks in the network + cost to run substitute servers - money earned by running swarm nodes. But when we don't run swarm nodes which run to serve the network, the total cost is not really affect right as the cost to run swarm nodes will be lessened and no money will be earned. So in the context of fallback mechanism the user client can send request to the substitute server for the missing block and the substitute server will serve the eris encoded data block back to the user (using HTTP). The responsibility of uploading these missing blocks back to the network is of the third party nodes (which are running to serve the desired content to the network). But how do we send the message to the node to report the missing block on the network? Can it be done by the user itself? "i can dream about a future where there's a social network that is based on digital signatures and encryption, and my Guix client authorizes compiled binaries based on some weighted transitive closure of signatures of my trusted peers"Interesting! In the case of accessing Guix substitutes from p2p network, we ensure authorization by Guix team by making sure the urn of the substitute is the urn mentioned in the narinfo (which we get from a trusted source like the substitute server). So in the case of accessing some random compiled binary from the network, we just need to verify the authority of the document providing the urn of the content? "i value consensual relationships, and that implies that the other party is well informed. and clogging someone's network bandwidth is not an expected behavior from installing a linux distribution." I agree with this point and also since there are already specialised nodes doing the work of uploading blocks onto the network I guess for now it is better to assign the task to them itself. Also I will give a read about how network's charge for uploading data onto them. As you had mentioned before if the payment is associated with a block id, then maybe we could have any random client upload data onto the network (if of course we are able to ship in with Guix itself, the configuration to upload data onto the said network). Vijaya Anand On Mon, 27 Mar 2023 at 2:49 AM Attila Lendvai wrote: > > Also I didn't really think about the point about having to pay for > > the p2p services at some point of time. > > > a quick note here: i forgot to mention that e.g. the Swarm Foundation has > programs for supporting opensource projects. so, chances are high that the > storage needs for Guix would be paid for by the foundation. > > > > In this case we will have to pay for the storage of substitutes both > > on the p2p storage backend as well as for storage in the substitute > > server am I right? > > > well, the substitute servers are currently already operated (and paid for) > by the Guix team. i don't think that p2p storage solutions have reached a > point of maturity that we could rely on them alone. there should definitely > be some time where both infrastructures are running in parallel. somewhere > down the road a choice could be made to stop running the current substitute > servers, but we are far away from that. > > also, running swarm nodes that serve the network can earn money. so, the > cost of running enough swarm nodes to pay for the storage needs of Guix on > the swarm network should be in the same ballpark of the costs of running > the current substitute servers. the storage price will be market based > (this part is just being rolled out on the live network), so it's > reasonable to expect that people will fire up nodes if the storage price go > well above the VPS costs. > > and not all p2p storage networks are made equal. e.g. IPFS is only a > registry of who is serving what. if you want to keep your data alive on > IPFS, then you need to run some nodes and make sure that they are serving > the content that you care about... and bear the costs of running these > nodes. i.e. the DoS attack surface of IPFS is much smaller. (IPFS stores > only the metadata in the DHT (i.e. where is what), while Swarm stores there > the data itself -- they are different architectures with different features) > > (i need to learn more about GNUnet) > > > > So ideally we will want to eliminate the usage of these substitute > > servers and shift totally to p2p services and in this case we will > > have to shift the responsibility of re-uploading the blocks to the > > clients itself. >
