Codename: apt-fetcher Mentors: Michael Vogt, David Kalnischkies Synopsis: The Debian Archive currently stores different types of package metadata, which is handled by individual package management applications. While this approach is currently working, it's not scalable, as different types of metadata will be added to the Archive e.g. the AppStream project.
Purpose: To unify the metadata acquire process and make it part of APT's functionality. This way, separate package management applications - e.g. apt-file, debtags - will only have to register their specific acquire plugin in APT, and then make use of the fetched package metadata, acquired through "apt-get update". One of the side purposes of the project is to provide a public parser for the sources.list file, that is exported from libapt. More details on the project proposal page: [0]. In the community bonding period I've mostly been doing research and browsing APT's source code. APT is one of Debian's fundamental components and it was a great thrill seeing how all its components work together. It's probably one of Debian's most tested packages, and its developers' expertise shows in the quality of the code. Working with this code is a great challenge to me. I haven't talked that much with my mentors, as it was my finals period at uni - I've done most of the work on my own. At the beginning of the coding period I started a wiki page, to present the proposed design as it was slowly evolving and have a written base for the discussions with the mentors: [1]. The first issue I had in mind was a way to enhance the sources.list file(s) with additional metadata information, for the future developed plugins, while keeping backwards compatibility. I've studied the sources.list man page, and I saw that the format supports, for each entry, an optional number of options, which can be silently ignored if not understood. David suggested using this insertion point for the enhancements, rather than defining new type of entry - my initial idea. Once I got this figured, it was easy to see how most of the project will come together in the future: the public parser will provide the sources.list info in a structured way to the pluggable acquire framework, which will decide which plugin to use for metadata fetching, based on the entry type and defined options. Metadata types can be activated for each entry through boolean values passed as options. The first task I had to work at, as suggested by the mentors, was the public parser. David suggested I should focus on different types of iteration through the metadata sources, and Michael advised me to take into account future compatibility with different formats for the sources.list file(s) - changing the format of these files should not result into refactoring the whole parser. Having these thoughts in mind, and a parser interface, I started working on predicate based sources iteration. After this, I made it possible to provide a read method for the public parser - this way, new types of sources can be supported, such as apt-rpm sources or rfc822 sources. I've made the parser sort of "pluggable" as well. Implementing the standard source format support was the next step - this was mostly porting the code from the existing APT package. I've had a little trouble getting used to C++ again - I haven't used it that much for the last 2 years. This slowed the development process for a bit, especially when it came to compiling and testing what I've done. But I managed to do that eventually, I've written a few tests as well, and it works as expected! I was very excited when my first test passed, and I already saw myself on the way of building code that will be used by millions. After some time doing research on my own, trying to figure out what's there and how it can be improved, this was a great reward. The parser is far from being complete: I must refactor it taking into account forward ABI compatibility (as suggested by Michael), test it thoroughly, and who know what else might come up on the way. At this point, I've done some of the project design and I've brought a component in a functional state. To sum up, here are the first two weeks' summary (and, more or less, the community bonding period): What I've done: - configured the build environment, the project repo [2], and the desing page wiki [1] - refreshed my C++ skills - designed the public sources.list parser and (partly) the pluggable acquire framework (all though I haven't written it down yet) - implemented the parser and a few tests for it What problems I've run into: - understanding the present APT code - not really a problem, but I can't say it was easy doing it - "getting back in touch" with C++ - bugs, bugs, bugs - designing is hard when you don't fully understand how a complex project works What I plan to do further: - think of some more tests for the parser and refactor it to be forward ABI compatible - design and implement the next component: the pluggable acquire framework - interface, generic plugin model, etc. As a conclusion, starting a project is the hardest part, and this applied to apt-fetcher. Fortunately, Michael and David had the right piece of advice every time I asked them for it, and their straightforward guildelines made development so much easier - thanks a lot! I'm really anxious to see what this project will evolve into. My contributions to the APT package can be found in the repo [2]: the header file [3], the implementation file [4] and the tester [5]. Bogdan Purcareata [0] http://wiki.debian.org/SummerOfCode2012/Projects#Pluggable_acquire-system_for_APT [1] http://wiki.debian.org/BogdanPurcareata/PluggableAptBackend [2] https://launchpad.net/apt-fetcher [3] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/parser.h [4] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/parser.cc [5] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/test/parser/parser_tester.cc _______________________________________________ Soc-coordination mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/soc-coordination
