On Friday 24 November 2006 21:06, David Boyes wrote: > > Thanks for your thoughts. I think there are several points that you > have > > minimized or overlooked in your response: > > > > 1. Bacula currently permits specifying multiple Media Types in a Pool. > > 2. Bacula currently permits Storage devices to be specified in the Job > > resource > > 3. Bacula currently permits both Pool and Storage overrides in Run > > resources. > > > > None of the above can be removed or changed without causing total > disaster > > to > > a very large Bacula community. > > All understood. I'm still thinking about how to get to that future state > where these things are controlled by Bacula, and the user doesn't have > to concern themselves with it. Since migration is a significant new > feature, it strikes me as a very good opportunity to start heading in > that direction.
Yes, that is a good point that I probably did not consider enough. > > > For version 1.39.x, you can run Bacula exactly as you want using > Storage > > resources in Pools and in the Next Pool. Both methodologies cooexist > in > > Bacula (though they probably don't function very well together). > > Yes. It's a tradeoff of one-time setup complexity for pain-free normal > operation. The bigger problem (as you pointed out) is how to get there > from here. > > > > > > Migration job: > > > > read storage: job storage resource > > > > > > Shouldn't the migration job always be using the Pool resource being > > > migrated > > > > One can do that, but it is not the only way of doing things. The read > > part > > remains compatible with the existing code. > > I guess I'm not seeing why it's useful to override device selection for > reads from pooled volumes -- I'm sure there's a reason, but I can't > figure it out. The volumes (and thus the jobs you are migrating) are in > a pool and are in a changer -- otherwise they wouldn't be eligible for > migration -- thus they are associated implicitly with a SD managing that > pool. Other than academic completeness, why complicate operations by > allowing/encouraging the user to mix into Bacula's device selection and > management process outside of real total disaster? That seems to be the > core of the problem -- Bacula having to cope with things outside it's > control or users doing things that are dumb. Should we use the > opportunity to try to remedy the possibility of that? Migration is a > *major* change in capability -- I guess I'm thinking that it's also the > beginning of the kinds of changes I suggested in order to scale up. Yes, on second thought we probably could do Migration a bit differently. However, unless one wants to overly complicate the existing complicated code, it would either be all or nothing -- that is either accept all possible existing ways of setting the storage device, or do it only from the Pool. If it is done only from the Pool, then modification from the command line would be out, and all flexibility would be lost. Though it is possible, I find adding even more code to restrict devices only for Migration to be a bit of overkill. Maybe I am wrong ... > > I know we can't break the world as it is now. It might be worth breaking > migration to work this way and start things in the direction we want to > go in the future. As alluded to above, it is not so easy to break migration and still maintain some flexibility. It is much easier to just let Bacula do its thing. Migration has been the hardest project I have worked on in Bacula, and it is sort of limping along, but I am not really very happy about it. At the moment I don't see any way to improve it, but I am unhappy because it is rather complicated -- try figuring out what happens when Pools, SQL statements, and Regexes are involved to determine what gets migrated -- it is a brain buster. What worries me more though is that the current implementation is not very scalable. I mentioned this some months ago -- I shudder to think of what will happen when some poor guy fires off a Migration job that spawns 1000 child jobs :-) As I think you said, it will be interesting to see how Bacula stands up to the stress test. :-) I'm a bit tired of working on it, but at the same time would really like to see it working well, and even more, I would really like to see about adding a copy feature, and even more than that a "consolidation" feature (or what some people call a synthetic Full backup -- I think) where the current state of the system will be created from existing backups, but written to one set of tapes ... > > > > For migration, we're looking at > > > volumes in pools, not individual volumes (other than the case where > we > > > have deliberately limited the selection criteria to a specific > volume > > > within a pool via the selection keyword), > > > > No, that is not really correct. For migration, as it is currently > > implemented, we are looking at Jobs. Everything breaks down to a Job. > > This > > has certain constraints, but I could see no other way of implementing > > Migration in the current Bacula otherwise. > > I think this is one of those which-perspective-do-you-start-from issues. > At the lowest level, that's true - jobs are the most granular thing in > Bacula, and that's the units that the migration code deals with. From > the larger-scale storage manager's perspective, jobs are the *contents* > of a volume, and what they're interested is managing the availability of > volumes (empty, partially-full, full). The migration code identifies and > moves the jobs contained on volumes or the volumes contained within a > pool, but what I care about is if I have enough available space on > volumes to meet the requirements. > > > > so there should be no reason > > > to specify individual devices for migration jobs. > > > > You have to be able to properly get to the original data, and I don't > > think it > > would be wise to have one syntax for restores and a different one to > find > > the > > read side of a Migration. > > I probably didn't explain this point well. I intend the syntax to be the > same in both cases, but in the case of a total disaster (where the > bacula configuration and/or database is lost or otherwise unavailable), > you would have to ADD parameters identifying the SD and device to the > normal syntax. In a normal case, the job would get that data from the > Pool definitions (and their associated Storage definitions), and the > user should let Bacula do it's job and select appropriate devices. > > > > > > read storage: pool storage resource > > > > > > Correct. > > > > > > > read storage: run override resource > > > > > > See above. Migration is about pools, not devices. > > > > No, it is about Jobs, with the caveat that Next Pool does define what > set > > of > > devices it can be migrated to. > > See above discussion on perspective. The mechanics in the code are about > jobs, but that's more detail than the average storage manager cares > about. > > > > > > If you need to migrate > > > to volumes that are not in a pool then you set up a temporary pool > and > > > draw it's volumes from the scratch pool. > > > > All Volumes by definition are in one and only one pool, and as I said, > we > > are > > migrating jobs not volumes and not pools. > > OK, my bad explanatory skills again. I was thinking of the case where I > need to dump a bunch of data to tapes that have not been previously used > with Bacula. Since I typically label and assign volumes to specific > pools for specific reasons, a bunch of "generally available" volumes > aren't part of that scheme. If I define a pool for that purpose (say > "Hurricane-Coming-Mass-Export" and allow that pool to draw from the > scratch pool (rather than specifically assigning tapes to it), then > that's what I would use as the NextPool for the migration job. I would > be clearing all jobs on my normal volumes onto the new tapes in the > Hurricane-Coming-Mass-Export pool so I could put the tapes in the > station wagon and flee. In that case *I* don't care about the individual > jobs; I care about whether I have copies of all the data on the normal > volumes -- it's the job of the Migration code (in my view) to worry > about those details while I worry about the physical things I can see > and manage. Hopefully Bacula will be able to do this. I think so, but I don't think we have had enough testing to really know ... > > > > > write storage: pool next pool storage resource > > > > (yea, try to figure out the above in C it is > > > > job->pool->next_pool->storage > > > > > > Correct as I understand the implementation. See comments above. > > > > Yes, but it is terribly complicated, and IMO the "average" user > doesn't > > understand Pools so is going to break his brain on this. > > On the other hand, it boils down to: > > Initial setup of Bacula: > > 1) Define your devices in a Storage resource. > 2) Define a Pool resource to hold your normal backup media and specify > which Storage resource will contain the media for this pool. > 3) Define your normal backup media in the pool. > 4) Define a backup job that references the pool of media by Pool name > 5) run backup job. > > Implementing migration (or replacing the current spooling code): > > 1) If necessary, define a new Storage resource (if your migration pool > uses different hardware than the normal stuff) > 2) Define a new Pool to hold the media you want to use for > migration/consolidation > 3) Define new media volumes in the new Pool. > 4) Update the original Pool with a Next Pool = resource specifying the > new Pool. > 5) Define a migration job for the original Pool with appropriate > selection criteria. > 6) run your migration job periodically. Put that way, it all sounds simple enough, but it assumes you understand the concept of a pool, which I have noticed from reading the email lists is not so easy for most users. > > I think that's pretty clear (or could be made to be that way). Basic job > description specifies the Pool name and Bacula takes it from there. If > you need to go directly to tape for some reason, you override the Pool > resource in the job with the name of the Pool that is associated with > the Storage resource that has the tapes. User doesn't get involved with > devices at all > > It might be interesting to think about making a change to the default > configuration that gets initially installed with the .debs or .rpms to > configure a disk changer and disk pool by default, and then use the > enabling-migration process above to add tape capability. That would make > the Bacula installation immediately useable on install (and with > disk-based backup becoming commonplace, this would be a very handy > thing, particularly when the Windows daemons become mainstream), and > propagate the changes forward w/o breaking the world for the existing > users. One could easily write a tool to do the definition steps for the > tape enablement. This isn't such a bad idea. All the pieces are already there except the Migration job. I'll think about it, though it would be nice if someone else took on this task since I still have a huge amount of documentation work to do. > > Maybe 1.4-ish timeframe to change the initial config? > > > > > > Backup Job: > > > > write storage: job storage resource > > > > > > See above for discussion of pools. If pools drive the selection of > > > volumes, the pool definition will clearly define what SD should be > used, > > > unambiguously and consistently across all actions in Bacula. > > > > Unfortunately the above, however good it may be, is not compatible > with > > the > > existing Bacula (if you force it on the user), and probably more than > 50% > > of > > beginning users will not understand how pools work. Any big > enterprise > > backup expert will, but not the majority of Bacula users, IMO. > > I understand that there are migration issues. Consider this a discussion > of future direction -- there will need to be some explaining done. > > > > > Restore Job: > > > > read storage: who knows, probably the same as > > > > Migration. To be checked. > > > > > > In a normal case, the pool definition of the pool containing the > volume > > > should determine the SD to use. > > This is now possible, but it is not *enforced* so you just need to use > a > > little discipline in writing your .conf file. > > OK. I'll see if I can come up with a cookbook install doc that reflects > this approach and we can kick it around here. As outlined above, it's > (IMHO) pretty straightforward, and would be more amenable to > autogeneration and long term manageability. Yes, I agree with your conclusions, and if you can put something reasonably simple together, I am sure it would be a big help for the users. > > > > See comments above wrt to Migration. The same cases exist; migration > > > just makes this problem more visible. > > > > Yes, it is all a bit too much complicate. Perhaps over time, I can > > deprecate > > features that add to the complexity such as Run overrides, but that is > not > > something that will happen any time soon. > > Understood. I don't think I expected that you'd do it right now; I tend > to think about the "release after next", so I get ahead of the gory > details. > I do think it's worth the discussion, though, especially if we end up > with a much simpler "default" configuration, or a way to autogenerate > same. Certainly we can encourage a lot of much better practices by how the default configuration is setup. Possibly even if we don't do your full suggestion with Migration and all, a default conf file that puts the Storage directive in the Pool would help open user's eyes. Thanks for the input, Kern ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
