I’d like to add my strong agreement to what Robbie said, but also point out a wrinkle. When the PDB runs validation, it just takes the data that are in the first reflection loop of the reflections.cif file. So if you want the validation statistics to match your reported refinement statistics, that loop should contain the set of data you gave the refinement program, especially if you’ve done something like apply an elliptical truncation, correct for anisotropy, or convert intensities to amplitudes, all of which change the data in ways that can’t be reversed later. Whenever you’ve done any of this (and many people are using StarAniso these days, which does all of those things), please put in a second reflection loop containing the whole set of intensities to the highest resolution you used, without any anisotropic scaling or elliptical cutoffs. Then anyone wanting to re-refine your structure or check your data for artefacts will have more information available.
Of course, I hope we’re moving to a world in which we all also deposit the intensities before merging, which in principle allows even more quality control to be done. Best wishes, Randy Read > On 18 Apr 2024, at 05:04, Robbie Joosten <robbie_joos...@hotmail.com> wrote: > > If I may add to that: Please deposit the full dataset, not just the set of > reflections you end up using. This allows people to use all the data if they > are interested. > > Cheers, > Robbie > > On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> > wrote: > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod <mjmcleo...@gmail.com> > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu> > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > <doeke_heks...@harvard.edu> wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -----Original Message----- > From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > Matthew Jordan McLeod, PhD > Post-Doctoral Fellow - Cornell University > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> > wrote: > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod <mjmcleo...@gmail.com> > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu> > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > <doeke_heks...@harvard.edu> wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -----Original Message----- > From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > Matthew Jordan McLeod, PhD > Post-Doctoral Fellow - Cornell University > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> > wrote: > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod <mjmcleo...@gmail.com> > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu> > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > <doeke_heks...@harvard.edu> wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -----Original Message----- > From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > Matthew Jordan McLeod, PhD > Post-Doctoral Fellow - Cornell University > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> > wrote: > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod <mjmcleo...@gmail.com> > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu> > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > <doeke_heks...@harvard.edu> wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -----Original Message----- > From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > Matthew Jordan McLeod, PhD > Post-Doctoral Fellow - Cornell University > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> > wrote: > Hi Matt, > > I appreciate disagreement and comments from colleagues. My two cents are that > it seems unnecessary to repeat scaling and merging, or any earlier step. If > you want to remove structure factor amplitudes or merged intensities from the > MTZ file you can do so using MTZUTILS or similar functionality in CCP4 > (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For > refinement, you can specify the desired resolution range in your favorite > refinement program. > > My personal convention is to use CC1/2 = 0.30 as the point to which retain > data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have > the HKL2000 scaling log, you should be able to retrieve this information. I > frankly wish we’d just deposit all data in the PDB rather than truncate based > on some criterion or another. > > Best, Doeke > > From: Matt Mcleod <mjmcleo...@gmail.com> > Sent: Wednesday, April 17, 2024 4:12 PM > To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu> > Cc: CCP4BB@JISCMAIL.AC.UK > Subject: Re: [ccp4bb] Rescale merged data? > > Sure thing. > > A former student left somewhere between 30-50 datasets but they scaled the > data to the detector corners (or maybe edge) in HKL2000. There are many of > the high-resolution bins with no reflections in them. He then went forward > and merged this data, presumably in HKL2000 again and did his model > building/refinement. We now need to re-refine the models against this data > for publication but we need a more suitable resolution cutoff for the data. > > Rather than go back and index/integrate all the data and then rescale the > data to a more appropriate place (then merge), I was wondering if there was a > way to take the merged reflections as either .sca or .mtz (from > scalepacktomtz output) and then rescale to a more appropriate resolution. It > doesn't seem like the student left unmerged data. > > So, nothing fancy (aniostropy etc), there is just a lot of data that needs to > be adjusted and I am trying to avoid reprocessing all the frames again. > > Matt > > On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke > <doeke_heks...@harvard.edu> wrote: > Hi Matt, > > It would be helpful if you could describe your case in more detail. Do you > want to change the resolution cutoff after scaling? Do you want to keep more > data? Fewer? Or do you mean something different such as truncation to > generate amplitudes, application of anisotropic resolution cutoffs, or > outlier rejection? Are you referring to data that were scaled in HKL2000? > > Best, Doeke > > -----Original Message----- > From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod > Sent: Wednesday, April 17, 2024 3:04 PM > To: CCP4BB@JISCMAIL.AC.UK > Subject: [ccp4bb] Rescale merged data? > > Hi all, > > I am looking at a old students data and it looks like they didn't properly > cut off the data during scaling. All of the files I have appear to be the > merged .sca (or mtz after converting with scalepacktomtz) - is there a way to > retruncate the data after merging or do I have to reprocess the data? > > Thanks, > > ######################################################################## > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing > list hosted by www.jiscmail.ac.uk, terms & conditions are available at > https://www.jiscmail.ac.uk/policyandsecurity/ > > > -- > Matthew Jordan McLeod, PhD > Post-Doctoral Fellow - Cornell University > > > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 > > To unsubscribe from the CCP4BB list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 ----- Randy J. Read Department of Haematology, University of Cambridge Cambridge Institute for Medical Research Tel: +44 1223 336500 The Keith Peters Building Hills Road E-mail: rj...@cam.ac.uk Cambridge CB2 0XY, U.K. www-structmed.cimr.cam.ac.uk ######################################################################## To unsubscribe from the CCP4BB list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1 This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list hosted by www.jiscmail.ac.uk, terms & conditions are available at https://www.jiscmail.ac.uk/policyandsecurity/
