Is it easy/non-arcane or indeed automatic for non-experts to add these loops?  Because that's the only way this will be achieved systemically.

Presumably ccp4i2 can be wrangled into making it happen magically. (Apologies if it does already, if so, a comment here would help the discussion...)

Frank



On 18/04/2024 11:02, Randy John Read wrote:
I’d like to add my strong agreement to what Robbie said, but also point out a 
wrinkle. When the PDB runs validation, it just takes the data that are in the 
first reflection loop of the reflections.cif file. So if you want the 
validation statistics to match your reported refinement statistics, that loop 
should contain the set of data you gave the refinement program, especially if 
you’ve done something like apply an elliptical truncation, correct for 
anisotropy, or convert intensities to amplitudes, all of which change the data 
in ways that can’t be reversed later. Whenever you’ve done any of this (and 
many people are using StarAniso these days, which does all of those things), 
please put in a second reflection loop containing the whole set of intensities 
to the highest resolution you used, without any anisotropic scaling or 
elliptical cutoffs. Then anyone wanting to re-refine your structure or check 
your data for artefacts will have more information available.

Of course, I hope we’re moving to a world in which we all also deposit the 
intensities before merging, which in principle allows even more quality control 
to be done.

Best wishes,

Randy Read

On 18 Apr 2024, at 05:04, Robbie Joosten <robbie_joos...@hotmail.com> wrote:

If I may add to that: Please deposit the full dataset, not just the set of 
reflections you end up using. This allows people to use all the data if they 
are interested.

Cheers,
Robbie

On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> wrote:
Hi Matt,
I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod <mjmcleo...@gmail.com>
Sent: Wednesday, April 17, 2024 4:12 PM
To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu>
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Rescale merged data?
Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke <doeke_heks...@harvard.edu> wrote:
Hi Matt,

It would be helpful if you could describe your case in more detail. Do you want 
to change the resolution cutoff after scaling? Do you want to keep more data? 
Fewer? Or do you mean something different such as truncation to generate 
amplitudes, application of anisotropic resolution cutoffs,  or outlier 
rejection? Are you referring to data that were scaled in HKL2000?

Best, Doeke

-----Original Message-----
From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod
Sent: Wednesday, April 17, 2024 3:04 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] Rescale merged data?

Hi all,

I am looking at a old students data and it looks like they didn't properly cut 
off the data during scaling.  All of the files I have appear to be the merged 
.sca (or mtz after converting with scalepacktomtz) - is there a way to 
retruncate the data after merging or do I have to reprocess the data?

Thanks,

########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

--
Matthew Jordan McLeod, PhD
Post-Doctoral Fellow - Cornell University



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> wrote:
Hi Matt,
I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod <mjmcleo...@gmail.com>
Sent: Wednesday, April 17, 2024 4:12 PM
To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu>
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Rescale merged data?
Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke <doeke_heks...@harvard.edu> wrote:
Hi Matt,

It would be helpful if you could describe your case in more detail. Do you want 
to change the resolution cutoff after scaling? Do you want to keep more data? 
Fewer? Or do you mean something different such as truncation to generate 
amplitudes, application of anisotropic resolution cutoffs,  or outlier 
rejection? Are you referring to data that were scaled in HKL2000?

Best, Doeke

-----Original Message-----
From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod
Sent: Wednesday, April 17, 2024 3:04 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] Rescale merged data?

Hi all,

I am looking at a old students data and it looks like they didn't properly cut 
off the data during scaling.  All of the files I have appear to be the merged 
.sca (or mtz after converting with scalepacktomtz) - is there a way to 
retruncate the data after merging or do I have to reprocess the data?

Thanks,

########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

--
Matthew Jordan McLeod, PhD
Post-Doctoral Fellow - Cornell University



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> wrote:
Hi Matt,
I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod <mjmcleo...@gmail.com>
Sent: Wednesday, April 17, 2024 4:12 PM
To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu>
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Rescale merged data?
Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke <doeke_heks...@harvard.edu> wrote:
Hi Matt,

It would be helpful if you could describe your case in more detail. Do you want 
to change the resolution cutoff after scaling? Do you want to keep more data? 
Fewer? Or do you mean something different such as truncation to generate 
amplitudes, application of anisotropic resolution cutoffs,  or outlier 
rejection? Are you referring to data that were scaled in HKL2000?

Best, Doeke

-----Original Message-----
From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod
Sent: Wednesday, April 17, 2024 3:04 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] Rescale merged data?

Hi all,

I am looking at a old students data and it looks like they didn't properly cut 
off the data during scaling.  All of the files I have appear to be the merged 
.sca (or mtz after converting with scalepacktomtz) - is there a way to 
retruncate the data after merging or do I have to reprocess the data?

Thanks,

########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

--
Matthew Jordan McLeod, PhD
Post-Doctoral Fellow - Cornell University



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> wrote:
Hi Matt,
I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod <mjmcleo...@gmail.com>
Sent: Wednesday, April 17, 2024 4:12 PM
To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu>
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Rescale merged data?
Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke <doeke_heks...@harvard.edu> wrote:
Hi Matt,

It would be helpful if you could describe your case in more detail. Do you want 
to change the resolution cutoff after scaling? Do you want to keep more data? 
Fewer? Or do you mean something different such as truncation to generate 
amplitudes, application of anisotropic resolution cutoffs,  or outlier 
rejection? Are you referring to data that were scaled in HKL2000?

Best, Doeke

-----Original Message-----
From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod
Sent: Wednesday, April 17, 2024 3:04 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] Rescale merged data?

Hi all,

I am looking at a old students data and it looks like they didn't properly cut 
off the data during scaling.  All of the files I have appear to be the merged 
.sca (or mtz after converting with scalepacktomtz) - is there a way to 
retruncate the data after merging or do I have to reprocess the data?

Thanks,

########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

--
Matthew Jordan McLeod, PhD
Post-Doctoral Fellow - Cornell University



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

On 17 Apr 2024 22:35, "Hekstra, Doeke Romke" <doeke_heks...@harvard.edu> wrote:
Hi Matt,
I appreciate disagreement and comments from colleagues. My two cents are that it seems unnecessary to repeat scaling and merging, or any earlier step. If you want to remove structure factor amplitudes or merged intensities from the MTZ file you can do so using MTZUTILS or similar functionality in CCP4 (https://www.ccp4.ac.uk/html/mtzutils.html#generalresolution). For refinement, you can specify the desired resolution range in your favorite refinement program. My personal convention is to use CC1/2 = 0.30 as the point to which retain data and <I/sigI> = 2 as the nominal resolution of the dataset. If you have the HKL2000 scaling log, you should be able to retrieve this information. I frankly wish we’d just deposit all data in the PDB rather than truncate based on some criterion or another. Best, Doeke From: Matt Mcleod <mjmcleo...@gmail.com>
Sent: Wednesday, April 17, 2024 4:12 PM
To: Hekstra, Doeke Romke <doeke_heks...@harvard.edu>
Cc: CCP4BB@JISCMAIL.AC.UK
Subject: Re: [ccp4bb] Rescale merged data?
Sure thing. A former student left somewhere between 30-50 datasets but they scaled the data to the detector corners (or maybe edge) in HKL2000. There are many of the high-resolution bins with no reflections in them. He then went forward and merged this data, presumably in HKL2000 again and did his model building/refinement. We now need to re-refine the models against this data for publication but we need a more suitable resolution cutoff for the data. Rather than go back and index/integrate all the data and then rescale the data to a more appropriate place (then merge), I was wondering if there was a way to take the merged reflections as either .sca or .mtz (from scalepacktomtz output) and then rescale to a more appropriate resolution. It doesn't seem like the student left unmerged data. So, nothing fancy (aniostropy etc), there is just a lot of data that needs to be adjusted and I am trying to avoid reprocessing all the frames again. Matt On Wed, 17 Apr 2024 at 15:59, Hekstra, Doeke Romke <doeke_heks...@harvard.edu> wrote:
Hi Matt,

It would be helpful if you could describe your case in more detail. Do you want 
to change the resolution cutoff after scaling? Do you want to keep more data? 
Fewer? Or do you mean something different such as truncation to generate 
amplitudes, application of anisotropic resolution cutoffs,  or outlier 
rejection? Are you referring to data that were scaled in HKL2000?

Best, Doeke

-----Original Message-----
From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> On Behalf Of Matt McLeod
Sent: Wednesday, April 17, 2024 3:04 PM
To: CCP4BB@JISCMAIL.AC.UK
Subject: [ccp4bb] Rescale merged data?

Hi all,

I am looking at a old students data and it looks like they didn't properly cut 
off the data during scaling.  All of the files I have appear to be the merged 
.sca (or mtz after converting with scalepacktomtz) - is there a way to 
retruncate the data after merging or do I have to reprocess the data?

Thanks,

########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

--
Matthew Jordan McLeod, PhD
Post-Doctoral Fellow - Cornell University



To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
-----
Randy J. Read
Department of Haematology, University of Cambridge
Cambridge Institute for Medical Research     Tel: +44 1223 336500
The Keith Peters Building
Hills Road                                                       E-mail: 
rj...@cam.ac.uk
Cambridge CB2 0XY, U.K.                              
www-structmed.cimr.cam.ac.uk


########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/


########################################################################

To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1

This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list 
hosted by www.jiscmail.ac.uk, terms & conditions are available at 
https://www.jiscmail.ac.uk/policyandsecurity/

Reply via email to