Skip to content

cam6_4_181: Modified config_archive.xml to archive CAM+DART files. #1302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cacraigucar merged 8 commits into from
Jun 11, 2026
Merged

cam6_4_181: Modified config_archive.xml to archive CAM+DART files. #1302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f69921e
Added code to make st-archive handle DART files
kdraeder Apr 22, 2025
1084a90
Draft of entry to satisfy development workflow recommendations
kdraeder Apr 22, 2025
a42315e
Added line to archive .i. files as both restarts and history
kdraeder May 27, 2025
1f31cd7
resolve merge conflicts
jedwards4b Jun 24, 2025
e268bbc
still trying to fix ChangeLog
jedwards4b Jun 24, 2025
b0cfb07
Merge tag 'cam6_4_163' into cam_st-arch_dart
cacraigucar Apr 14, 2026
ca405db
Merge tag 'cam6_4_180' into cam_st-arch_dart
cacraigucar Jun 10, 2026
5989ce1
Update ChangeLog for cam6_4_181
cacraigucar Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions cime_config/config_archive.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@
<rest_file_extension>r</rest_file_extension>
<rest_file_extension>rh\d*</rest_file_extension>
<rest_file_extension>rs</rest_file_extension>
<hist_file_extension>h\d*.*\.nc$</hist_file_extension>
<hist_file_extension>i\..*\.nc$</hist_file_extension>
Comment on lines -7 to -8

@cacraigucar cacraigucar Apr 24, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering about the implications of removing this line: <hist_file_extension>i\..*\.nc$</hist_file_extension> on other CAM jobs. I will admit that I am by no means an expert on this file and archiving in general, but it appears that we'd no longer be archiving intantaneous history files? Should this line perhaps remain?

@kdraeder kdraeder Apr 24, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That line was replaced by a <rest_file_extension>i\..* line.
This line handles the CAM "initial file", which is the only '.i.' file that I know of.
There may be '.i[a-z0-9]+.' files (someday), but they would be handled differently.
But I would like to hear the opinion of CAM developers about this.
DART needs to have the .i. files archived with the restarts because that's the file
containing the model state which DART needs to use to start CAM at the beginning of each DA cycle.

@cecilehannay cecilehannay Apr 24, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to be sure we are talking about the same thing.
instantaneous files are the .h0i., .h1i., ... which is different from the initial file .i..

Where will these files go?

@kdraeder kdraeder Apr 24, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that the .h[0-9]i. files will still be handled by the xlm line:
<hist_file_extension>h\d*.*\.nc(\.gz)?$</hist_file_extension>
which I parse as "h+any_digit+any_number_of_any_characters".

The '.i.' file is currently handled by the line:
<hist_file_extension>i\.\d.*\.nc(\.gz)?$</hist_file_extension>
I believe that the parsing code adds '.' before the regex listed in these .xml files,
so this line only applies to '.i.' and not to the .h[0-9]i. files.

@swrneale swrneale Apr 25, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto @cecilehannay . We need to know if this change moves the location of history output and breaks everyone's analysis scripts. Fixable, yes, but it would be good to know in advance.

@kdraeder kdraeder Apr 25, 2025

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my test the .h0i. files ended up in archive/atm/hist, archive/rest/$date, and in $rundir.
I only requested 1 CAM history file.
The .i. files ended up in archive/rest/$date and $rundir.
That's what I intended, but someone else should test it too, especially if I failed to create a CAM file type.
The archive directory is /glade/derecho/scratch/raeder/St_BHISTC_LTso-SE_st-arch/archive.
Thanks for looking into this!

@cacraigucar cacraigucar Apr 25, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cecilehannay - Would you be able to checkout @kdraeder's branch and make a quick run similar to ones you are currently making and make sure all that all the history files end up where they should?

@cacraigucar cacraigucar Apr 28, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cecilehannay - I have two CAM checkouts for you to test:

  • /glade/derecho/scratch/cacraig/cam6_4_089 - A straight checkout of cam6_4_089
  • /glade/derecho/scratch/cacraig/cam6_4_089_raeder - A checkout of cam6_4_089 with the DART changes

Please let us know what your runs indicate for both the history and the initial condition files (and any other files for that matter if they change)

@cacraigucar cacraigucar Apr 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cecilehannay - I have merged up to the most recent CAM (cam6_4_163).

For your comparison runs I have two derecho directories you can use:

  • /glade/derecho/scratch/cacraig/cam6_4_163_raeder
  • /glade/derecho/scratch/cacraig/cam6_4_163

<rest_file_extension>i\.\d.*\.nc(\.gz)?$</rest_file_extension>
<hist_file_extension>i\..*\.nc(\.gz)?$</hist_file_extension>
<hist_file_extension>h\d*.*\.nc(\.gz)?$</hist_file_extension>
<hist_file_extension>e</hist_file_extension>
<rest_history_varname>nhfil</rest_history_varname>
<rpointer>
Expand Down
48 changes: 48 additions & 0 deletions doc/ChangeLog
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,51 @@

===============================================================

Tag name: cam6_4_181
Originator(s): kdraeder, cacraigucar
Date: June 11, 2026
One-line Summary: Modified config_archive.xml to archive CAM+DART files
Github PR URL: https://github.com/ESCOMP/CAM/pull/1302

Purpose of changes (include the issue number and title text for each relevant GitHub issue):
- Make st_archive archive CAM+DART output files: https://github.com/ESCOMP/CAM/issues/1301

Describe any changes made to build system: N/A

Describe any changes made to the namelist: N/A

List any changes to the defaults for the boundary datasets: N/A

Describe any substantial timing or memory changes: N/A

Code reviewed by: cacraigucar, cecilehannay

List all files eliminated: N/A

List all files added and what they do: N/A

List all existing files that have been modified, and describe the changes:
M cime_config/config_archive.xml
- modify how DART files are archived

M src/dynamics/mpas/dycore
- commit missing dycore external update
If there were any failures reported from running test_driver.sh on any test
platform, and checkin with these failures has been OK'd by the gatekeeper,
then copy the lines from the td.*.status files for the failed tests to the
appropriate machine below. All failed tests must be justified.

derecho/intel/aux_cam: all BFB

derecho/nvhpc/aux_cam: all BFB

izumi/nag/aux_cam:
FAIL ERC_D_Ln9.f10_f10_mt232.FHIST_C5.izumi_nag.cam-outfrq3s_subcol COMPARE_base_rest
- pre-existing failure -- see https://github.com/ESCOMP/CAM/issues/1514

izumi/gnu/aux_cam: all BFB

===============================================================
===============================================================

Tag name: cam6_4_180
Expand Down
Loading