Skip to content

Instructions for Developers

Jump to bottom
HankHerr-NOAA edited this page Mar 18, 2025 · 14 revisions

This wiki is under construction; see GitHub #57

Basic instructions for cloning, building, and unit testing the WRES software, as well as executing it as a standalone with a simple example, are provided in the landing page for this WRES GitHub project as well as this subsection of the wiki, Obtaining and using the WRES as a standalone application. This wiki provides more detailed instructions related to specific needs of developers or links to other wiki pages to provide that information.

Instructions for developing WRES in an IDE

Instructions below are provided for Eclipse and IntelliJ IDEA(community edition), which are the IDEs that have been employed by the WRES team developers. Of those two, Intellij IDEA is recommended. Note that, when setting up Eclipse, choose "Java Developer" profile to be installed.

What version of Java do I need?

Use your package manager to install the latest OpenJDK >17. As described below, the WRES uses a Gradle toolchain, so builds through Gradle will automatically employ the correct version of Java. This local install is necessary in order for the IDE to employ the correct version of Java, or a compatible version, when building through the IDE during development.

How do I prepare my IDE for WRES development?

The WRES uses the Gradle build tool:

https://docs.gradle.org/current/userguide/userguide.html

You will need to install a Gradle plugin for your IDE. It is also recommended that you install tools to help with static analysis, such as SonarLint, and a protocol buffers plugin to work with protocol buffers, which are used for messaging between various modules.

How do I setup a development project in my IDE given I already have a clone of the repository?

In both Eclipse and IntelliJ IDEA, import the WRES project, using your clone, into your IDE. Be sure to import it as a Gradle project.

How should I build the code and unit test the software within the IDEA?

You should delegate to Gradle when building and testing the project in order to ensure accurate/consistent builds. The project may not build accurately when delegating to your IDE.

To select the build environment in IDEA, navigate to Settings -> Build, Execution, Deployment -> Gradle and choose Gradle (default) for Build and run using and Run tests using (tested with IDEA version 2022.3.2 Community Edition). In between a full clean/build, you may elect to use the IDE build tools, in order to achieve improved performance. For example, when developing unit tests, it is quicker to build and run these tests using the IDE.

As of 20230224, the WRES is built against Java 17. This is managed with a Gradle toolchain in the build.gradle of each project. The purpose of the toolchain is to acquire (if needed) and apply a compatible JDK during the build pipeline, rather than delegate to whatever JDK is being used to run the Gradle daemon.

How do I setup code formatting rules employed for WRES development?

To facilitate following the established OWP/WRES guidelines on formatting, there are formatting rules in the repository for Eclipse and IntelliJ IDEA.

For Eclipse users, there are two files to import. Navigate to,

Window -> Preferences -> Java -> Code Style -> Clean Up

and import,

[repository clone]/devUtils/wres_eclipse_cleanup_profile.xml

Then, navigate to,

Window -> Preferences -> Java -> Code Style -> Formatter

and import,

[repository clone]/devUtils/wres_eclipse_format_settings.xml

For intellij idea users, there is one file to import. Navigate to,

File -> Settings -> Editor -> Java

and select the Scheme (Gear button/downarrow),

[repository clone]/devUtils/wres_idea_format_settings.xml

Instructions for running system tests

In addition to unit tests, the WRES includes a suite of system tests declared within the repository under [repository clone]/systests. Those system tests should be run before pushing code that could impact the results of those tests; in other words, any changes except those to the service layer, which is not tested through the system tests.

How do I setup the system testing data?

The system tests require large data sets currently only available from NOAA's Google Drive. Until such time as a publicly available location can be identified, if you don't have access to NOAA's Google Drive and would like the system test data, then post a ticket requesting the system testing data, and we will provide it to you.

The system testing data consists of multiple zip files. Once received, create a directory to hold the data; that directory will be referred to as systests_data, below. Do the following with each file provided by the WRES team:

  1. wres_system_testing_data.nwmVector.[timestamp].tgz: Unpackage the file into the systests_data directory, itself, creating the subdirectory nwmVector.

  2. wres_system_testing_data.[timestamp].tgz: Unpackage the file into the systests_data directory, creating several directories.

  3. wres_system_testing_data.baselineValidation.[timestamp].tgz: Unpackage the file into the systests_data directory, creating subdirectory archive/baselineValidationExample.

  4. wres_system_testing_data.griddedExamples.masked.[timestamp].tgz: Unpackage the file into the systests_data directory, creating griddedExamples. Then do the following:

cd griddedExamples
mv land_ellicott_city_masked.snow land_snow_ellicott_city
mv land_ellicott_city_masked.soilsat land_soilsat_ellicott_city
mv precip_ellicott_city_masked precip_ellicott_city

In addition to the above data, data within the systests/smalldata directory, which is already in the repository, is also employed in system testing.

For each of the standard series of tests, the data employed are as follows:

  • 000 - 400 series: Package 2, above.
  • 500 series: The smalldata directory.
  • 600 - 700 series (excluding 650 series, but including 700 series in full): Package 1, above, in addition to the USGS web service for observations.
  • 650 series: Package 4, above.
  • 800 series: The smalldata directory.
  • 900-912 scenarios: Package 3, above.

Lastly, if provided, the file CB_DRRC2_enspost_gefs_climatology_20170215.full.tgz is to be placed in systests_data/archive and is used for long-run, manual tests stored in the repository.

How do I execute the system tests?

To run the system tests, do the following:

1. Build the release zip files that are employed by the system tests. Run the following command at the command line (for Windows, use ./gradlew.bat):

./gradlew distZip

2. Identify the versions associated with the following built zip files. The core, wres-vis, and wres-writing zip files need to be identified as follows:

  • Core zip file is ./build/distrubtions/wres-[date]-[revision].zip.
  • wres-vis zip file is ./wres-vis/build/distrubtions/wres-[date]-[revision].zip
  • wres-writing zip file is ./wres-writing/build/distrubtions/wres-[date]-[revision].zip

In each case, the version is [date]-[revision]. If changes have been made locally, a -dev suffix might be added to the revision. If so, be sure to include that with the version when noted below.

3. From the ./systests directory, remove the build directory, setup the build directory, and cd into the created directory. On Linux, its the following commands:

cd systests
rm -rf build
../gradlew installDist
cd build/install/systests

On Windows, be sure to use gradlew.bat.

4. Create the data directory within the current working directory. In the current working directory after changing directories above, the data directory must be created. In Linux environments, a symbolic link named data should be created to the systests_data directory that was created in the previous section. In Windows, the directory created in the previous section should be copied to the current working directory. Per these instructions, the data directory will be removed whenever system tests are rerun (by the rm -rf build command, above). Hence, be sure to keep the systests_data in a separate directory on the side.

5. In the current working directory, run the Gradle command, below, to execute the tests. The current working directory should be, [repository clone]/systests/build/install/systests. Do the following (be sure to set the [core version], [wres-vis version], and [wres-writing version] appropriately based on Step 2, above):

./gradlew cleanTest test -PversionToTest=[core version] -PgraphicsVersionToTest=[wres-vis version] -PwritingVersionToTest=[wres-writing version] -PwresZipDirectory=../../../../build/distributions/ -PwresGraphicsZipDirectory=../../../../wres-vis/build/distributions/ -PwresWritingZipDirectory=../../../../wres-writing/build/distributions/ -PtestJvmSystemProperties="-Xmx4g -Xms4g -Dwres.dataDirectory=. -Dwres.systemTestingSeed=2389187312693262 -Djava.io.tmpdir=./outputs -Dwres.eventsBrokerAddress=localhost -Dwres.eventsBrokerPort=5673 -Dwres.externalGraphics=true -Dwres.externalNumerics=true -Dwres.startBroker=false" -PtestJvmSystemPropertiesGraphics="-Djava.io.tmpdir=./outputs -Dwres.eventsBrokerAddress=localhost -Dwres.eventsBrokerPort=5673 -Dwres.startBroker=true" -PtestJvmSystemPropertiesWriting="-Djava.io.tmpdir=./outputs -Dwres.eventsBrokerAddress=localhost -Dwres.eventsBrokerPort=5673 -Dwres.startBroker=false" --tests="SystemTestSuite"

In Windows, be sure to use gradlew.bat. This gradlew command will take several minutes to complete. Individual test logging will be output to the screen as it progresses.

5. View the system test results by opening the file build/reports/tests/test/index.xml relative to the current working directory in a browser.

The current working directory should be, [repository clone]/systests/build/install/systests; the path above is relative to that directory. The file will summarize the test results overall. Individual html files for each test are in build/reports/tests/test/classes. Outputs generated by the tests are written to subdirectories of the outputs. Again, stdout is output to the terminal (i.e., stdout) in which the tests are being executed.

Can I run the system tests using a database?

Yes, you can. Instructions are provided in ???. The parameters that support use of the database noted in [this section]TBD must be inserted into the command, above, immediately after -Xmx4g -Xms4g.

Instructions for editing wikis

Writing and editing wikis comes with two rules:

RULE 1: If a Table of Contents is deemed necessary, then use the add_toc.sh script in the wres.wiki repository. To do so, do the following:

  1. Check out the wres.wiki repository and identify the markdown to which a ToC is to be added.
  2. Run add_toc.sh markdown.md for the desired markdown file.
  3. Review the .new version of the file that it creates. Edit as necessary.
  4. Move the .new version into place.
  5. Push the changes to the wres.wiki repository.

The above process needs to be repeated whenever a section header is modified, added, or deleted in a wiki with a ToC.

RULE 2: If enumerations (i.e. numbered sections) are deemed necessary, then declare an anchor for each enumerated heading, as described below, and refer to the non-enumerated anchor instead of the raw link whenever you want to refer to the section. In this way, we are able to keep section numbers out of references so that section numbers can be modified (e.g., due to new sections or reordering) without having to update every reference.

An anchor is created by using the a HTML tag and defining a name for the anchor as an attribute. For example:

<a name="what-are-the-other-capabilities-of-the-wres"></a>
# 1.1 What are the other capabilities of the WRES?

Whenever that header/section is referenced elsewhere, the anchor must be used instead of the direct header link. In the example above, the anchor link would be referenced as, ...#what-are-the-other-capabilities-of-the-wres, where ... is the wiki containing the anchor.

The WRES Wiki

Home

Clone this wiki locally