11. ROC diagram

11.1. Description

ROC (Receiver Operating Characteristic) curves are useful in weather forecasting. ROC curve plots show the true positive rate (sensitivity) versus the false positive rate (1 - specificity) for different cut-off points of a parameter. In addition to creating ROC diagrams directly from the source code in the METplotpy repository, ROC diagrams can be generated through METviewer. For more information on ROC diagrams, please refer to the METviewer documentation.

11.2. Example

11.2.1. Sample Data

Sample data used to create an example ROC diagram is available in the METplotpy repository, where the ROC diagram code is located:

$METPLOTPY_BASE/test/roc_diagram/plot_20200507_074426.data

$METPLOTPY_BASE is the directory where the METplotpy source code is installed. (e.g. /username/myworkspace):

e.g.

/usr/path/to/METplotpy if the source code was cloned or forked from the Github repository

or

/usr/path/to/METplotpy-x.y.z if the source code was downloaded as a zip or gzip’d tar file from the Release link of the Github repository. The x.y.z is the release number.

11.2.2. Configuration Files

The ROC diagram utilizes YAML configuration files to indicate where input data is located and to set plot attributes. These plot attributes correspond to values that can be set via the METviewer tool. YAML is a recursive acronym for “YAML Ain’t Markup Language” and according to yaml.org, it is a “human-friendly data serialization language”. It is commonly used for configuration files and in applications where data is being stored or transmitted. Two configuration files are required. The first is a default configuration file, the first is a default configuration file, roc_diagram_defaults.yaml that is found in the $METPLOTPY_BASE/metplotpy/plots/config directory. All default configuration files are located in the $METPLOTPY_BASE/metplotpy/plots/config directory. Default configuration files are automatically loaded by the plotting code and do not need to be explicitly specified when generating a plot.

The second, required YAML configuration file is a user-supplied “custom” configuration file that is used to customize/override the default settings in the roc_diagram_defaults.yaml file. The custom configuration file can be an empty file if default settings are to be applied.

11.3. METplus Configuration

11.3.1. Default Configuration File

The following is the mandatory, roc_diagram_defaults.yaml configuration file, which serves as a starting point for creating a ROC diagram plot as it represents the default values set in METviewer.

NOTE: This default configuration file is automatically loaded by roc_diagram.py.

---
# Default settings specific to ROC diagram plot
# Line and marker plots of false alarm rate (x-axis) vs probability of detection (y-axis).

# Write points file. Set to True for METviewer use,
# False otherwise
dump_points_1: 'False'

# Title settings
title: ROC CTC # required
title_align:  0.5 # optional
title_offset: -2 #required
title_size: 1.4 #required
title_weight: 2.0 #required


plot_width: 11 #required
plot_height: 8.5 #required
plot_res: 72 #required
plot_units:  in #required

# one of the following MUST be set to False whilst the other
# is set to True.  Both MUST be set to a boolean value.
roc_pct: False
roc_ctc: True

add_point_thresholds: True  #optional

series_val_1: {}
series_val_2:   {}

fcst_var_val_1: {}  #optional
fcst_var_val_2: {}  #optional

indy_var:    'fcst_valid_beg'
indy_vals:    #optional

user_legend:
  #user_legend:
  - 'CTC ROC'

# Required setting. Two types supported, 'n' for none and 'o' for box
legend_box: 'n'
# number of columns in legend
legend_ncol: 3 #optional
legend_inset:
  x: 0.  #required
  y: -0.14 #required
legend_size: 0.8  #required

line_type: N/A

log_filename: stdout
log_level: ERROR
plot_disp:
  - True #required
series_order:
  - 1  #required
stat_curve: 'None' #required

colors:
  -  "#8000ff" #optional
series_symbols:
  -  "Small circle" #required
series_line_width:
  -  1  #optional
series_type:
  - "joined lines" #optional

series_line_style:
  # solid line
  -  "-" #optional

# Perform event equalization to check for missing data
event_equal: True #False

# Caption settings
plot_caption: "This is caption" #required, set to empty string if no caption text
caption_weight: 1 #optional
caption_col: "#333333" #optional
caption_size: 0.8 # relative magnification, required
caption_offset: 3  # axis perpendicular location adjustment, required
caption_align: 0 # axis parallel location adjustment, required

# X1 axis settings
xaxis: test x_label #required
# unsupported by Python plotting
xlab_align: 0.5 #not used for this plot
xlab_offset: 2 #required
xlab_size: 1 #required
xlab_weight: 1 #optional
xlim: [] #optional
xtlab_decim: 0 #optional
xtlab_horiz: 0.5 #optional
xtlab_orient: 1 #optional
xtlab_perp: -0.75 #optional
xtlab_size: 1 #required

# Y1 axis settings
yaxis_1: test y_label #required, set to empty string if no label
yaxis_2: '' #required, set to empty string if no label
#unsupported by Python plotting
ylab_align: 0.5 #unused by this plot
ylab_offset: -2 #required
ylab_size: 1 #required
ylab_weight: 1 #optional
ylim: [] #optional
ytlab_horiz: 0.5 #optional
ytlab_orient: 1 #optional
ytlab_perp: 0.5 #optional
ytlab_size: 1 #required

reverse_connection_order: False

create_html: True  #optional
stat_input:  ../../test/roc_diagram/plot_20200507_074426.data #required
plot_filename: ./roc_diagram_default.png #required

In the default config file, logging is set to stdout and the log level is ERROR (i.e. only log messages of type ERROR will be logged). If the log_filename and log_level are not specified in the custom configuration file, these settings will be used.

11.3.2. Custom Configuration File

A second, mandatory configuration file is required, which is used to customize the settings to the ROC diagram plot. The custom_roc_diagram.yaml file is included with the source code and looks like the following:

---
# Write points file. Set to True for METviewer use,
# False otherwise
dump_points_1: 'False'
# Optional, uncomment and set to directory to store the .points1 file
# that is used by METviewer (created when dump_points_1 is set to True)
# if dump_points_1 is True and this is uncommented, the points1 file
# will be saved in the default location (i.e. where the input data file is stored).

# points_path: '/dir_to_save_points1_file'

# Title settings
title: "Custom: ROC PCT 20200507_074426 "
# value of 0.5 centers the title, smaller values moves text to the left
title_align: 0.3
# -2 locates title at top, -1.5 puts title inside plotting region
title_offset: -2
# font size multiplier
title_size: 1.5
#1=plain text, 2=bold,3=italic,4=bold italic
title_weight: 4


plot_width: 6
plot_height: 6
plot_res: 72
plot_units: in

roc_pct: False
roc_ctc: True

add_point_thresholds: False

series_val_1: {}
series_val_2: {}

user_legend:
  - 'CTC ROC'

legend_box: 'o' # legend box type - o: box, n: none
legend_inset:
  x: 0.
  y: -0.14
legend_size: 0.8 # relative magnification

plot_disp:
    - True

series_order:
    - 1

stat_curve: 'None'

colors:
    - "#8000ff"
series_symbols:
    - "Small circle"
series_line_width:
        - 1
series_type:
    - 'joined lines'
series_line_style:
    - 'solid'

event_equal: False

# Caption settings
plot_caption: "This is caption"
# weight of text: 1=plain text, 2=bold, 3=italic, 4=bold italic
caption_weight: 3
# caption text color
caption_col: "#ff4444"
caption_size: 1.2 # relative magnification
#up-down position of caption text
caption_offset: 2.9 #3  # axis perpendicular location adjustment
#left-right positioning of caption. Reasonable values are between 0 and 1: 0=leftmost,1=rightmost
caption_align: .1 #0 # axis parallel location adjustment

# X1 axis settings
xaxis: Custom POFD #"POFD (False Detection Rate)"
xlab_offset: 0
xlab_size:  1.75
xlab_weight: 2
xtlab_horiz: 0.5
xtlab_orient: 2
xtlab_size: 1.2

# Y1 axis settings
yaxis_1: "PODY Custom"
ylab_offset: 0
ylab_size: 2.5
ylab_weight: 1
# orientation: 0=-90 degrees, 1=0 degrees, 2=0 degrees, 3=-90 degrees
ytlab_orient: 1
ytlab_size: 0.9

reverse_connection_order: False

# Make the plot generated in METviewer interactive
create_html: 'True'
stat_input: ./plot_20200507_074426.data
plot_filename: ./roc_diagram_custom.png

# To save your log output to a file, specify a path and filename and uncomment the line below.  Make sure you have
# permissions to the directory you specify.  The default, as specified in the default config file is stdout.
#log_filename: ./roc_diagram.log

# To change the log level, specify a log level: debug, info, warning, error and uncomment the line below.
# The debug and info log levels will produce more log output.
#log_level: WARNING

Copy this custom config file from the directory where the source code was saved to the working directory:

cp $METPLOTPY_BASE/test/roc_diagram/custom_roc_diagram.yaml $WORKING_DIR/custom_roc_diagram.yaml

Modify the stat_input setting in the $WORKING_DIR/custom_roc_diagram.yaml file to explicitly point to the $WORKING_DIR directory (where the custom config files and sample data reside). Replace the relative path ./plot_20200507_074426.data with the full path $WORKING_DIR/plot_20200507_074426.data (including replacing $WORKING_DIR with the full path to the working directory). Modify the plot_filename setting to point to the output path where the plot will be saved, including the name of the plot.

For example:

stat_input: /username/myworkspace/METplotpy/test/roc_diagram/plot_20200507_074426.data

plot_filename: /username/working_dir/output_plots/roc_diagram_custom.png

This is where /username/myworkspace/METplotpy is $METPLOTPY_BASE and /username/working_dir is $WORKING_DIR. Make sure that the $WORKING_DIR directory that is specified exists and has the appropriate read and write permissions.The path listed for plot_filename may be changed to the output directory of one’s choosing. If this is not set, then the plot_filename setting specified in the $METPLOTPY_BASE/metplotpy/plots/config/roc_diagram_defaults.yaml configuration file will be used.

To save the intermediate .points1 file (used by METviewer and useful for debugging), set the dump_points_1 setting to True. Uncomment or add (if it doesn’t exist) the points_path setting.

dump_points_1: ‘True’

points_path: ‘/dir_to_save_points1_file’

Replace the /dir_to_save_points1_file to the directory where the .points1 file is saved. If points_path is commented out (indicated by a ‘#’ symbol in front of it), remove the ‘#’ symbol to uncomment the points_path so that it will be used by the code. Make sure that this directory exists and has the appropriate read and write permissions. NOTE: the points_path setting is optional and does not need to be defined in the configuration file unless saving the intermediate .points1 file is desired.

To save the log output to a file, uncomment the log_filename entry and specify the path and name of the log file. Select a directory with the appropriate read and write privileges. To modify the verbosity of logging than what is set in the default config file, uncomment the log_level entry and specify the log level (debug and info are higher verbosity, warning and error are lower verbosity).

11.3.3. Using Defaults

If the user wishes to use the default settings defined in the roc_diagram_defaults.yaml file, specify a minimal custom configuration file (minimal_roc_diagram.yaml), which consists of only a comment block, but can be any empty file (as long as the user has write permissions for the output filename path corresponding to the plot_filename setting in the default configuration file. Otherwise, this will need to be specified in plot_filename in the minimal_roc_diagram.yaml file):

---
# This is the minimal configuration file you must provide
# If there is no content, as we have here, then all the
# settings in the default config file, roc_diagram_defaults.yaml
# will be employed.  A plot named roc_diagram_default.png will be
# created.

Copy this file to the working directory:

cp $METPLOTPY_BASE/test/roc_diagram/minimal_roc_diagram.yaml $WORKING_DIR/minimal_roc_diagram.yaml

Add the stat_input (input data) and plot_filename (output file/plot path) settings to the $WORKING_DIR/minimal_roc_diagram.yaml file (anywhere below the comment block). The stat_input setting explicitly indicates where the sample data and custom configuration files are located. Set the stat_input to $WORKING_DIR/plot_20200507_074426.data and set the plot_filename to $WORKING_DIR/output_plots/roc_diagram_default.png:

stat_input: $METPLOTPY_BASE/test/roc_diagram/plot_20200507_074426.data

plot_filename: $WORKING_DIR/output_plots/roc_diagram_default.png

$WORKING_DIR is the working directory where all the custom configuration files are being saved (including replacing $METPLOTPY_BASE with the full path to the METplotpy installation on the system). NOTE: To specify the plot_filename (output directory) to a directory other than the $WORKING_DIR/output_plots, this can be done as long as it is an existing directory where the user has read and write permissions.

To save the intermediate .points1 file (used by METviewer and useful for debugging), add the following lines to the minimal_roc_diagram.yaml file (anywhere below the comment block):

dump_points_1: ‘True’

points_path: ‘/dir_to_save_points1_file’

Replace the /dir_to_save_points1_file to the same directory where the .points1 file is saved. Make sure that this directory exists and has the appropriate read and write permissions. NOTE: the points_path setting is optional and does not need to be defined in the configuration file unless saving it to the intermediate .points1 file is desired.

11.4. Run from the Command Line

The ROC diagram plot that uses only the default values defined in the roc_diagram_defaults.yaml configuration file looks like the following:

../_images/roc_diagram_default.png

Perform the following:

  • To use the conda environment, verify the conda environment is running and has has the required Python packages outlined in the Python Requirements section.>`_:

  • Set the METPLOTPY_BASE environment variable to point to $METPLOTPY_BASE

    For the ksh environment:

    export METPLOTPY_BASE=$METPLOTPY_BASE
    

    For the csh environment:

    setenv METPLOTPY_BASE $METPLOTPY_BASE
    

    Replacing the $METPLOTPY_BASE with the directory where the METplotpy source code was saved.

    To generate the above “defaults” plot (i.e using default configuration settings), use the “minimal” custom configuration file, minimal_roc_diagram.yaml.

  • Enter the following command:

    python $METPLOTPY_BASE/metplotpy/plots/roc_diagram/roc_diagram.py $WORKING_DIR/minimal_roc_diagram.yaml
    
  • A roc_diagram_default.png output file will be created in the directory specified in the plot_filename configuration setting in the minimal_roc_diagram.yaml config file.

    To generate a customized ROC diagram (i.e. some or all default configuration settings are to be overridden), use the custom_roc_diagram.yaml config file.

  • Enter the following command:

    python $METPLOTPY_BASE/metplotpy/plots/roc_diagram/roc_diagram.py $WORKING_DIR/custom_roc_diagram.yaml
    

    In this example, this custom config file changes the title and axis labels. The caption magnification, caption weight, and title offset values were modified, resulting in the drastically different plot shown below. The user will need to experiment with these values to achieve the desired appearance.

    ../_images/roc_diagram_custom.png
  • A custom_roc_diagram.png output file will be created in the directory that was specified in the plot_filename configuration setting in the custom_roc_diagram.yaml config file.