Reporting

Reporting tool allows to collect masks and metrics from several Mlflow runs and storages. It assembles an HTML presentation with direct links to xOpat browser.

The presentation is composed of a set of image items (typically slides). Each item needs to have a background image (e.g. WSI). Optionally, items can have masks and metrics. Aggregated metrics for the whole set can be displayed as well.

Metrics

Metrics are collected from Mlflow runs (by their IDs) and resolved by their names:

• Item metrics are resolved by */item_name/metric_name.
• Aggregated metrics are resolved by */metric_name.

Masks

Masks can be collected from various sources (Mlflow runs, artefacts or local files on /mnt/data). See the available mask retrievers in report/masks.py. Feel free to implement your own mask retriever if needed.

Exactly one set of masks have to be set as background for xOpat. Optionally many sets of masks can be set as xOpat layers.

Saving

The HTML presentation can be saved:

• locally,
• attached to an existing Mlflow run or,
• as a new Mlflow run. See the available storers in report/save.py.

NOTE:
The library is also available as a trainer's callback in HistoPipe.

Installation

Using Pip

For user creation of reports, this is the preferred way.

pip install git+ssh://git@gitlab.ics.muni.cz/rationai/digital-pathology/pipeline/report.git

Clone the repository

Especially when you need to contribute with a new retriever or storer.

git clone https://gitlab.ics.muni.cz/rationai/digital-pathology/pipeline/report.git

And install dependencies:

cd report && pdm install

Usage

When running locally (not on Kubas cluster), you need to add mlflow=kubas_external to your command.

NOTE
.yml extension does not work with Hydra.

Using installed package

python -m report --config-dir <dir> reporter=<config_group> user=<your_name>

where config_group is the name of your configuration file config_group.yaml in <dir>/reporter.

NOTE!
The user directory <dir> will be ADDED to the default config directory .../site-packages/report/conf, it will NOT replace it.

If you need to retrieve the default configs (e.g. for a reference before you create your own config files), run:

python -m report.give_me_conf -h  # for help
python -m report.give_me_conf output_dir  # to get the template config files

Inside the cloned repository

pdm report reporter=<config_group> user=<your_name>

where config_group is the name of your configuration file config_group.yaml in <config_dir>/reporter.

The default <config_dir> is report/conf. If you want to use a different directory for the config files, create your own config dir with your own config groups and specify it using hydra.searchpath when running the report module:

pdm report hydra.searchpath=<dir> reporter=<config_group> user=<your_name>

NOTE!
The user directory <dir> will be ADDED to the default config directory report/conf, it will NOT replace it.

If you need to retrieve the default configs (e.g. for a reference before you create your own config files), run:

pdm help  # for help
pdm template output_dir  # to get the template config files

Configuration

The tool uses Hydra for configuration.

Example

A configuration config_name may look like this:

defaults:
  - default
  - background: mlflow_run_id
  - save: local

title: Example local report
background:
  run_id: e14468e91bad492e96fc142220cbf311
  dir_name: dataset/he
  layer_name: H&E background
mask_retrievers:
  - _target_: report.masks.RunIDMlflowMaskRetriever
    run_id: e14468e91bad492e96fc142220cbf311
    dir_name: dataset/ce
    layer_name: Segmentation mask
save:
  output_path: report.html

This configuration will:

• collect the H&E background from the Mlflow run,
• collect the segmentation masks from the Mlflow run (crated by ImageRegistration),
• attaches no metrics,
• save the report locally as report.html.

Example with metrics

The following configuration will:

• collect the H&E background from Mlflow dataset produced by SlideTyler
• collect two sets segmentation masks from two Mlflow runs
• collect metrics from the two Mlflow runs
• save the report as a new Mlflow run

defaults:
  - default
  - background: mlflow_tyler
  - save: mlflow_standalone_run

title: Example Mlflow report
background:
  artifact_uris:
    - "mlflow-artifacts:/4/42a6acd9b91942b4838646b6e9a2c158/artifacts/dataset"
  item_name_col: slide_name
  background_masks_directory: rationai/Prostata/slide_tif
  ground_truth_col: is_cancer
  layer_name: Prostate WSI
metrics_run_ids:
  - run_ID_1
  - run_ID_2
mask_retrievers:
  - _target_: report.masks.RunIDMlflowMaskRetriever
    dir_name: test_heatmaps
    run_id: run_ID_1
    layer_name: Cancer prediction 1
  - _target_: report.masks.RunIDMlflowMaskRetriever
    run_id: run_ID_2
    dir_name: test_heatmaps
    layer_name: Cancer prediction 1
save:
  metadata:
    experiment_name: Prostate
    run_name: Heatmaps comarison
    description: Heatmaps comparison of two models...

Check the conf tree for more info.

Configuring visualization layers (shaders)

You can add key shader_conf for mask_retrievers to specify what to render. This is an object that maps to the shader configuration. Supported are built-in configurations, and any xOpat compatible configuration in case you want to have advanced control (see the xOpat visualization docs in the onboarding section). Note that some are identical to the xOpat visualization configuration.

Example:

mask_retrievers:
  - _target_: ...
    # ...
    shader_conf:
      type: "classify"
      classes: 3  # compulsory with type classify
      
      # optionally, we can override
      modifiable: true  # allow users to change class value ranges 
      color: ["#000000", "#fafafa", "#ffffff"]  # instead of buildin, use custom colors
      scheme: "sequential"  # if color is a string (name of a map), scheme is a category of colors  
      opacity: 0.75  # 75%
      use_channel0: "b"  # read blue channel

Predefined configurations have nothing in common with xOpat docs, these just hardcode common use-cases. You always specify the type using type. Available built-in types include:

• heatmap (customizes color, opacity)
• highlight (customizes color, opacity)
• classify with integer property classes (compulsory if color is not an array`)
  • customizes: color, scheme, modifiable, opacity
  • color can be either a scheme compatible string (see available here) or classes-length array of hex color names

Every build-in type also customizes use_channel0 property. Identity ignores this setting. Other shaders accept only single channel, e.g. "r". Choose from r, g, b, a.

Note the difference in parameters: build-in parameters can be overridden on the type level configuration. But with xOpat, a shader parameters are specified inside params dict (which you must use when not using built-in configurations, or overriding advanced properties). For more information, see xOpat docs in the onboarding.

shader_conf:
  type: "classify"
  classes: 5 
   
  # built-in opacity modifier will override our manual params opacity
  opacity: 0.75
  params: {
    opacity: 0.85
  }

Testing

Install requirements pdm install. Run pdm test