{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Generating Benchmark Files and Serpent Scripts\n", "\n", "It is really important to validate trained models using criticality benchmarks. NucML provides a couple of scripts to aid automate this tedious process. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:17:53.935202Z", "start_time": "2021-05-07T22:17:53.932201Z" } }, "outputs": [], "source": [ "# Prototype\n", "import sys\n", "# This allows us to import the nucml utilities\n", "sys.path.append(\"..\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:17:57.741286Z", "start_time": "2021-05-07T22:17:54.421117Z" } }, "outputs": [], "source": [ "import pandas as pd\n", "import os\n", "import logging\n", "logger = logging.getLogger()\n", "logger.setLevel(logging.CRITICAL)\n", "\n", "pd.set_option('display.max_columns', 500)\n", "pd.set_option('display.max_rows', 50)\n", "pd.options.mode.chained_assignment = None # default='warn'\n", "\n", "import nucml.datasets as nuc_data\n", "import nucml.ace.data_utilities as ace_utils\n", "import nucml.model.utilities as model_utils" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:17:58.062378Z", "start_time": "2021-05-07T22:17:58.059880Z" } }, "outputs": [], "source": [ "figure_dir = \"Figures/\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading Datasets\n", "\n", "In our work, several models were trained with Datasets 0-4. Since the models will be used to query data at the original ACE's energy grid, we need to load the original data to expand the energy grid for the isotopes of interest (among other processes)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2021-04-09T02:20:33.992108Z", "start_time": "2021-04-09T02:09:46.504840Z" } }, "outputs": [], "source": [ "# LOADING DATASET\n", "df_b0, _, _, _, _, to_scale_b0, _ = nuc_data.load_exfor(pedro=True, basic=0, normalize=False)\n", "df_b1, _, _, _, _, to_scale_b1, _ = nuc_data.load_exfor(pedro=True, basic=1, normalize=False)\n", "df_b2, _, _, _, _, to_scale_b2, _ = nuc_data.load_exfor(pedro=True, basic=2, normalize=False)\n", "df_b3, _, _, _, _, to_scale_b3, _ = nuc_data.load_exfor(pedro=True, basic=3, normalize=False)\n", "df_b4, _, _, _, _, to_scale_b4, _ = nuc_data.load_exfor(pedro=True, basic=4, normalize=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading Decision Tree Results\n", "\n", "`NucML` will create a directory per model and create subdirectories for every criticality benchmark case. Therefore, we need to specify the directories where the model directories and subdirectories will be stored. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2021-04-09T02:33:29.671649Z", "start_time": "2021-04-09T02:33:29.668647Z" } }, "outputs": [], "source": [ "dt_ml_ace_dir_b0 = \"ml/DT_B0/\"\n", "dt_ml_ace_dir_b1 = \"ml/DT_B1/\"\n", "dt_ml_ace_dir_b2 = \"ml/DT_B2/\"\n", "dt_ml_ace_dir_b3 = \"ml/DT_B3/\"\n", "dt_ml_ace_dir_b4 = \"ml/DT_B4/\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Having defined the directories, we can read in the training results. In this example, I read the samples provided with the repository. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:18:00.262284Z", "start_time": "2021-05-07T22:18:00.196997Z" } }, "outputs": [], "source": [ "# read in the training results\n", "results_b0 = pd.read_csv(\"../ML_EXFOR_neutrons/2_DT/dt_resultsB0.csv\").sort_values(by=\"max_depth\")\n", "results_b1 = pd.read_csv(\"../ML_EXFOR_neutrons/2_DT/dt_resultsB1.csv\").sort_values(by=\"max_depth\")\n", "results_b2 = pd.read_csv(\"../ML_EXFOR_neutrons/2_DT/dt_resultsB2.csv\").sort_values(by=\"max_depth\")\n", "results_b3 = pd.read_csv(\"../ML_EXFOR_neutrons/2_DT/dt_resultsB3.csv\").sort_values(by=\"max_depth\")\n", "results_b4 = pd.read_csv(\"../ML_EXFOR_neutrons/2_DT/dt_resultsB4.csv\").sort_values(by=\"max_depth\")\n", "\n", "results_b0 = results_b0[results_b0.normalizer == \"none\"]" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:19:24.592239Z", "start_time": "2021-05-07T22:19:24.588238Z" } }, "source": [ "Let us take a look at the columns included in the results:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:18:06.266233Z", "start_time": "2021-05-07T22:18:06.260235Z" } }, "outputs": [ { "data": { "text/plain": [ "Index(['id', 'max_depth', 'mss', 'msl', 'mt_strategy', 'normalizer',\n", " 'train_mae', 'train_mse', 'train_evs', 'train_mae_m', 'train_r2',\n", " 'val_mae', 'val_mse', 'val_evs', 'val_mae_m', 'val_r2', 'test_mae',\n", " 'test_mse', 'test_evs', 'test_mae_m', 'test_r2', 'model_path',\n", " 'training_time', 'scaler_path'],\n", " dtype='object')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results_b0.columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that there are a lot of performance metrics available. You can include more information in your own results files. **THE ONLY CONDITION IS THAT THE RESULTING DATAFRAME CONTAINS THE FOLLOWING COLUMNS:**\n", "\n", "- model_path\n", "- scaler_path" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2021-05-07T22:18:45.293352Z", "start_time": "2021-05-07T22:18:45.284851Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", " | model_path | \n", "scaler_path | \n", "
---|---|---|
52 | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS15_MSL10_none... | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS15_MSL10_none... | \n", "
47 | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS15_MSL3_none_... | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS15_MSL3_none_... | \n", "
45 | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS15_MSL1_none_... | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS15_MSL1_none_... | \n", "
43 | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS10_MSL7_none_... | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS10_MSL7_none_... | \n", "
41 | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS10_MSL5_none_... | \n", "E:\\ML_Models_EXFOR\\DT_B0\\DT60_MSS10_MSL5_none_... | \n", "