stoner is a package to help with various tasks involving VIMC touchstones. Its purpose is evolving somewhat with the needs of the VIMC project; stoner is becoming an umbrella to keep these needs expressed in a tested package. As such, it can be used in a number of modes.
Creation of touchstones is quite a common process, although new touchstones will often be based on a previous one. However, creating the touchstone involves additions to various related tables, so the code to create touchstones is not always trivial to review.
Dettl has somewhat helped here, encouraging separation of extract, transform and load stages of an import, with testing of each stage, forcing the code for touchstone creation to be written in a way that separates those concerns and makes reviewing easier. Furthermore, it has often been possible to review a new import as a diff to a previously reviewed import.
Stoner takes this a step further by allowing the touchstone creation to be expressed in csv meta-data, providing function calls for the extract, transform and load stages.
The code for a stoner touchstone import is very simple. Dettl requires that we write extract, transform, and load functions, and tests for the extract and load. So we create a dettl import as usual (see dettl::dettl_new), which begins a new import in our imports repo.
Dettl requires us to write various functions, which we can satisfy with single line functions for a start.
| Dettl.function | Stoner.call |
|---|---|
| extract(con) | stoner::stone_extract(‘.’, con) |
| test-extract(extracted_data) | stoner::stone_test_extract(extracted_data) |
| transform(extracted_data) | stoner::stone_transform(extracted_data) |
| test-transform(transformed_data) | stoner::stone_test_transform(transformed_data) |
| load(transformed_data, con) | stoner::stone_load(transformed_data, con) |
So for the minimal example, when writing the dettl import, delegate each of dettl’s functions to the stoner handlers, passing the same arguments.
The minimal example on its own will do nothing and exit cleanly. To make stoner do some useful work, we write csv files in a folder called meta within the import folder. These csv files should be thought of a specification of “how you would like things to be” - rather than “what you want stoner to do”. If rows in your csv file identically exist in the database, stoner will use the existing ones and not add duplicates. If the rows in your csv are provably absent from the database, stoner will add new ones.
If stoner detects in some way that the items already exist, but not all csv data matches with those items, then some other factors come into play that affect whether stoner can update the database content or not. Imports are incremental to the database, yet on some occasions, it is useful to be able to do in-place edits of touchstones that are still in preparation, for example.
Following are the csv files that stoner will recognise, and their columns and formats, and notes on the requirements. Any failure to meet the requirements will cause an abort with an error message.
You do not have to provide all of the csvs, only ones where you expect something to change, but you may find it good practice to “over-specify”, since the result is that Stoner will check the database tables are all as you expect. It also may be helpful to be able to compare complete touchstone definitions (ie, sets of stoner csv files) as a diff between two imports.
A touchstone_name refers to a broad ensemble of closely related runs; there will be at least one version for each touchstone_name, and it is the specific version that we colloquially refer to as ‘a touchstone’.
| Column | Example |
|---|---|
| id | 201910gavi |
| description | October 2019 touchstone |
| comment | Standard GAVI |
id is not found in the touchstone_name db table, then the row is added.id is found, and description and comment match, then the row in the csv is ignored.touchstone_name with that id exists, but description and/or comment differ from the original, then the fields in the database are updated in-place if either:
in-preparation.Description and Comment must be non-empty. Conventionally, description has been used to describe the date and basic purpose, and comment for any further detail required.A touchstone is a particular version of a touchstone_name, and is the basic unit of currency for touchstones in Montagu. Coverage, expectations and burden estimates are all attached to one of these versioned touchstones.
| Column | Example |
|---|---|
| id | 201910gavi-1 |
| touchstone_name | 201910gavi |
| version | 1 |
| status |
in-preparation, open or finished
|
| description | 201910gavi (version 1) |
| comment | GAVI Version 1 |
id must be in the form touchstone_name-version.touchstone_name must match a touchstone_name.id, either in the database, or in the accompanying touchstone_name.csv
id is not found, then stoner will add the new row.id is found and all other columns match, stoner ignores it.id is found, but any other column differs, then stoner will update the fields in the existing touchstone only if its status is in-preparation. Otherwise, it will fail with an error.description and comment must be non-empty. Typically, description has been very minimal about the existence of the touchstone, and comments to record more details about why the touchstone version exists.The touchstone_country table in the database should really be called touchstone_country_disease. For a given touchstone, it records which countries should be returned when groups download their demographic data. This might differ from the countries a group is expected to model for a certain touchstone; see the responsibilities.csv section for that.
| Column | Example |
|---|---|
| touchstone | 201910gavi-1 |
| disease | Measles;MenA |
| country | AFG;BEN;IND;ZWE |
id must exist in the database already, or in the touchstone.csv file included in your import.id field of the disease table. Stoner cannot currently add new diseases.id column of the country table. Stoner cannot currently add new countries, but the country table should be complete.The touchstone_demographic_dataset table determines which demographic_statistic_types from which demographic_source will be used when providing demographic data for a particular touchstone. Generally, there will be a new demographic source each year, when either the IGME child mortality data, or the UNWPP population data, or both get updated. Because these updates happen at different times (UNWPP bi-yearly, and IGME yearly), sometimes a touchstone_demographic_dataset might incorporate fields from different sources, hence this table.
| Column | Example |
|---|---|
| demographic_source | dds-201910_2 |
| demographic_statistic_type | int_pop |
| touchstone | 201910gavi-1 |
demographic_statistic_type and demographic_source are strings, that must exist in the code column of the respective database tables.id must exist in the database already, or in the touchstone.csv file included in your import.| Column | Example |
|---|---|
| id | stop |
| name | VIMC stop scenario |
id is not found in the database table, then new rows will be added.id is found in the database table, and the name matches that in your csv file, then the row is ignored.id exists, but the name differs then:-
id, the name in the database table for this id will be updated.id, then stoner looks up the status of any touchstones that refer to that scenario_description, and will only perform the update if all are in the in-preparation state.
stoner::stone_load(transformed_data, con, allow_overwrite_scenario_type = TRUE)
| Column | Example |
|---|---|
| id | mena-routine-no-vaccination |
| description | Description free text |
| disease | MenA |
| scenario_type | stop |
id has conventionally been in lower-case, and in the form disease-coverage_type.id does not exist in the database, stoner will add the new scenario_description.id exists, and all other columns match the existing values too, then the row is ignored. If the id exists, but other columns differ, then:
scenario exists that refers to this scenario_description, and the touchstone associated with that scenario is not in the in-preparation state, then the import fails with an error.in-preparation requirement, in the load phase of the import, use:
stoner::stone_load(transformed_data, con, allow_overwrite_scenario_description = TRUE)
disease must currently be one of Cholera, HepB, Hib, HPV, JE, Measles, MenA, PCV, Rota, Rubella, Typhoid or YF. These match the id column of the disease table. Stoner cannot currently add new diseases; this is an admin task done separately.scenario_type must be the id of a scenario_type, either in the database table, or in a scenario_type.csv as part of your import.Most of the work for implementing a touchstone is done here, in which we add the scenario, responsibility and expectations (including countries and outcomes) that form the tasks different groups must perform.
| Column | Example |
|---|---|
| modelling_group | IC-Hallet |
| disease | HepB |
| touchstone | 201910gavi-1 |
| scenario | hepb-no-vaccination;hepb-bd-routine-bestcase |
| scenario_type | standard |
| age_min_inclusive | 0 |
| age_max_inclusive | 99 |
| cohort_min_inclusive | 1901 |
| cohort_max_inclusive | 2100 |
| year_min_inclusive | 2000 |
| year_max_inclusive | 2100 |
| countries | AFG;BEN;COD |
| outcomes | dalys;deaths;cases |
The modelling_group must match an id of the modelling_group table. Stoner can’t add new modelling groups.
The disease must match an id of the disease table. Stoner can’t add new diseases either.
The touchstone must exist either in the touchstone table, or in the touchstone.csv as part of your import.
scenario here is a semi-colon separated list of scenarios - which are actually scenario_description ids. The matching description must exist in either the scenario_description table, or scenario_description.csv in your import.
The minimum and maximum ages, cohorts and years above are implying that this group should provide ages 0-99 inclusive, for calendar years 2000-2100. The 99 year olds in 2000 were born in 1901, so this is the minimum cohort, whereas the maximum cohort will be the last year in which people were born, which will be the 0-year olds in 2100.
All countries must be found in the id column of the country table. Stoner cannot currently add new countries, but the country table should be complete.
All outcomes must be found in the code column of the burden_outcome` table. Stoner cannot currently add new burden outcomes.
The responsibilities.csv file may cause changes to the scenario, responsibility_set, responsibility, burden_estimate_expectation, burden_estimate_country_expectation and burden_estimate_outcome_expectation tables. Where possible, existing rows are re-used, rather than creating duplicates.
scenario table
scenario is defined by scenario_description and touchstone. If combinations of those exist in responsibilities_csv that aren’t in the scenario database table, then new rows get added.in-prep.responsibility_set table
responsibility_set is defined by modelling_group and touchstone. If combinations exist in responsibilities_csv that aren’t in the database, they get created.in-prep.burden_estimate_expectations table
touchstone_name - no version number), and a description, conventionally in the form disease:group:scenario_type, where the scenario_type might be a particular scenario (if expectations need to be specific to that scenario), or in many cases, then scenario_type has been defined as standard, allowing the same expectation definition to be shared for different scenarios (for a particular group and disease).in-prep.burden_estimate_country_expectation table
burden_estimate_expectation, the rows in the burden_estimate_country_expectation table list the countries for which we are expecting estimates to be uploaded by a particular group, for a particular disease, and a particular scenario.burden_estimate_expectation in this table is a numerical id, referring to either a newly created expectation as a result of your import, or an expectation that previous existed and matched the details exactly.burden_estimate_outcome_expectation table
burden_estimate_outcome_expectation table list the expected outcomes that a group will upload for a particular scenario and disease.burden_outcome table - but Stoner cannot change the contents of that table at present.responsibility table
in-prep.current_burden_estimate_set and current_stochastic_burden_estimate_set - both of which are nullable, and are left at NA by default.is_open, which Stoner will set to TRUE as default.Firstly, have your test-extract call stoner::stone_test_extract(extracted_data), and test-transform call stoner::stone_test_transform(transformed_data) for the built-in tests to be called. Most likely, there is nothing else useful you can write for these tests, if your extract and transform functions are simply calling Stoner’s.
Possibly the best approach to tests is to write the test-queries function for dettl in the following form:-
test_queries <- function(con) { list(
tn = DBI::dbGetQuery(con, "SELECT count(*) FROM touchstone")[1,1],
tddn = DBI::dbGetQuery(con, "SELECT count(*) FROM touchstone_demographic_dataset")[1,1],
tcn = DBI::dbGetQuery(con, "SELECT count(*) FROM touchstone_country")[1,1],
sdn = DBI::dbGetQuery(con, "SELECT count(*) FROM scenario_description")[1,1],
sn = DBI::dbGetQuery(con, "SELECT count(*) FROM scenario")[1,1],
been = DBI::dbGetQuery(con, "SELECT count(*) FROM burden_estimate_expectation")[1,1],
beoen = DBI::dbGetQuery(con, "SELECT count(*) FROM burden_estimate_outcome_expectation")[1,1],
becen = DBI::dbGetQuery(con, "SELECT count(*) FROM burden_estimate_country_expectation")[1,1],
rsn = DBI::dbGetQuery(con, "SELECT count(*) FROM responsibility_set")[1,1],
rn = DBI::dbGetQuery(con, "SELECT count(*) FROM responsibility")[1,1]
)}and a test_load.R that tests how many rows have been added, for example…
context("load")
testthat::test_that("Expected rows added", {
expect_equal(after$tn, before$tn + 1)
expect_equal(after$tddn, before$tddn + 23)
expect_equal(after$tcn, before$tcn + 946)
expect_equal(after$sdn, before$sdn)
expect_equal(after$sn, before$sn + 67)
expect_equal(after$been, before$been + 26)
expect_equal(after$beoen, before$beoen + 128)
expect_equal(after$becen, before$becen + 2137)
expect_equal(after$rsn, before$rsn + 17)
expect_equal(after$rn, before$rn + 127)
})For this though, you will have to have prior knowledge about how many of the rows in your various CSV files exist already in the database, and how many you are expecting will need to be created.
The need for fast-forwarding arises when the following events happen.
Therefore, fast-forwarding is a process where burden estimates are moved from one touchstone to another - or more specifically, one responsibility_set to another (since responsibility_set is defined by modelling_group and touchstone).
Suppose then, that we the new touchstone ready, and we have, potentially, some burden estimate sets to migrate. Fast-forwarding would do the following. Let’s consider it first for a single scenario, and a single modelling_group.
We specify that we want to fast-forward an existing burden estimate set for a certain modelling_group and scenario, from one touchstone to another. We’ll see how to specify that in a simple CSV file shortly. A stoner import running on that CSV file then does essentially the following:-
If necessary, create a new responsibility_set for the modelling_group, in the destination touchstone.
responsibility_set already exists, that’s fine, we’ll use the existing one.If necessary, create a new responsibility within the new responsibility_set, for the specified scenario.
responsibility already exists, but there is no burden estimate set associated with it, then we can continue using the existing responsibility.responsibility, we abort and don’t fast forward.Fast-forwarding a burden_estimate_set then means copying the current_burden_estimate_set value from one responsibility to another; from the older into the newer, and setting the older to NA.
Additionally, when a new responsibility_set is created by stoner, it will copy the most recent responsibility_set_comment from the old, to the new responsibility_set, noting that the new one was created by fast-forwding.
For responsibility_comments - if any work is done (either creating a new responsibility, or setting current_burden_estimate_set on the new responsibility for the first time, then the most recent responsibility_comment (if there is one) will be copied to the new responsibility, with a note about fast-forwarding.
Write a fast_forward.csv file in the following form.
| Column | Example |
|---|---|
| modelling_group | IC-Hallett;Li |
| scenario | hepb-no-vaccination |
| touchstone_from | 202110gavi-2 |
| touchstone_to | 202110gavi-3 |
Note that fast-forwarding must be the only thing in the import, and the only .csv file in use. Combining fast-forwarding with other touchstone creation or management functions is too stressful to contemplate. Do them separate, and test them separately.
Also, while you can do fastforwarding for different touchstones in the same CSV, be careful with it as it gets confusing. Stoner will not let you fastforward into, and out of, the same touchstone (ie, from version 1 to 2, and 2 to 3) in the same CSV file.
The modelling_group and scenario columns can either be single items, with multiple rows in the csv file. Or, they can be semi-colon separated, to give multiple combinations of groups and scenarios. Finally, they can be wildcard * to match anything.
Include all the standard stoner one-lines, for the extract, test-extract, transform, test-transform, and load stages, as above, and ensure that in dettl.yml for the report, automatic loading is not enabled.
When modelling groups upload more than one burden estimate set for the same responsibility (that is, the same touchstone, scenario, disease), only the most recent is regarded as interesting, and is marked as the current_burden_estimate_set for the responsibility. To save space (for some groups, a considerable amount of space), the old orphaned burden estimate sets can be deleted.
Note that this should be considered a “final” delete; rows will be dropped from the burden_estimate_set table, and especially the burden_estimate table. While rolling the database back via backups is possible, it’s not desirable. That said, there should be no reason to keep previous versions of a burden estimate set. If both the old and new versions are important, they should both be “current” burden estimate sets, in different touchstones or reponsibilities perhaps.
Write a prune.csv file in the following form.
| Column | Example |
|---|---|
| modelling_group | IC-Hallett;Li |
| disease | * |
| scenario | hepb-no-vaccination |
| touchstone | 202110gavi-2;202110gavi-3 |
Each field can be semi-colon-separated, and the result is that all possibilities are multipled out. (So in the above example, both touchstones, for both modelling groups will be examined for pruning opportunities).
You can also include multiple lines in the CSV file, which will be considered one at a time, thus allowing flexibility to look at a number of specific combinations for pruning.
The * is a wildcard, and in the simplest case, all the fields can be left as *, to look for pruning opportunities in the entire history of burden estimate sets.
Note that if prune.csv exists, no other csv file should be included - that is: a pruning import should just do pruning, and not any other functionality. This keeps things simple, which is a good thing since here we are (somewhat uniquely) performing a deletion of data.
Include all the standard stoner one-lines, for the extract, test-extract, transform, test-transform, and load stages, as above, and ensure that in dettl.yml for the report, automatic loading is not enabled.
stoner::stone_dump(con, touchstone, path), called with a database connection, a touchstone, and an output path, will produce csv files of everything connected with that touchstone, in the form stoner would use to import, as described above. This might be useful if you want to download an existing touchstone, edit some details (including the touchstone id), and upload a modified version.
Modelling groups submit stochastic data to VIMC by responding to a Dropbox File Request. A stochastic set consists of 200 runs for each scenario for that group, using a range of different parameters that are intended to capture the uncertainty in the model.
After some initial sanity checks (which are manual at present), the incoming csvs are compressed with xz with maximum settings, which provides the best compression for csvs, but fast decompression, and seamless decompression in R. (Windows command-line xz -z -k -9 -e *.csv)
The incoming stochastics are separated certainly by scenario, and may be further separated for convenience; some groups have provided a file per country, others a file per stochastic run. From these, we create four intermediate files for each group, which eliminate age by summing over a calendar year, summing over a birth cohort (year - age), and for each option, either including all ages, or filtering just ages 0 to 4. They include just the cases, deaths and dalys outcomes (which might be calculated by summing more detailed outcomes a group provides) for each scenario in columns. The idea is so that calculating impact between scenarios can then be calculated simply by doing maths on values from the same row of the file.
These four files are later uploaded to four separate tables on the annex database.
Note that the production of the intermediate files can take a few hours per group, whereas the upload to annex takes only a few minutes. Storing the intermediate files can be useful should we need to redeploy annex at any point.
Also note the examples below assume you have a connection to the production database (con), and later, a connection to the annex database (annex). See the end for notes on getting those connections in different ways.
In the simplest case, a group uploads a single csv file per scenario as follows:-
| disease | run_id | year | age | country | country_name | cohort_size | cases | deaths | dalys |
|---|---|---|---|---|---|---|---|---|---|
| YF | 1 | 2000 | 0 | AGO | Angola | 677439 | 59 | 22 | 1233 |
| YF | 1 | 2001 | 0 | AGO | Angola | 700540 | 61 | 23 | 1390 |
| YF | 1 | 2002 | 0 | AGO | Angola | 725742 | 66 | 24 | 1330 |
| YF | 1 | 2003 | 0 | AGO | Angola | 753178 | 69 | 25 | 1196 |
| YF | 1 | 2004 | 0 | AGO | Angola | 782967 | 71 | 26 | 1490 |
which would continue for all the countries, years and ages, for 200 runs of a particular scenario. A separate file would exist for each scenario. To transform this into the four intermediate files, we might write below - where the argument names are included just for clarity, and are not needed.
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario.csv.xz",
cert = "certfile",
index_start = NA, index_end = NA,
out_path = "E:/Stochastic_Outputs")This assumes that in the in_path folder, 6 .csv files are present. The file argument indicates the template for those files. In this case we are assuming all the files follow the same template, where :scenario will be replaced by each of the 6 specified scenarios in turn. If the files do not obey such a simple templating, then you can supply a vector of strings for file, to indicate which files; just note there should be either a one-to-one mapping, or a many-to-one mapping between the different scenarios, and the different files indicated.
In this example, there is only one file per scenario; the index_start and index_end arguments are set to NA, and there is no reference to :index in the file template. We will see later multi-file examples where these three fields are changed to describe the sequence of files we are expecting.
The result is that four files are written - below is an abbreviated section of each.
| run_id | year | country | cases_novac | dalys_novac | deaths_novac | cases_prevbest | dalys_prevbest | deaths_prevbest |
|---|---|---|---|---|---|---|---|---|
| 1 | 2000 | 24 | 1219 | 21388 | 452 | 1165 | 20219 | 432 |
| 1 | 2001 | 24 | 1269 | 22884 | 471 | 1199 | 21353 | 444 |
| 1 | 2002 | 24 | 1319 | 24129 | 494 | 1235 | 22207 | 461 |
So here, we have in each row, the cases, deaths and dalys summed over age for a country and calendar year, for each scenario.
| run_id | year | country | cases_novac | dalys_novac | deaths_novac | cases_prevbest | dalys_prevbest | deaths_prevbest |
|---|---|---|---|---|---|---|---|---|
| 1 | 2000 | 24 | 269 | 5710 | 100 | 215 | 4541 | 80 |
| 1 | 2001 | 24 | 280 | 6220 | 105 | 210 | 4689 | 78 |
| 1 | 2002 | 24 | 290 | 6564 | 110 | 213 | 4849 | 80 |
This is similar to the calendar year, but ages five and above are ignored, when summing over age, so the numbers are all smaller.
| run_id | cohort | country | cases_novac | dalys_novac | deaths_novac | cases_prevbest | dalys_prevbest | deaths_prevbest |
|---|---|---|---|---|---|---|---|---|
| 1 | 1900 | 24 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 1901 | 24 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 1902 | 24 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 2000 | 24 | 3149 | 44542 | 1184 | 774 | 15763 | 280 |
| 1 | 2001 | 24 | 3261 | 47051 | 1222 | 809 | 16902 | 284 |
| 1 | 2002 | 24 | 3384 | 51399 | 1269 | 799 | 17573 | 283 |
The cohort is calculated by subtracting age from year; it asks the question when were people of a certain age in a certain calendar year born. Notice the cohort column instead of year. This model includes 100-year-olds alive in calendar year 2000, so these were born in the year 1900, but no yellow fever cases or deaths for these scenarios are recorded for that birth cohort.
| run_id | cohort | country | cases_novac | dalys_novac | deaths_novac | cases_prevbest | dalys_prevbest | deaths_prevbest |
|---|---|---|---|---|---|---|---|---|
| 1 | 1996 | 24 | 49 | 1010 | 18 | 49 | 1010 | 18 |
| 1 | 1997 | 24 | 102 | 2196 | 38 | 86 | 1854 | 32 |
| 1 | 1998 | 24 | 160 | 3626 | 60 | 122 | 2778 | 45 |
| 1 | 1999 | 24 | 221 | 4483 | 83 | 152 | 3086 | 57 |
| 1 | 2000 | 24 | 289 | 6057 | 108 | 207 | 4346 | 78 |
| 1 | 2001 | 24 | 297 | 6915 | 112 | 225 | 5232 | 84 |
| 1 | 2002 | 234 | 310 | 7223 | 116 | 234 | 5464 | 87 |
This is similar to birth cohort, but only considering those age 4 or less. Hence, the oldest age group in the year 2000 (where calendar years begin for this model) will be 4, and they were born in 1996, which is the first birth cohort.
Some groups submit a file per stochastic run, or a file per country. Some have even arbitrarily started a new file when one file has become, say, 10Mb in size. Stoner doesn’t mind at what point the files are split, except that data for two scenarios cannot exist in the same file, and the files that make up a set must be numbered with contiguous integers.
The example below will expect runs numbered from 1 to 200, as indicated with index_start and index_end. Also notice the presence of the :index placeholder in the file stub, which will be replaced with the sequence number when the files are parsed.
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario_:index.csv.xz",
cert = "certfile",
index_start = 1, index_end = 200,
out_path = "E:/Stochastic_Outputs")Some groups might also submit different numbers of files for each scenario. For example, HepB for some groups requires different numbers of countries to be modelled for different scenarios, dependingn on what campaigns were made in those countries. If a group wishes to split their results by country, they will then have different numbers of files per scenario. In this case, index_start and index_end can be vectors, of the same length as the scenarios vector, giving the start and end ids for each scenario.
Stoner can also support a mixture of single and multi-files for different scenarios. For that case, you’ll need vectors for both the file stub, and the index_start and index_end - Stoner will test that whenever the file stub contains :index, the index_start and index_end are specified, otherwise not.
Some groups provide multiple deaths or cases categories which need to be summed to give the total deaths or cases. The example below uses the optional deaths, cases and dalys arguments, where a vector of outcomes is provided (which must exist in the incoming data, and in the responsibilities for that group and disease too), to be summed to give the final outcome.
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario_:index.csv.xz",
cert = "certfile",
index_start = 1, index_end = 200,
out_path = "E:/Stochastic_Outputs"),
deaths = c("deaths_cat1", "deaths_cat2"),
cases = c("cases_cat1", "cases_cat2"),
dalys = "dalys")Occasionally, a group omit the run_id column in their input data. In practice this only happens when the run_id is specified as part of the filename. To handle this, set the optional runid_from_file argument to TRUE - and in that case, index_start and index_end must be 1 and 200 respectively, and :index must be included in the file template for all scenarios (either specified as a vector, or a singleton).
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario_:index.csv.xz",
cert = "certfile",
index_start = 1, index_end = 200,
out_path = "E:/Stochastic_Outputs"),
runid_from_file = TRUE)Some groups have also omitted the constant disease from their stochastic results. This would normally generate a warning but work correctly in any case; to silence the warning, set the optional allow_missing_disease to be TRUE.
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario_:index.csv.xz",
cert = "certfile",
index_start = 1, index_end = 200,
out_path = "E:/Stochastic_Outputs"),
allow_missing_disease = TRUE)As we said, this can occur, with HepB being an example. If this is the case, besides dealing with a different number of files per scenario (if the group split their files by country), there is nothing you need to do for Stoner to process this properly. In the output CSV files, any country for which there is no data for a particular scenario will have NA for those scenarios. Care might be needed in analysis later on in ensuring comparisons or impact calculations only occur where all the values are not NA.
When groups upload the parameters for their stochastic runs into Montagu, they are provided with a certificate - a small JSON file providing metadata, and confirmation of the upload information. The certificate should be provided by the group along with the stochastic data files that were produced using the parameters they uploaded.
By default, stoner will verify that the certificate file exists, and checks that the metadata (modelling group, touchstone, disease) on production that match the certificate also match with the arguments you provide when you call stoner_stochastic_process.
Should you be lacking a group’s certificate, but still want to attempt to process the stochastic data, then set the option bypass_cert_check to be TRUE:-
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario_:index.csv.xz",
cert = "",
index_start = 1, index_end = 200,
out_path = "E:/Stochastic_Outputs"),
bypass_cert_check = TRUE)You can also manually perform validation of a certificate file without processing stochastic data, with the call:-
stone_stochastic_cert_verify(con, "certfile", "IC-Garske", "201910gavi-5", "YF")This call will stop with an error if either the modelling group, or the touchstone do not match with the details used to submit the parameter set, and retrieve the certfile provided here.
The processed CSV files can be uploaded to annex automatically, if an additional database connection annex is provided, and the upload_to_annex is set to TRUE. The files will be uploaded after processing.
stone_stochastic_process(
con = con,
modelling_group = "IC-Garske",
disease = "YF",
touchstone = "201910gavi-4",
scenarios = c("yf-no-vaccination", "yf-preventive-bestcase",
"yf-preventive-default", "yf-routine-bestcase",
"yf-routine-default", "yf-stop"),
in_path = "E:/Dropbox/File Requests/IC-Garske",
file = ":scenario_:index.csv.xz",
cert = "certfile",
index_start = 1, index_end = 200,
out_path = "E:/Stochastic_Outputs"),
upload_to_annex = TRUE,
annex = annex,
allow_new_database = FALSE)If allow_new_database is set to TRUE, then Stoner will try to create the stochastic_file index table on annex; this will only be wanted on the first time of uploading data to a new empty database, so typically, this will be left as FALSE.
The result of uploading is that four new rows will be added to the stochastic_file table, for example:-
| id | touchstone | modelling_group | disease | is_cohort | is_under5 | version | creation_date |
|---|---|---|---|---|---|---|---|
| 1 | 201910gavi-4 | IC-Garske | YF | FALSE | TRUE | 1 | 2020-08-06 |
| 2 | 201910gavi-4 | IC-Garske | YF | TRUE | TRUE | 1 | 2020-08-06 |
| 3 | 201910gavi-4 | IC-Garske | YF | FALSE | FALSE | 1 | 2020-08-06 |
| 4 | 201910gavi-4 | IC-Garske | YF | TRUE | FALSE | 1 | 2020-08-06 |
Four new tables named in the form stochastic_ followed by the id field listed in the table above will also have been made, which are uploaded copies of the final CSV files. If further uploads are made that match the touchstone, modelling_group, disease, is_cohort and is_under5, then the new data will overwrite the existing data, and the version and creation_date in the table above will be updated.
You can also call the stone_stochastic_upload directly, if you have CSV files ready to upload. Call the function as below, to upload a single CSV file. (Vectors for multiple scenarios in one go are not currently supported in the function).
file = 'IC-Garske_YF_calendar_u5.csv',
con = con,
annex = annex,
modelling_group = 'IC-Garske',
disease = 'YF',
touchstone = '201910gavi-4',
is_cohort = FALSE,
is_under5 = TRUE
)The filename is treated as arbitrary; is_cohort and is_under5 need specifying to describe the data being uploaded. If this is the first ever upload to a new database, then the optional allow_new_database will enable creation of the stochastic_file table.
stone_stochastic_process and stone_stochastic_upload both take a testing logical argument; ignore this, as it is only used to as part of the tests, in which a fake annex database is set up.
We use the vaultr package, and assume that the VAULT_ADDR and VAULT_AUTH_GITHUB_TOKEN environment variables are set up - we won’t go into doing that here.
A read-only connection to the production database is used to validate the outcomes and countries against those in a group’s expectations. To get the connection to production:-
vault <- vaultr::vault_client(login = "github")
password <- vault$read("/secret/vimc/database/production/users/readonly")$password
con <- DBI::dbConnect(RPostgres::Postgres(),
dbname = "montagu",
host = "production.montagu.dide.ic.ac.uk",
port = 5432, password = password,
user = "readonly")To get a connection to annex:-
However, rather than acquiring connections as above and manually running ad hoc database queries on annex, it will be better to express imports to annex using dettl. The imports are made a little more complex than usual by the length of time taken to do the data reduction, the RAM they require can be very large, and the possibility that data will be replaced on annex with subsequent versions. Never-the-less, it would be good to have a formal process for uploading data to annex, and dettl would be a good way.
For example:
Extract stage: read meta data, which would contain a list of groups and locations for different stochastic datasets to be processed. Also look up the necessary metadata for those files - responsibilities and outcomes.Transform stage: Perform the reduction, producing csv files. We would need dettl to not try to validate the output against specific database tables, since we often need to create those tables on annex as part of the upload.Load stage: Perform the uploads on the created csv files, updating the stochastic_file and adding new tables on annex.