Manage R based data import.

Manage R based data import.

Manage R based data import.

Details

This object should not be initialised directly. Use dettl to create the object.

Import can be run by working with import object returned by dettl or by running top-level functions. Run the import by working with this object if you want to step through the import process stage by stage and inspect the data after each stage.

Super class

dettl::Import -> RImport

Methods

Public methods

• RImport$reload()

• RImport$read_config()

• RImport$get_extracted_data()

• RImport$get_transformed_data()

• RImport$extract()

• RImport$transform()

• RImport$pre_modify_checks()

• RImport$run_import()

Inherited methods

• dettl::Import$begin_transaction()
• dettl::Import$commit_transaction()
• dettl::Import$format()
• dettl::Import$get_connection()
• dettl::Import$get_log_table()
• dettl::Import$help()
• dettl::Import$initialize()
• dettl::Import$load()
• dettl::Import$rollback_transaction()

---

Method reload()

Reload the objects sources to refresh source code or repair a broken Postgres connection.

Usage

RImport$reload()

---

Method read_config()

Read and parse config from path.

Usage

RImport$read_config()

---

Method get_extracted_data()

Get the extracted data created by the extract step

Usage

RImport$get_extracted_data()

Returns

The extracted data

---

Method get_transformed_data()

Get the transformed data created by the transform step

Usage

RImport$get_transformed_data()

Returns

The transformed data

---

Method extract()

Run the extract stage of the data import

Usage

RImport$extract()

---

Method transform()

Run the transform stage of the data import

Usage

RImport$transform()

---

Method pre_modify_checks()

Run suite of checks to verify that db can be modified

Usage

RImport$pre_modify_checks(dry_run, allow_dirty_git)

Arguments

dry_run

Whether to run in dry run mode. If TRUE then any database changes will be rolled back. Defaults to FALSE.

allow_dirty_git

If TRUE then skips check that the import is up to date with remote git repo. FALSE by default.

---

Method run_import()

Run multiple stages of the data import

Usage

RImport$run_import(
  comment = NULL,
  dry_run = FALSE,
  allow_dirty_git = FALSE,
  stage = c("extract", "transform"),
  save = FALSE
)

Arguments

comment

Optional comment to be written to db log table when import is run.

dry_run

If TRUE then any changes to the database will be rolled back.

allow_dirty_git

If TRUE then skips check that the import is up to date

stage

The stage or stages of the import to be run.

save

Path and name to save data from each stage at, if TRUE then will save to a tempfile.

Examples

path <- dettl:::prepare_test_import(
  system.file("examples", "person_information", package = "dettl"),
  system.file("examples", "dettl_config.yml", package = "dettl"))
import_path <- file.path(path, "person_information")

import <- dettl::dettl(import_path, db_name = "test")
import$extract()
#> Running extract /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/RtmpiqOfW1/file14a91c74a0c1/person_information
#> Running extract tests R/test_extract.R
#> ✔ | F W S  OK | Context
#> 
#> ⠏ |         0 | extract                                                         
#> ⠏ |         0 | extract                                                         
#> ✔ |         1 | extract [0.2s]
#> 
#> ══ Results ═════════════════════════════════════════════════════════════════════
#> Duration: 0.2 s
#> 
#> [ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
#> All extract tests passed.
import$transform()
#> Running transform /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/RtmpiqOfW1/file14a91c74a0c1/person_information
#> Running transform tests R/test_transform.R
#> ✔ | F W S  OK | Context
#> 
#> ⠏ |         0 | transform                                                       
#> ⠏ |         0 | transform                                                       
#> ✔ |         1 | transform
#> 
#> ══ Results ═════════════════════════════════════════════════════════════════════
#> Duration: 0.2 s
#> 
#> [ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
#> All transform tests passed.
import$load()
#> Running load /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/RtmpiqOfW1/file14a91c74a0c1/person_information
#> Running load in a transaction:
#> 	- Running test queries before making any changes
#> 	- Running load step
#> 	- Running test queries after making changes
#> 	- Running load tests R/test_load.R
#> ✔ | F W S  OK | Context
#> 
#> ⠏ |         0 | load                                                            
#> ⠏ |         0 | load                                                            
#> ✔ |         1 | load
#> 
#> ══ Results ═════════════════════════════════════════════════════════════════════
#> [ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
#> All tests passed, commiting changes to database.