Search for orderly reports matching criteria — orderly

Search for orderly reports matching criteria. This can be used to find reports where a particular parameter or tag was used (it will likely be expanded as time goes on - let us know if that would be useful). We search within versions of a single report only.

Usage

orderly_search(
  query,
  name,
  parameters = NULL,
  draft = FALSE,
  root = NULL,
  locate = TRUE,
  remote = NULL
)

Arguments

query: The query string - see details and examples
name: Name of the report to search. Only a single report can be searched at once.
parameters: Named list of parameters (as would be passed to orderly_run()) if your query uses parameters on the right-hand-side of an expression.
draft: Should draft reports be used searched? This should be used only in development. Valid values are logical (TRUE, FALSE) or use the string newer to use draft reports where they are newer than archive reports. For consistency, always and never are equivalent to TRUE and FALSE, respectively.
root: The path to an orderly root directory, or NULL (the default) to search for one from the current working directory if locate is TRUE.
locate: Logical, indicating if the configuration should be searched for. If TRUE and config is not given, then orderly looks in the working directory and up through its parents until it finds an orderly_config.yml file.
remote: A remote to use, if you want to apply the query remotely. If this is used then draft cannot be set to TRUE as remotes do not expose draft reports.

Value

A character vector of matching report ids, possibly zero-length. If the query is a "latest" query, then exactly one report id, possibly NA.

Details

The query syntax is deliberately very simple; it may expand a bit later. At this point you can search for parameters and for tags, and these can be combined. Note that if you are using OrderlyWeb, then only orderly (and not OrderlyWeb) tags are searched.

The idea here is that the queries can be used to find ids that match certain criteria for use as dependencies. This function lets you work out what would be resolved by the query, and using this query string in a depends: section will let you select a report that matches some criteria. For example, suppose that you have report A that takes a parameter "fruit" with values like "apple", "banana", and a report B that depends on A. You could then write:

depends:
  A:
    id: latest(parameter:fruit == "apple")
    uses:
      summary.csv: summary.csv

To get the summary.csv file out of the latest report A that was run with the "fruit" parameter set to "apple". If "B" itself takes parameters, you can use those parameters in these query expressions like

depends:
  A:
    id: latest(parameter:fruit == target_fruit)
    uses:
      summary.csv: summary.csv

(assuming that B takes a parameter target_fruit).

The syntax for tags is simpler, one uses tag:tagname to test for presence of a tag called "tagname".

Search queries can be joined by && and || and grouped using parentheses, these groups (or tags) can be negated with !, so a complicated query expression might look like:

(parameter:fruit == "apple" && !tag:weekly) || parameter:fruit == "banana"

Be careful of comparing floating point numbers with == or != as they may not always return what you expect (for example sqrt(3)^2 == 3 is FALSE).

In the documentation and error messages we may refer to the left-hand-side of : as a "namespace". At this point the only supported namespaces are tag and parameter.

Examples

# We need a few reports here to actually query.  There is a report in
# the "demo" example called "other" that takes a parameter "nmin",
# which is used to filter data - it's not terribly important what it
# does here, but it can give us a set of reports to use.

# The demo set also includes configuration for two tags, called
# "dataset" and "plot" - the "dataset" tag will always be applied
# as it is listed in the orderly.yml but we can still add the
# "plot" tag interactively
root <- orderly::orderly_example("demo")

# A helper function to mass-produce reports will reduce noise a bit
run1 <- function(nmin, tags = NULL) {
  id <- orderly_run("other", root = root, echo = FALSE,
                    parameters = list(nmin = nmin), tags = tags)
  orderly_commit(id, root = root)
  id
}

ids <- c(run1(0.1), run1(0.2, "plot"), run1(0.3))
#> [ name       ]  other
#> [ id         ]  20230621-105039-0c9814f2
#> [ sources    ]  functions.R
#> [ parameter  ]  nmin: 0.1
#> [ start      ]  2023-06-21 10:50:39
#> [ data       ]  source => extract: 19 x 2
#> [ parameter  ]  nmin: 0.1
#> [ end        ]  2023-06-21 10:50:39
#> [ elapsed    ]  Ran report in 0.0174346 secs
#> [ artefact   ]  summary.csv: 08a4566d063098080bfd318f675926f2
#> [ ...        ]  graph.png: 9d98e56ce7726a1523c1b03eb0e4efb4
#> [ commit     ]  other/20230621-105039-0c9814f2
#> [ copy       ]
#> [ import     ]  other:20230621-105039-0c9814f2
#> [ success    ]  :)
#> [ name       ]  other
#> [ id         ]  20230621-105039-4d2d7e97
#> [ sources    ]  functions.R
#> [ parameter  ]  nmin: 0.2
#> [ start      ]  2023-06-21 10:50:39
#> [ data       ]  source => extract: 19 x 2
#> [ parameter  ]  nmin: 0.2
#> [ end        ]  2023-06-21 10:50:39
#> [ elapsed    ]  Ran report in 0.016361 secs
#> [ artefact   ]  summary.csv: cd50fcc04b94e221751a0e9d9b03baeb
#> [ ...        ]  graph.png: 994b6ce8c97b6c137d1db1382be0580d
#> [ commit     ]  other/20230621-105039-4d2d7e97
#> [ copy       ]
#> [ import     ]  other:20230621-105039-4d2d7e97
#> [ success    ]  :)
#> [ name       ]  other
#> [ id         ]  20230621-105039-7679a8d2
#> [ sources    ]  functions.R
#> [ parameter  ]  nmin: 0.3
#> [ start      ]  2023-06-21 10:50:39
#> [ data       ]  source => extract: 17 x 2
#> [ parameter  ]  nmin: 0.3
#> [ end        ]  2023-06-21 10:50:39
#> [ elapsed    ]  Ran report in 0.01739478 secs
#> [ artefact   ]  summary.csv: 54a56f371a78d7dabeb7593b81f5f5d6
#> [ ...        ]  graph.png: 27583938cf0eb43dd486f546efe593c7
#> [ commit     ]  other/20230621-105039-7679a8d2
#> [ copy       ]
#> [ import     ]  other:20230621-105039-7679a8d2
#> [ success    ]  :)

# We can then ask for all reports where the parameter nmin was more
# than some value
orderly::orderly_search("parameter:nmin > 0.15", "other", root = root)
#> [1] "20230621-105039-4d2d7e97" "20230621-105039-7679a8d2"

# Or use "&&" to find tags within a range
orderly::orderly_search("parameter:nmin > 0.1 && parameter:nmin < 0.3",
                        "other", root = root)
#> [1] "20230621-105039-4d2d7e97"

# If a parameter is not present in some versions of a report you
# can use is.null to test for it (this is only ever the case if
# you have altered a report definition to add or remove a
# parameter)
orderly::orderly_search("is.null(parameter:nmin)", "other", root = root)
#> character(0)

# We can look for tags
orderly::orderly_search("tag:plot", "other", root = root)
#> [1] "20230621-105039-4d2d7e97"

# or exclude them
orderly::orderly_search("!tag:plot", "other", root = root)
#> [1] "20230621-105039-0c9814f2" "20230621-105039-7679a8d2"

# or combine that with the presence/absence of a tag
orderly::orderly_search("parameter:nmin > 0.15 && !tag:plot",
                        "other", root = root)
#> [1] "20230621-105039-7679a8d2"

# Use latest() over a query to find the latest report matching the
# query expression.
orderly::orderly_search("latest(parameter:nmin > 0.15)",
                        "other", root = root)
#> [1] "20230621-105039-7679a8d2"

# If no reports are found, then a zero-length character vector is returned
orderly::orderly_search("parameter:nmin > 0.4", "other", root = root)
#> character(0)

# Or, in the case of latest(), NA
orderly::orderly_search("latest(parameter:nmin > 0.4)",
                        "other", root = root)
#> [1] NA