Interfaceο
Command lineο
usage: beaver [-h] [--cache CACHE] [--file FILE]
[--log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
{build,list,reset} ...
Named Argumentsο
- --cache, -c
file containing cached information, including composite digests
Default: β.beavercacheβ
- --file, -f
file containing artifact and transform definitions
Default: βbeaver.pyβ
- --log_level, -l
Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL
level of log messages to emit
Default: info
Sub-commands:ο
buildο
Build artifacts.
beaver build [-h] [--num_concurrent NUM_CONCURRENT] [--dry-run] [--all]
[patterns ...]
Positional Argumentsο
- patterns
patterns to match artifacts against
Named Argumentsο
- --num_concurrent, -c
number of concurrent transforms
Default: 1
- --dry-run, -n
print transforms without executing them
Default: False
- --all, -a
match all artifacts
Default: False
listο
List artifacts.
beaver list [-h] [--stale] [--raw] [--all] [patterns ...]
Positional Argumentsο
- patterns
patterns to match artifacts against
Named Argumentsο
- --stale, -s
list only stale artifacts
Default: False
- --raw, -r
list names only without status indicators
Default: False
- --all, -a
match all artifacts
Default: False
resetο
Reset the composite digest of artifacts
beaver reset [-h] [--all] [patterns ...]
Positional Argumentsο
- patterns
patterns to match artifacts against
Named Argumentsο
- --all, -a
match all artifacts
Default: False
Transformsο
- class Download(output: beaver_build.artifacts.File, url: str)ο
Download a file.
- Parameters
output β Output artifact for the downloaded data.
url β Url to download from.
Example
>>> data = bb.File("20news.tar.gz", expected_digest="af604312") >>> bb.Download(data, url="https://ndownloader.figshare.com/files/5975967") Download([] -> [File(20news.tar.gz)])
- class Functional(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact], func: Callable, *args, **kwargs)ο
Apply a python function.
- Parameters
outputs β Artifacts to generate.
inputs β Artifacts consumed by the transform.
func β Function to execute.
*args β Positional arguments passed to
func
.*kwargs β Keyword arguments passed to
func
.
- class Shell(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact], cmd: str, *, env: Optional[dict[str, str]] = None, **kwargs)ο
Execute a command in the shell. See
Subprocess
for details.Example: >>> Shell(βoutput.txtβ, None, βecho hello > output.txtβ) Shell([] -> [File(output.txt)])
- class Subprocess(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact], cmd: Union[str, Iterable[str]], *, env: Optional[dict[str, str]] = None, shell: bool = False, **kwargs)ο
Execute a subprocess.
The transform supports variable substitution using f-strings and Makefile syntax:
$@
represents the first output.$<
represents the first input.$^
represents all inputs.$$@
represents the literal$@
, i.e. double dollars are properly escaped.{outputs[0].name}
represents the name of the first output. The available f-string variables areoutputs
andinputs
, each a list ofArtifact
s.$!
represents the current python interpreter.
Environment variables are inherited by default, but global environment variables for all
Subprocess
transforms can be specified byset_global_env()
or modifyingget_global_env()
, and specific environment variables can be specified using theenv
argument. Environment variables that areNone
are removed from the environment of the transform.- Parameters
outputs β Artifacts to generate.
inputs β Artifacts consumed by the transform.
cmd β Command to execute. Must be a single string if
shell
is truthy and a sequence of strings otherwise.env β Mapping of environment variables.
shell β Whether to execute the command through the shell, e.g. to expand the home directory
~
or pipe information between processes or files using|
,<
, or>
. Seesubprocess.Popen
for details, including security considerations.**kwargs β Keyword arguments passed to
asyncio.subprocess.create_subprocess_shell()
(ifshell == True
) orasyncio.subprocess.create_subprocess_exec()
(ifshell == False
).
- Raises
ValueError β If
shell == True
andcmd
is not a string, orshell == False
andcmd
is not a list of strings.
- ENVο
Default mapping of environment variables for all
Subprocess
transforms.
Example
>>> bb.Subprocess("copy.txt", "input.txt", ["cp", "$<", "$@"]) Subprocess([File(input.txt)] -> [File(copy.txt)])
- classmethod get_global_env() dict ο
Get the global environment variables used by all
Subprocess
transforms.
- classmethod set_global_env(value: dict) None ο
Set the global environment variables used by all
Subprocess
transforms.
- class Transform(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact])ο
Base class for transforms that generates outputs given inputs. Inheriting classes should implement
execute()
.- Parameters
outputs β Artifacts to generate.
inputs β Artifacts consumed by the transform.
- stale_outputsο
Sequence of outputs that are stale and need to be updated.
- evaluate_composite_digests(outputs: Iterable[beaver_build.artifacts.Artifact], inputs: Iterable[beaver_build.artifacts.Artifact]) dict['artifacts.Artifact', bytes] ο
Evaluate composite digests for all
outputs
.A composite digest is defined as the digest of the digests of inputs and the digest of the output, i.e. it is a joint, concise summary of both inputs and outputs. A composite digest is
None
if the corresponding output digest isNone
or any input digests areNone
.- Parameters
inputs β Sequence of input artifacts.
outputs β Sequence of output artifacts.
- Returns
digests β Mapping from outputs to composite digests.
Artifactsο
- class Artifact(name, *args, **kwargs)ο
Artifact generated by a transform.
- Parameters
name β Unique name of the artifact.
expected_digest β Digest expected when the artifact is available.
ignore_groups β Ignore any groups and create a root-level artifact.
- Raises
ValueError β If the given
name
is already in use by another artifact.
- childrenο
Transforms that consume this artifact; empty if the artifact is a leaf of the directed acyclic graph.
- Type
Iterable[beaver_build.transforms.Transform]
- parentο
Transform that generates this artifact;
None
if the artifact is a root of the directed acyclic graph.
- class ArtifactFactory(name, bases, members)ο
Metaclass that creates
Artifact
instances or returns existing instances.
- class File(name, *args, **kwargs)ο
Artifact representing a file.
- Parameters
name β Unique name of the artifact.
expected_digest β Digest expected when the artifact is available.
ignore_groups β Ignore any groups and create a root-level artifact.
- classmethod glob(pattern: str) Iterable[beaver_build.artifacts.File] ο
Create a list of file artifacts based on a glob pattern.
- Parameters
pattern β Pattern passed to
glob.glob()
.- Returns
artifacts β File artifacts.
- class Group(name, *args, **kwargs)ο
Artifact representing a group of other artifacts.
- Parameters
name β Name of the group. The group name is added as a prefix for all artifacts created within the context manager of the group.
ignore_groups β Ignore any groups and create a root-level artifact.
- membersο
Members of the group.
- classmethod append(artifact: beaver_build.artifacts.Artifact) None ο
Append an artifact to the current group if available.
- Parameters
artifact β Artifact to append to the current group.
- async gather_artifacts(*artifacts_or_transforms, num_concurrent: Optional[int] = None) Coroutine ο
Gather one or more artifacts and wrap them in a coroutine.
- Parameters
*artifacts_or_transforms β Artifacts to gather and/or transforms whose artifacts to gather.
num_concurrent β Maximum number of concurrent transforms. Defaults to unrestricted.
- Returns
gathered β Coroutine that awaits all gathered artifacts.
- group_artifacts(*names: str, squeeze=True) list[beaver_build.artifacts.Group] ο
Group artifacts in a context.
- Parameters
names β Name of nested groups.
squeeze β Whether to return a single element if there is only one group.
- Returns
groups β Artifacts representing the nested groups.
- normalize_artifacts(artifacts: Union[str, beaver_build.artifacts.Artifact, Iterable[Union[str, beaver_build.artifacts.Artifact]]]) Iterable[beaver_build.artifacts.Artifact] ο
Normalize one or more artifacts. Strings are implicitly converted to
File
s.- Parameters
artifacts β One or more artifacts or transforms.
- Returns
normalized β Normalized artifacts.