scriptconfig.config module¶
Write simple configs and update from CLI, kwargs, and/or json.
The scriptconfig
provides a simple way to make configurable scripts using a
combination of config files, command line arguments, and simple Python keyword
arguments. A script config object is defined by creating a subclass of
Config
with a default
dict class attribute. An instance of a custom
Config
object will behave similar a dictionary, but with a few
conveniences.
Note
This class implements the old-style legacy Config class, new applications should favor using DataConfig instead, which has simpler boilerplate.
To get started lets consider some example usage:
Example
>>> import scriptconfig as scfg
>>> # In its simplest incarnation, the config class specifies default values.
>>> # For each configuration parameter.
>>> class ExampleConfig(scfg.Config):
>>> __default__ = {
>>> 'num': 1,
>>> 'mode': 'bar',
>>> 'ignore': ['baz', 'biz'],
>>> }
>>> # Creating an instance, starts using the defaults
>>> config = ExampleConfig()
>>> # Typically you will want to update default from a dict or file. By
>>> # specifying cmdline=True you denote that it is ok for the contents of
>>> # `sys.argv` to override config values. Here we pass a dict to `load`.
>>> kwargs = {'num': 2}
>>> config.load(kwargs, cmdline=False)
>>> assert config['num'] == 2
>>> # The `load` method can also be passed a json/yaml file/path.
>>> import tempfile
>>> config_fpath = tempfile.mktemp()
>>> open(config_fpath, 'w').write('{"num": 3}')
>>> config.load(config_fpath, cmdline=False)
>>> assert config['num'] == 3
>>> # It is possible to load only from CLI by setting cmdline=True
>>> # or by setting it to a custom sys.argv
>>> config.load(cmdline=['--num=4', '--mode' ,'fiz'])
>>> assert config['num'] == 4
>>> assert config['mode'] == 'fiz'
>>> # You can also just use the command line string itself
>>> config.load(cmdline='--num=4 --mode fiz')
>>> assert config['num'] == 4
>>> assert config['mode'] == 'fiz'
>>> # Note that using `config.load(cmdline=True)` will just use the
>>> # contents of sys.argv
Todo
[ ] Handle Nested Configs?
[ ] Integrate with Hyrda
[x] Dataclass support - See DataConfig
- class scriptconfig.config.Config(data=None, default=None, cmdline=False, _dont_call_post_init=False)[source]¶
-
Base class for custom configuration objects
A configuration that can be specified by commandline args, a yaml config file, and / or a in-code dictionary. To use, define a class variable named
__default__
and passing it to a dict of default values. You can also use specialValue
classes to denote types. You can also define a method__post_init__
, to postprocess the arguments after this class receives them.Basic usage is as follows.
Create a class that inherits from this class.
Assign the “__default__” class-level variable as a dictionary of options
The keys of this dictionary must be command line friendly strings.
The values of the “defaults dictionary” can be literal values or instances of the
scriptconfig.Value
class, which allows for specification of default values, type information, help strings, and aliases.You may also implement
__post_init__
(function with that takes no args and has no return) to postprocess your results after initialization.When creating an instance of the class the defaults variable is used to make a dictionary-like object. You can override defaults by specifying the
data
keyword argument to either a file path or another dictionary. You can also specifycmdline=True
to allow the contents ofsys.argv
to influence the values of the new object.An instance of the config class behaves like a dictionary, except that you cannot set keys that do not already exist (as specified in the defaults dict).
Key Methods:
dump - dump a json representation to a file
dumps - dump a json representation to a string
argparse - create an
argparse.ArgumentParser
object that is defined by the defaults of this config.load - rewrite the values based on a filepath, dictionary, or command line contents.
- Variables:
_data – this protected variable holds the raw state of the config object and is accessed by the dict-like
_default – this protected variable maintains the default values for this config.
epilog (str) – A class attribute that if specified will add an epilog section to the help text.
Example
>>> # Inherit from `Config` and assign `__default__` >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __default__ = { >>> 'option1': scfg.Value((1, 2, 3), tuple), >>> 'option2': 'bar', >>> 'option3': None, >>> } >>> # You can now make instances of this class >>> config1 = MyConfig() >>> config2 = MyConfig(default=dict(option1='baz'))
- Parameters:
data (object) – filepath, dict, or None
default (dict | None) – overrides the class defaults
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead
of sys.argv.
If a dictionary grants fine grained controls over the args passed to
Config._read_argv()
. Can contain:strict (bool): defaults to False
argv (List[str]): defaults to None
special_options (bool): defaults to True
autocomplete (bool): defaults to False
Defaults to False.
Note
Avoid setting
cmdline
parameter here. Instead prefer to use thecli
classmethod to create a command line aware config instance..- classmethod cli(data=None, default=None, argv=None, strict=True, cmdline=True, autocomplete='auto')[source]¶
Create a commandline aware config instance.
Calls the original “load” way of creating non-dataclass config objects. This may be refactored in the future.
- Parameters:
data (dict | str | None) – Values to update the configuration with. This can be a regular dictionary or a path to a yaml / json file.
default (dict | None) – Values to update the defaults with (not the actual configuration). Note: anything passed to default will be deep copied and can be updated by argv or data if it is specified. Generally prefer to pass directly to data instead.
cmdline (bool) – Defaults to True, which creates and uses an argparse object to interact with the command line. If set to False, then the argument parser is bypassed (useful for invoking a CLI programatically with kwargs and ignoring sys.argv).
argv (List[str]) – if specified, ignore sys.argv and parse this instead.
strict (bool) – if True use
parse_args
otherwise useparse_known_args
. Defaults to True.autocomplete (bool | str) – if True try to enable argcomplete.
- classmethod demo()[source]¶
Create an example config class for test cases
CommandLine
xdoctest -m scriptconfig.config Config.demo xdoctest -m scriptconfig.config Config.demo --cli --option1 fo
Example
>>> from scriptconfig.config import * >>> self = Config.demo() >>> print('self = {}'.format(self)) self = <DemoConfig({...'option1': ...}...)...>... >>> self.argparse().print_help() >>> # xdoc: +REQUIRES(--cli) >>> self.load(cmdline=True) >>> print(ub.urepr(self, nl=1))
- getitem(key)[source]¶
Dictionary-like method to get the value of a key.
- Parameters:
key (str) – the key
- Returns:
the associated value
- Return type:
Any
- setitem(key, value)[source]¶
Dictionary-like method to set the value of a key.
- Parameters:
key (str) – the key
value (Any) – the new value
- update_defaults(default)[source]¶
Update the instance-level default values
- Parameters:
default (dict) – new defaults
- load(data=None, cmdline=False, mode=None, default=None, strict=False, autocomplete=False, _dont_call_post_init=False)[source]¶
Updates the configuration from a given data source.
Any option can be overwritten via the command line if
cmdline
is truthy.- Parameters:
data (PathLike | dict) – Either a path to a yaml / json file or a config dict
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead
of sys.argv.
If a dictionary grants fine grained controls over the args passed to
Config._read_argv()
. Can contain:strict (bool): defaults to False
argv (List[str]): defaults to None
special_options (bool): defaults to True
autocomplete (bool): defaults to False
Defaults to False.
mode (str | None) – Either json or yaml.
cmdline (bool | List[str] | str) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead of sys.argv. Defaults to False.
default (dict | None) – updated defaults. Note: anything passed to default will be deep copied and can be updated by argv or data if it is specified. Generally prefer to pass directly to data instead.
strict (bool) – if True an error will be raised if the command line contains unknown arguments.
autocomplete (bool) – if True, attempts to use the autocomplete package if it is available if reading from sys.argv. Defaults to False.
Note
if cmdline=True, this will create an argument parser.
Example
>>> # Test load works correctly in cmdline True and False mode >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __default__ = { >>> 'src': scfg.Value(None, help=('some help msg')), >>> } >>> data = {'src': 'hi'} >>> self = MyConfig(data=data, cmdline=False) >>> assert self['src'] == 'hi' >>> self = MyConfig(default=data, cmdline=True) >>> assert self['src'] == 'hi' >>> # In 0.5.8 and previous src fails to populate! >>> # This is because cmdline=True overwrites data with defaults >>> self = MyConfig(data=data, cmdline=True) >>> assert self['src'] == 'hi', f'Got: {self}'
Example
>>> # Test load works correctly in strict mode >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __default__ = { >>> 'src': scfg.Value(None, help=('some help msg')), >>> } >>> data = {'src': 'hi'} >>> cmdlinekw = { >>> 'strict': True, >>> 'argv': '--src=hello', >>> } >>> self = MyConfig(data=data, cmdline=cmdlinekw) >>> cmdlinekw = { >>> 'strict': True, >>> 'special_options': False, >>> 'argv': '--src=hello --extra=arg', >>> } >>> import pytest >>> with pytest.raises(SystemExit): >>> self = MyConfig(data=data, cmdline=cmdlinekw)
Example
>>> # Test load works correctly with required >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __default__ = { >>> 'src': scfg.Value(None, help=('some help msg'), required=True), >>> } >>> cmdlinekw = { >>> 'strict': True, >>> 'special_options': False, >>> 'argv': '', >>> } >>> import pytest >>> with pytest.raises(Exception): ... self = MyConfig(cmdline=cmdlinekw)
Example
>>> # Test load works correctly with alias >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __default__ = { >>> 'opt1': scfg.Value(None), >>> 'opt2': scfg.Value(None, alias=['arg2']), >>> } >>> config1 = MyConfig(data={'opt2': 'foo'}) >>> assert config1['opt2'] == 'foo' >>> config2 = MyConfig(data={'arg2': 'bar'}) >>> assert config2['opt2'] == 'bar' >>> assert 'arg2' not in config2
- _normalize_alias_dict(data)[source]¶
- Parameters:
data (dict) – dictionary with keys that could be aliases
- Returns:
keys are normalized to be primary keys.
- Return type:
- _read_argv(argv=None, special_options=True, strict=False, autocomplete=False)[source]¶
Example
>>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> description = 'my CLI description' >>> __default__ = { >>> 'src': scfg.Value(['foo'], position=1, nargs='+'), >>> 'dry': scfg.Value(False), >>> 'approx': scfg.Value(False, isflag=True, alias=['a1', 'a2']), >>> } >>> self = MyConfig() >>> # xdoctest: +REQUIRES(PY3) >>> # Python2 argparse does a hard sys.exit instead of raise >>> import sys >>> if sys.version_info[0:2] < (3, 6): >>> # also skip on 3.5 because of dict ordering >>> import pytest >>> pytest.skip() >>> self._read_argv(argv='') >>> print('self = {}'.format(self)) >>> self = MyConfig() >>> self._read_argv(argv='--src [,]') >>> print('self = {}'.format(self)) >>> self = MyConfig() >>> self._read_argv(argv='--src [,] --a1') >>> print('self = {}'.format(self)) self = <MyConfig({'src': ['foo'], 'dry': False, 'approx': False})> self = <MyConfig({'src': [], 'dry': False, 'approx': False})> self = <MyConfig({'src': [], 'dry': False, 'approx': True})>
>>> self = MyConfig() >>> self._read_argv(argv='p1 p2 p3') >>> print('self = {}'.format(self)) >>> self = MyConfig() >>> self._read_argv(argv='--src=p4,p5,p6!') >>> print('self = {}'.format(self)) >>> self = MyConfig() >>> self._read_argv(argv='p1 p2 p3 --src=p4,p5,p6!') >>> print('self = {}'.format(self)) self = <MyConfig({'src': ['p1', 'p2', 'p3'], 'dry': False, 'approx': False})> self = <MyConfig({'src': ['p4', 'p5', 'p6!'], 'dry': False, 'approx': False})> self = <MyConfig({'src': ['p4', 'p5', 'p6!'], 'dry': False, 'approx': False})>
>>> self = MyConfig() >>> self._read_argv(argv='p1') >>> print('self = {}'.format(self)) >>> self = MyConfig() >>> self._read_argv(argv='--src=p4') >>> print('self = {}'.format(self)) >>> self = MyConfig() >>> self._read_argv(argv='p1 --src=p4') >>> print('self = {}'.format(self)) self = <MyConfig({'src': ['p1'], 'dry': False, 'approx': False})> self = <MyConfig({'src': ['p4'], 'dry': False, 'approx': False})> self = <MyConfig({'src': ['p4'], 'dry': False, 'approx': False})>
>>> special_options = False >>> parser = self.argparse(special_options=special_options) >>> parser.print_help() >>> x = parser.parse_known_args()
- dump(stream=None, mode=None)[source]¶
Write configuration file to a file or stream
- Parameters:
stream (FileLike | None) – the stream to write to
mode (str | None) – can be ‘yaml’ or ‘json’ (defaults to ‘yaml’)
- dumps(mode=None)[source]¶
Write the configuration to a text object and return it
- Parameters:
mode (str | None) – can be ‘yaml’ or ‘json’ (defaults to ‘yaml’)
- Returns:
str - the configuration as a string
- property _description¶
- property _epilog¶
- property _prog¶
- port_to_dataconf()[source]¶
Helper that will write the code to express this config as a DataConfig.
CommandLine
xdoctest -m scriptconfig.config Config.port_to_dataconf
Example
>>> import scriptconfig as scfg >>> self = scfg.Config.demo() >>> print(self.port_to_dataconf())
- classmethod port_click(click_main, name='MyConfig', style='dataconf')[source]¶
Example
@click.command() @click.option(’–dataset’, required=True, type=click.Path(exists=True), help=’input dataset’) @click.option(’–deployed’, required=True, type=click.Path(exists=True), help=’weights file’) def click_main(dataset, deployed):
…
- classmethod port_argparse(parser, name='MyConfig', style='dataconf')[source]¶
Generate the corresponding scriptconfig code from an existing argparse instance.
- Parameters:
parser (argparse.ArgumentParser) – existing argparse parser we want to port
name (str) – the name of the config class
style (str) – either ‘orig’ or ‘dataconf’
- Returns:
code to create a scriptconfig object that should work similarly to the existing argparse object.
- Return type:
Note
The correctness of this function is not guarenteed. This only works perfectly in simple cases, but in complex cases it may not produce 1-to-1 results, however it will provide a useful starting point.
Todo
[X] Handle “store_true”.
[ ] Argument groups.
[ ] Handle mutually exclusive groups
Example
>>> import scriptconfig as scfg >>> import argparse >>> parser = argparse.ArgumentParser(description='my argparse') >>> parser.add_argument('pos_arg1') >>> parser.add_argument('pos_arg2', nargs='*') >>> parser.add_argument('-t', '--true_dataset', '--test_dataset', help='path to the groundtruth dataset', required=True) >>> parser.add_argument('-p', '--pred_dataset', help='path to the predicted dataset', required=True) >>> parser.add_argument('--eval_dpath', help='path to dump results') >>> parser.add_argument('--draw_curves', default='auto', help='flag to draw curves or not') >>> parser.add_argument('--score_space', default='video', help='can score in image or video space') >>> parser.add_argument('--workers', default='auto', help='number of parallel scoring workers') >>> parser.add_argument('--draw_workers', default='auto', help='number of parallel drawing workers') >>> group1 = parser.add_argument_group('mygroup1') >>> group1.add_argument('--group1_opt1', action='store_true') >>> group1.add_argument('--group1_opt2') >>> group2 = parser.add_argument_group() >>> group2.add_argument('--group2_opt1', action='store_true') >>> group2.add_argument('--group2_opt2') >>> mutex_group3 = parser.add_mutually_exclusive_group() >>> mutex_group3.add_argument('--mgroup3_opt1') >>> mutex_group3.add_argument('--mgroup3_opt2') >>> text = scfg.Config.port_argparse(parser, name='PortedConfig', style='dataconf') >>> print(text) >>> # Make an instance of the ported class >>> vals = {} >>> exec(text, vals) >>> cls = vals['PortedConfig'] >>> self = cls(**{'true_dataset': 1, 'pred_dataset': 1}) >>> recon = self.argparse() >>> print('recon._actions = {}'.format(ub.urepr(recon._actions, nl=1)))
- port_to_argparse()[source]¶
Attempt to make code for a nearly-equivalent argparse object.
This code only handles basic cases. Some of the scriptconfig magic is dropped so we dont need to rely on custom actions.
The idea is that sometimes we can’t depend on scriptconfig, so it would be nice to be able to translate an existing scriptconfig class to the nearly equivalent argparse code.
- SeeAlso:
Config.argparse()
- creates a real argparse object
- Returns:
code to construct a similar argparse object
- Return type:
CommandLine
xdoctest -m scriptconfig.config Config.port_to_argparse
Example
>>> import scriptconfig as scfg >>> class SimpleCLI(scfg.DataConfig): >>> data = scfg.Value(None, help='input data', position=1) >>> self_or_cls = SimpleCLI() >>> text = self_or_cls.port_to_argparse() >>> print(text) >>> # Test that the generated code is executable >>> ns = {} >>> exec(text, ns, ns) >>> parser = ns['parser'] >>> args1 = parser.parse_args(['foobar']) >>> assert args1.data == 'foobar' >>> # Looks like we cant do positional or key/value easilly >>> #args1 = parser.parse_args(['--data=blag']) >>> #print('args1 = {}'.format(ub.urepr(args1, nl=1)))
- property namespace¶
Access a namespace like object for compatibility with argparse
- Returns:
argparse.Namespace
- to_omegaconf()[source]¶
Creates an omegaconfig version of this.
- Return type:
omegaconf.OmegaConf
Example
>>> # xdoctest: +REQUIRES(module:omegaconf) >>> import scriptconfig >>> self = scriptconfig.Config.demo() >>> oconf = self.to_omegaconf()
- argparse(parser=None, special_options=False)[source]¶
construct or update an argparse.ArgumentParser CLI parser
- Parameters:
parser (None | argparse.ArgumentParser) – if specified this parser is updated with options from this config.
special_options (bool, default=False) – adds special scriptconfig options, namely: –config, –dumps, and –dump.
- Returns:
a new or updated argument parser
- Return type:
CommandLine
xdoctest -m scriptconfig.config Config.argparse:0 xdoctest -m scriptconfig.config Config.argparse:1
Todo
A good CLI spec for lists might be
# In the case where
key
ends with and=
, assume the list is # given as a comma separated string with optional square brakets at # each end.–key=[f]
# In the case where
key
does not end with equals and we know # the value is supposd to be a list, then we consume arguments # until we hit the next one that starts with ‘–’ (which means # that list items cannot start with – but they can contains # commas)FIXME:
In the case where we have an nargs=’+’ action, and we specify the option with an =, and then we give position args after it there is no way to modify behavior of the action to just look at the data in the string without modifying the ArgumentParser itself. The action object has no control over it. For example –foo=bar baz biz will parse as [baz, biz] which is really not what we want. We may be able to overload ArgumentParser to fix this.
Example
>>> # You can now make instances of this class >>> import scriptconfig >>> self = scriptconfig.Config.demo() >>> parser = self.argparse() >>> parser.print_help() >>> # xdoctest: +REQUIRES(PY3) >>> # Python2 argparse does a hard sys.exit instead of raise >>> ns, extra = parser.parse_known_args()
Example
>>> # You can now make instances of this class >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __description__ = 'my CLI description' >>> __default__ = { >>> 'path1': scfg.Value(None, position=1, alias='src'), >>> 'path2': scfg.Value(None, position=2, alias='dst'), >>> 'dry': scfg.Value(False, isflag=True), >>> 'approx': scfg.Value(False, isflag=False, alias=['a1', 'a2']), >>> } >>> self = MyConfig() >>> special_options = True >>> parser = None >>> parser = self.argparse(special_options=special_options) >>> parser.print_help() >>> self._read_argv(argv=['objection', '42', '--path1=overruled!']) >>> print('self = {!r}'.format(self))
Example
>>> # Test required option >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __description__ = 'my CLI description' >>> __default__ = { >>> 'path1': scfg.Value(None, position=1, alias='src'), >>> 'path2': scfg.Value(None, position=2, alias='dst'), >>> 'dry': scfg.Value(False, isflag=True), >>> 'important': scfg.Value(False, required=True), >>> 'approx': scfg.Value(False, isflag=False, alias=['a1', 'a2']), >>> } >>> self = MyConfig(data={'important': 1}) >>> special_options = True >>> parser = None >>> parser = self.argparse(special_options=special_options) >>> parser.print_help() >>> self._read_argv(argv=['objection', '42', '--path1=overruled!', '--important=1']) >>> print('self = {!r}'.format(self))
Example
>>> # Is it possible to the CLI as a key/val pair or an exist bool flag? >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __default__ = { >>> 'path1': scfg.Value(None, position=1, alias='src'), >>> 'path2': scfg.Value(None, position=2, alias='dst'), >>> 'flag': scfg.Value(None, isflag=True), >>> } >>> self = MyConfig() >>> special_options = True >>> parser = None >>> parser = self.argparse(special_options=special_options) >>> parser.print_help() >>> print(self._read_argv(argv=[], strict=True)) >>> # Test that we can specify the flag as a pure flag >>> print(self._read_argv(argv=['--flag'])) >>> print(self._read_argv(argv=['--no-flag'])) >>> # Test that we can specify the flag with a key/val pair >>> print(self._read_argv(argv=['--flag', 'TRUE'])) >>> print(self._read_argv(argv=['--flag=1'])) >>> print(self._read_argv(argv=['--flag=0'])) >>> # Test flag and positional >>> self = MyConfig() >>> print(self._read_argv(argv=['--flag', 'TRUE', 'SUFFIX'])) >>> self = MyConfig() >>> print(self._read_argv(argv=['PREFIX', '--flag', 'TRUE'])) >>> self = MyConfig() >>> print(self._read_argv(argv=['--path2=PREFIX', '--flag', 'TRUE']))
Example
>>> # Test groups >>> import scriptconfig as scfg >>> class MyConfig(scfg.Config): >>> __description__ = 'my CLI description' >>> __default__ = { >>> 'arg1': scfg.Value(None, group='a'), >>> 'arg2': scfg.Value(None, group='a', alias='a2'), >>> 'arg3': scfg.Value(None, group='b'), >>> 'arg4': scfg.Value(None, group='b', alias='a4'), >>> 'arg5': scfg.Value(None, mutex_group='b', isflag=True), >>> 'arg6': scfg.Value(None, mutex_group='b', alias='a6'), >>> } >>> self = MyConfig() >>> parser = self.argparse() >>> parser.print_help() >>> print(self.port_argparse(parser)) >>> import pytest >>> import argparse >>> with pytest.raises(SystemExit): >>> self._read_argv(argv=['--arg6', '42', '--arg5', '32']) >>> # self._read_argv(argv=['--arg6', '42', '--arg5']) # Strange, this does not cause an mutex error >>> self._read_argv(argv=['--arg6', '42']) >>> self._read_argv(argv=['--arg5']) >>> self._read_argv(argv=[])
- default = {}¶
- normalize()¶
overloadable function called after each load