When doing a grid-search over some hyper-parameters of an algorithm I often
need to specify the possible value space for each parameter and then
evaluate on every combination of these values.
In this case, the product function of the itertools module is handy:
from itertools import product
def train_and_evaluate(lr, num_layers, dataset):
print(f'training on {dataset} with {num_layers=} and {lr=}')
params = (
(0.01, 0.1, 0.5), #learning rate
range(10, 40, 10), #number of hidden neurons
('MNLI', 'SNLI', 'QNLI') # training dataset
)
for config in product(*params):
train_and_evaluate(*config)
However, this code relies on the order of parameters in the function
train_and_evaluate, which could be considered not very pythonic, as
explicit is better than implicit.
In the same way, the 'intention' for the values is only clear with the
context of the function signature (or the comments, if someone bothered to
write them).
Therefore I propose a new variant of the product() function that supports
Mappings instead of Iterables to enable to explicitly specify the function
of an iterables values in the product. A trivial implementation could be:
def named_product(repeat=1, **kwargs):
for combination in itertools.product(*kwargs.values(), repeat=repeat):
yield dict(zip(kwargs.keys(), combination))
which could then be used like this:
params = {
'lr': (0.01, 0.1, 0.5), #learning rate
'num_layers': range(10, 40, 10), #number of hidden neurons
'dataset': ('MNLI', 'SNLI', 'QNLI') # training dataset
}
for config in named_product(**params):
train_and_evaluate(**config)
This has the advantage that the order of the parameters in the kwargs does
not depend on the method signature of train_and_evaluate.
Open Questions
-
Support scalar values?
I the example use-case it may be nice to not have each kwarg to be an
iterable, because you may want to specify all parameters to the function,
even if they do not vary in the product items (factors?)
params = named_product(a=(1, 2), b=3)
instead of
params = named_product(a=(1, 2), b=(3, ))
However, this may be unexpected to users when they supply a string which
would then result in an iteration over its characters.
-
Would this may be suited for the more-itertools package?
However, I would love to have this build-in and not depend on a
non-stdlib module
-
More functionality:
In my toolbox-package I implemented a variant with much more
functionality:
<https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html>
https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html. This
version offers additional functionality compared to the proposed function:
- accept both, a dict as *arg or **kwargs and merges them
- accepts scalar values, not only iterables but treats strings as
scalars
- if the value of any item is a dict itself (not scalar or Sequence),
iterate over the nested dict as a seperate call to named_product
would and
update the 'outer' dict with those values
- copies values before yielding which is useful if the returned
values are mutable (can be excluded)
Maybe some of these features would also be useful for a general version?
However, this version is mostly developed for my special use-case and some
behaviour is solely based on my peculiarities
Example usage:
Here is an example from my actual code which may clarify some decisions on
functionality of the features of
<https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html>
https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html
configs = named_product({
'nb_epoch': 3,
'acquisition_iterations': 0,
'training_function': {
partial(train_bert, decoder_function=create_decoder):{
'decoder_name': 'FFNN'
},
partial(train_bert, decoder_function=create_cnn_decoder):{
'decoder_name': 'CNN'
}
},
'training_data': partial(load_glue, initial_training_data_size=10),
'number_queries': 100,
'batch_size': 64,
'dataset_class': [QNLIDataset, MNLIDataset],
'num_frozen_layers': 0,
'acquire_function': {
acquire_uncertain: {
'certainty_function': calculate_random,
'predict_function': 'predict',
'dropout_iterations': 1
}
}
})
_______________________________________________
Python-ideas mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/IAVDAMJUQXI4GBNH6JXBYJE5MJYQRVXP/
Code of Conduct: http://python.org/psf/codeofconduct/
