Re: 2 + 2 = 5
On Friday, July 6, 2012 8:39:58 AM UTC+10, Andrew Cooper wrote: > On 05/07/2012 22:46, Evan Driscoll wrote: > > On 01/-10/-28163 01:59 PM, Alexander Blinne wrote: > >> 5+0 is actually 4+0, because 5 == 4, so 5+0 gives 4. > >> 5+1 is actually 4+1, which is 5, but 5 is again 4. > >> 5+2 is 4+2 which is 6. > > > > Now all I can think is "Hoory for new math, new-hoo-hoo math" > :-) > > > > Evan > > It wont do you a bit of good to read new math! > > (My mind was on exactly the same track) > > ~Andrew +1 -- http://mail.python.org/mailman/listinfo/python-list
Force Python ast to emit lines no longer than $length
I've written a library that works at the ast level. Sometimes the generated output goes over the linter line length limit. "foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz" How do I generate this kind of code instead? "foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_" "foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz" (also happy with the \ and parenthesised variants) [cross-posted: stackoverflow.com/q/65800797] The only thing I can think of doing—retaining support for 3.6, 3.7, 3.8, 3.9, and 3.10a4—is to contribute to both astor and the builtin ast.unparse… -- https://mail.python.org/mailman/listinfo/python-list
Re: Force Python ast to emit lines no longer than $length
I ended up adding word-wrap support directly to my code-generation: https://github.com/SamuelMarks/doctrans/commit/6147b21e168b66623aa1be95cb38b1969daa5147 Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn <https://linkedin.com/in/samuelmarks> On Wed, Jan 20, 2021 at 2:05 PM Samuel Marks wrote: > I've written a library that works at the ast level. Sometimes the > generated output goes over the linter line length limit. > > > "foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz" > > How do I generate this kind of code instead? > > "foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz_" > "foo_bar_can_haz_foo_bar_can_haz_foo_bar_can_haz" > > (also happy with the \ and parenthesised variants) [cross-posted: > stackoverflow.com/q/65800797] > > The only thing I can think of doing—retaining support for 3.6, 3.7, > 3.8, 3.9, and 3.10a4—is to contribute to both astor and the builtin > ast.unparse… > -- https://mail.python.org/mailman/listinfo/python-list
ABC with abstractmethod: kwargs on Base, explicit names on implementation
After a discussion on #python on Freenode, I'm here.
The gist of it is:
> Falc - Signature of method 'Pharm.foo()' does not match signature of base
> method in class 'Base'
What's the right way of specialising the children and leaving the Base
pretty much empty?
Specifically I want:
• All implementers to be forced to implement Base's method
• Base to be able to do very basic things with kwargs, namely log it
(to a file like stdout or a db)
• Support for [at least] Python 3.6–3.9 (I don't think `Protocol` has
been backported?)
• Implementors to have access to the proper name, rather than having
to unpack kwargs
Should I be using a Base class? - Metaclass? - Something else
entirely? - I know giving `b` a default value would resolve the
[PyCharm] linter error… but I want it to be a required argument.
Full example:
from abc import ABC, abstractmethod
class Base(ABC):
@abstractmethod
def foo(self, a, **kwargs):
"""
:param a: var
:type a: ```int```
:param **kwargs: keyword args
:type **kwargs: ``dict``
"""
print(a)
class Pharm(Base):
def foo(self, a, b):
"""
:param a: var
:type a: ```int```
:param b: var
:type b: ```int```
"""
super(Pharm, self).foo(a=a)
Thanks,
Samuel Marks
Charity <https://sydneyscientific.org> | consultancy
<https://offscale.io> | open-source <https://github.com/offscale> |
LinkedIn <https://linkedin.com/in/samuelmarks>
--
https://mail.python.org/mailman/listinfo/python-list
Re: ABC with abstractmethod: kwargs on Base, explicit names on implementation
The main thing I want is type safety. I want Python to complain if the callee uses the wrong argument types, and to provide suggestions on what's needed and info about it. Without a base class I can just have docstrings and type annotations to achieve that. What can I use that will require all implementers to have a minimum of the same properties and arguments, but also allow them to add new properties and arguments? Preferably I would like this all to happen before compile/interpret time. This also opens it up to AST driven stub generation; as can be found in various IDEs (and also a little project I wrote that `import ast`). What are my options here? - It doesn't seem like the metaclass or decorator approaches will help here… Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn <https://linkedin.com/in/samuelmarks> Samuel Marks wrote at 2020-8-24 18:24 +1000: >After a discussion on #python on Freenode, I'm here. > >The gist of it is: >> Falc - Signature of method 'Pharm.foo()' does not match signature of base >> method in class 'Base' > >What's the right way of specialising the children and leaving the Base >pretty much empty? > >Specifically I want: >• All implementers to be forced to implement Base's method >• Base to be able to do very basic things with kwargs, namely log it >(to a file like stdout or a db) >• Support for [at least] Python 3.6–3.9 (I don't think `Protocol` has >been backported?) >• Implementors to have access to the proper name, rather than having >to unpack kwargs > >Should I be using a Base class? - Metaclass? - Something else >entirely? - I know giving `b` a default value would resolve the >[PyCharm] linter error… but I want it to be a required argument. > >Full example: > >from abc import ABC, abstractmethod > > >class Base(ABC): >@abstractmethod >def foo(self, a, **kwargs): > ... >class Pharm(Base): >def foo(self, a, b): > ... Python make a distinction between positional and keyword arguments. A positional argument is identified by its position in the parameter list; a keyword argument is identified by its name. `**` introduces arbitrary keyword arguments. In a call, all those arguments must be passed as "name=value". In your case above, `b` is not a keyword argument and thus is not matched by `**kwargs`. The error you observe is justified. You can try: class Base(ABC): @abstractmethod def foo(self, a, *args, **kwargs): ... class Pharm(Base): def foo(self, a, b, *args, **kwargs): ... Note that the base method signature allows arbitrary positional and keyword arguments. As a consequence, derived methods must do the same. If this is not what you want, you might want to explore the use of a decorator or a meta class rather than a base class. -- https://mail.python.org/mailman/listinfo/python-list
Re: ABC with abstractmethod: kwargs on Base, explicit names on implementation
So there is no way to get a AOT error when the two functions aren't implemented, if the two functions have different required arguments on implementors? To paint this picture for why I need this, say the first is: class Base(ABC): @abstractclass def train(self, epochs): asset epochs is not None and epochs > 0 …and the implementation is: class TensorFlow(Base): def train(self, learning_rate, epochs, callbacks, metrics, loss, optimizer): super(Base, self).__init__(epochs=epochs) [+ docstrings and PEP484 types; excluded here for concision] So how do I enable this use-case? - Obviously it doesn't make sense to include these on the base class, and giving them default values probably doesn't make sense either. You're saying I shouldn't be using ABC for this. So what should I be using? The requirement I'm trying to enforce is that each implementor has a `train` callable attached. Preferably with one required field (but this is just a nice-to-have). BTW: I have enabled an argument inheritance use-case through creating a library with the builtin `ast` module. So now you can have `def train_c(self, config)` with config referring to an instance of a class which inherits from a base config class. See here for the implementation https://github.com/SamuelMarks/doctrans However if I only have functions which accept an instance of a class as the argument, then that will make it less user-friendly to the API caller. So I really am looking for handling both interfaces in a straightforward manner. Thanks for your suggestions, Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn <https://linkedin.com/in/samuelmarks> On Fri, Aug 28, 2020 at 3:51 AM Dieter Maurer wrote: > > Samuel Marks wrote at 2020-8-27 15:58 +1000: > >The main thing I want is type safety. I want Python to complain if the > >callee uses the wrong argument types, and to provide suggestions on > >what's needed and info about it. > > > >Without a base class I can just have docstrings and type annotations > >to achieve that. > > > >What can I use that will require all implementers to have a minimum of > >the same properties and arguments, but also allow them to add new > >properties and arguments? > > A main paradigm of object oriented programming is the > ability to use a derived class instance with knowledge only > about the base class. This implies that in many cases, you > need not know the concrete class because any instance of a derived > class must have the essential behavior of the base class instances. > > This paradigm imposes limitations on the allowable signature changes. > An overriding method may add further parameters but all those > must have default values - otherwise, the use with base class knowledge > only would cause errors. > > > Preferably I would like this all to happen before compile/interpret > time. > > Use a "lint" version to check compatibilty. -- https://mail.python.org/mailman/listinfo/python-list
Re: ABC with abstractmethod: kwargs on Base, explicit names on implementation
Putting Liskov substitution principal to one side, I had a suggestion to follow PEP3102, and do `def train(self, *, epochs):`… It's a rather simple suggestion that I just might take aboard. In response to Dieter: My purpose for using a base class is so that the highest level interface—say a CLI, GUI, REST and RPC API—can all be configured in one place, but enable the greatest possible divergence from a parameter standpoint. An alternative here [in this domain] would be to create a new Keras/scikit.learn. I.e., one consistent interface that is to be implemented in full by each framework. The reason I don't want to take this alternative is manyfold, that I don't need to get in here (happy to start a new thread if you're interested). Two major advantages of this approach are: 0. Implementers can be [pretty much] as divergent as they want, with different default and required parameters; and even semantics [though I do have `assert`s to force `method0` to be executed before `method1`]; 1. Implementers don't need to create their own CLIs, GUIs, REST and RPC APIs Two major disadvantages: 0. Parameters aren't known ahead of time, so you can do the whole `Animal` [duck type] trick where `Animal` could actually be `Dog` or `Horse` [making the obvious Base `ABC` & `abstractmethod` approach 'wrong'] 1. Each implementer framework can maintain wildly different internal APIs, making more hardcore integrations—in a multi-ML sense—far more difficult Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn <https://linkedin.com/in/samuelmarks> On Sat, Aug 29, 2020 at 8:24 PM Dieter Maurer wrote: > > Samuel Marks wrote at 2020-8-29 19:14 +1000: > >So there is no way to get a AOT error when the two functions aren't > >implemented, if the two functions have different required arguments on > >implementors? > > > >To paint this picture for why I need this, say the first is: > > > >class Base(ABC): > >@abstractclass > >def train(self, epochs): > >asset epochs is not None and epochs > 0 > > > >…and the implementation is: > > > >class TensorFlow(Base): > >def train(self, learning_rate, epochs, callbacks, metrics, loss, > > optimizer): > >super(Base, self).__init__(epochs=epochs) > > > >[+ docstrings and PEP484 types; excluded here for concision] > > > >So how do I enable this use-case? - Obviously it doesn't make sense to > >include these on the base class, and giving them default values > >probably doesn't make sense either. > > > >You're saying I shouldn't be using ABC for this. So what should I be using? > > What is your purpose to use a base class in the first place -- > and especially one where `train` has this signature? > > You signal with this base class, that it makes sense to > call `train` with a single positional argument. > But `train` in the derived class cannot be called in this way. > > Base classes are there to capture common features of several > related classes. In your example above, this is not the case. > Do not use base classes in this way. > > >The requirement I'm trying to enforce is that each implementor has a > >`train` callable attached. > >Preferably with one required field (but this is just a nice-to-have). > > Read the Python documentation about metaclasses and the special methods > related to class derivation. Both approaches would allow you > to check whatever you want. > > > ... > >However if I only have functions which accept an instance of a class > >as the argument, then that will make it less user-friendly to the API > >caller. So I really am looking for handling both interfaces in a > >straightforward manner. > > Any system makes some things easy and other things difficult. > Try to stay with the essential paradigms underlaying the system -- > in order to use the easy things whenever possible. > > In your case, this means that the signature of an overriding > method must be quite similar to that of the overridden method. -- https://mail.python.org/mailman/listinfo/python-list
CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?
Previously I have solved related problems with explicit `-}` and `-{`
(or `-b`) as in `nginxctl`:
```
$ python -m nginxctl serve --temp_dir '/tmp' \
-b 'server' \
--server_name 'localhost' --listen '8080' \
-b location '/' \
--root '/tmp/wwwroot' \
-} \
-}
nginx is running. Stop with: /usr/local/bin/nginx -c /tmp/nginx.conf -s stop
```
To illustrate the issue, using `ml-params` and ml-params-tensorflow:
```
$ python -m ml_params --engine 'tensorflow' train --help
usage: python -m ml_params train [-h]
--optimizer
{Adadelta,Adagrad,Adam,Adamax,Ftrl,Nadam,RMSprop}
Run the training loop for your ML pipeline.
optional arguments:
-h, --helpshow this help message and exit
--optimizer {Adadelta,Adagrad,Adam,Adamax,Ftrl,Nadam,RMSprop}
The optimizer
```
Idea: preprocess `sys.argv` so that this syntax would work
`--optimizer Adam[learning_rate=0.01]`*
*square rather than round so as not to require escape characters or
quoting in `sh`
Unfortunately this approach wouldn't give nice `--help` text.
What's the right solution here?
Thanks,
Samuel Marks
Charity <https://sydneyscientific.org> | consultancy
<https://offscale.io> | open-source <https://github.com/offscale> |
LinkedIn <https://linkedin.com/in/samuelmarks>
--
https://mail.python.org/mailman/listinfo/python-list
Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?
Yes it’s my module, and I’ve been using argparse https://github.com/SamuelMarks/ml-params No library I’ve found provides a solution to CLI argument parsing for my use-case. So I’ll write one. But what should it look like, syntactically and semantically? On Fri, 16 Oct 2020 at 3:14 am, Dieter Maurer wrote: > Samuel Marks wrote at 2020-10-15 20:53 +1100: > > ... > >To illustrate the issue, using `ml-params` and ml-params-tensorflow: > > ... > >What's the right solution here? > > While Python provides several modules in its standard library > to process parameters (e.g. the simple `getopt` and the flexible > `argparse`), > it is up to the "application" whether it uses such a module (and which one) > or whether it handle arguments on its own. > > Apparently, `ml_param` is not a staudard Python module. > Is it a package of your own? Then I suggest to check `argparse` whether > it supports your use case (I know, it can be customized to do it, > but maybe, it does it already out of the box). > > If `ml_param` is a third party module, then the question > is actually an `ml_param` question. Ask its support mailing lists > or have a look at its source. > > -- Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn < https://linkedin.com/in/samuelmarks> -- https://mail.python.org/mailman/listinfo/python-list
Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?
--optimizer Adam,learning_rate=0.01,something_else=3 That syntax isn’t so bad! =] How would you suggest the help text for this looks? (don’t worry about implementation, just what goes to stdout/stderr) PS: Yeah I used square brackets for my Bash arrays On Fri, 16 Oct 2020 at 10:26 am, Cameron Simpson wrote: > One other thing: > > On 15Oct2020 20:53, Samuel Marks wrote: > >Idea: preprocess `sys.argv` so that this syntax would work > >`--optimizer Adam[learning_rate=0.01]`* > > > >*square rather than round so as not to require escape characters or > >quoting in `sh` > > Square brackets are also shell syntax, introducing glob character > ranges. You're only not noticing probably because an unmatched glob is > normally let through unchanged. > > For example: > >somecmd x[b] > > would go through as "x[b]" _unless_ you had a matching local filename > (in this case a file named "xb") in which case the shell would match the > glob and pass through the match (in this case "xb"). > > You'll find this issue a fair bit: if you want some punctuation, someone > else has probably already wanted that punctuation for something. You'll > still need to quote things for a lot of stuff. > > Commas are not special. You could try: > > --optimizer Adam,learning_rate=0.01,something_else=3 > > which would work for a fair amount of stuff. > > >Unfortunately this approach wouldn't give nice `--help` text. > >What's the right solution here? > > To get nice help text you need your command line parser to be able to > compose that help text in some way. Usually the help text in argparse is > supplied when you define the option. > > Cheers, > Cameron Simpson > -- Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn < https://linkedin.com/in/samuelmarks> -- https://mail.python.org/mailman/listinfo/python-list
Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?
Hi Dan, The feature that existing CLI parsers are missing is a clean syntax for specifying options on the second parameter (the "value"), where there may be different options available depending on which you choose. For example: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam has `learning_rate`, `beta_1`, `beta_2`, `epsilon`, and `amsgrad`* Whereas https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/RMSprop has `learning_rate`, `rho`, `momentum`, `epsilon`, `centered`* *with clipnorm, clipvalue, decay hidden behind kwargs So the question is how to expose this as CLI options. `--optimizer Adam` is a good first step, but it should error if I try and give it `momentum`. The comma syntax is my favourite so far. I guess I'll just have to write a validator outside the CLI parser to handle this… Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn <https://linkedin.com/in/samuelmarks> On Fri, Oct 16, 2020 at 11:19 AM <2qdxy4rzwzuui...@potatochowder.com> wrote: > > On 2020-10-16 at 10:20:40 +1100, > Cameron Simpson wrote: > > > On 16Oct2020 10:09, Samuel Marks wrote: > > >Yes it’s my module, and I’ve been using argparse > > >https://github.com/SamuelMarks/ml-params > > > > > >No library I’ve found provides a solution to CLI argument parsing for my > > >use-case. > > Out of curiosity, what do your command line interfaces require that > argparse and other libraries don't have? > > Yes, if my only tool is a hammer, then every problem looks like a nail, > but I've yet to have a requirement that the POSIX rules don't cover. > > > >So I’ll write one ... > > Been there. Done that. :-) > > > > [...] But what should it look like, syntactically and semantically? > > [...] > > > In particular, I would not invent yet another command line syntax. > > I agree. The POSIX Utility Syntax and Guidelines: > > https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html > > (I believe argparse supports a superset of the POSIX syntax.) > > I haven't found a definitive definition of GNU long options, only > examples. > > HTH, > Dan > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?
Yeah I've played with custom actions before https://github.com/offscale/argparse-utils/tree/master/argparse_utils/actions But this would only help in one phase, the important phase of providing help text will need to be provided out-of-argparse and thrown in (like my trivial absl alternative, exposing a function which takes an argparse instance and returns an argparse instance) The only hack remaining is that I have to pass through `sys.argv` at least once before giving it to argparse. I wonder if there's a way to not need to explicitly go through it at all… https://github.com/SamuelMarks/ml-params/blob/d1fb184/ml_params/__main__.py#L89 [I didn't know `getopt` was exposed otherwise I'd use that , but there has to be a solution just using argparse?] Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn <https://linkedin.com/in/samuelmarks> Samuel Marks Charity | consultancy | open-source | LinkedIn On Fri, Oct 16, 2020 at 3:47 PM Dieter Maurer wrote: > > Samuel Marks wrote at 2020-10-16 10:09 +1100: > >Yes it’s my module, and I’ve been using argparse > >https://github.com/SamuelMarks/ml-params > > > >No library I’ve found provides a solution to CLI argument parsing for my > >use-case. > > Do you know that with `argparse` you can specify how many arguments an option > expects? Thus, it should be quite easily possible to > have --opt ... > Do you know that you can define new `Action`s for `argparse`? > This way, you could properly process `--opt ,, ...`. -- https://mail.python.org/mailman/listinfo/python-list
Re: CLI parsing—with `--help` text—`--foo bar`, how to give additional parameters to `bar`?
Yeah, but the risk with config files is you need a website—and/or full JSON schema output—to figure out what’s needed. (Although I should mention that with my doctrans project you can generate a config class—and class method—from/to your argparse parser; enabling the config file scenario rather cleanly) Also there is a project which takes your CLI and turns it into a GUI (Gooey). At some future point doctrans will be “complete”, and then you can provide inputs via: - CLI arguments - Config file - RPC* - REST* *TODO, will also optionally generate ORM classes for persistence On Fri, 16 Oct 2020 at 6:08 pm, Karen Shaeffer wrote: > Passing a lot of command line parameters is very error prone. Opening a > file and specifying flags is much more user friendly, because you have any > necessary help documented right there. In my eyes, the command line is only > useful for simple scripts. > > Karen > > > On Oct 15, 2020, at 11:44 PM, Karen Shaeffer > wrote: > > > > In my work, for complex ML modeling code, I never pass anything through > the command line. I implement absl.flags in each module, specifying all the > defaults. At runtime I instantiate the flags as class variables of a > ModelConfiguration class that gets passed into the model class. Absl > supports all I need for this scheme. Of course, for a package, I can have a > configuration module for all these configuration classes. > > > > This scheme is easy to maintain. It can be dynamically customized at run > time. It’s very easy to document and maintain and to use. In one project, I > even supported console inputs that dynamically managed configurations of > pipelines. Nothing was ever passed in on command lines. I wonder why you > need to play on the command line. > > > > And yea, I’ve written a command line parsing function in C a long time > ago. I thought that really cool at the time. I wouldn’t want to do it now. > > > > Karen. > > > >> On Oct 15, 2020, at 9:58 PM, Samuel Marks > wrote: > >> > >> Yeah I've played with custom actions before > >> > https://github.com/offscale/argparse-utils/tree/master/argparse_utils/actions > >> > >> But this would only help in one phase, the important phase of > >> providing help text will need to be provided out-of-argparse and > >> thrown in > >> > >> (like my trivial absl alternative, exposing a function which takes an > >> argparse instance and returns an argparse instance) > >> > >> The only hack remaining is that I have to pass through `sys.argv` at > >> least once before giving it to argparse. I wonder if there's a way to > >> not need to explicitly go through it at all… > >> > https://github.com/SamuelMarks/ml-params/blob/d1fb184/ml_params/__main__.py#L89 > >> > >> [I didn't know `getopt` was exposed otherwise I'd use that >> sometimes do in C>, but there has to be a solution just using > >> argparse?] > >> > >> Samuel Marks > >> Charity <https://sydneyscientific.org> | consultancy > >> <https://offscale.io> | open-source <https://github.com/offscale> | > >> LinkedIn <https://linkedin.com/in/samuelmarks> > >> > >> > >> Samuel Marks > >> Charity | consultancy | open-source | LinkedIn > >> > >> > >> On Fri, Oct 16, 2020 at 3:47 PM Dieter Maurer > wrote: > >>> > >>> Samuel Marks wrote at 2020-10-16 10:09 +1100: > >>>> Yes it’s my module, and I’ve been using argparse > >>>> https://github.com/SamuelMarks/ml-params > >>>> > >>>> No library I’ve found provides a solution to CLI argument parsing for > my > >>>> use-case. > >>> > >>> Do you know that with `argparse` you can specify how many arguments an > option > >>> expects? Thus, it should be quite easily possible to > >>> have --opt ... > >>> Do you know that you can define new `Action`s for `argparse`? > >>> This way, you could properly process `--opt ,, ...`. > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > > > > -- Samuel Marks Charity <https://sydneyscientific.org> | consultancy <https://offscale.io> | open-source <https://github.com/offscale> | LinkedIn < https://linkedin.com/in/samuelmarks> -- https://mail.python.org/mailman/listinfo/python-list
Static typing—with annotations—function name & arguments, or result of call, or string
From my understanding, `ast.arguments` and `inspect.Signature` are the
two builtin ways of defining the function name and arguments in a
structured way.
What I am creating is a way of configuring… well let's be specific to
my use-case. I am building a way to configure TensorFlow.
One which is very type-driven, and will error as early-as-possible
when incorrect types or lack of required parameters are provided.
I can dynamically infer things like so:
```
import inspect
import tensorflow as tf
sig = inspect.signature(tf.keras.optimizers.Adam)
tuple({
"default": sig.parameters[param].default,
"name": param,
"typ": type(sig.parameters[param].default).__name__
if sig.parameters[param].default is not inspect._empty
and sig.parameters[param].annotation is inspect._empty
else sig.parameters[param].annotation,
}
for param in sig.parameters if param != 'name'
)
```
I can also parse the docstring, as I do in my doctrans library and tool.
Which will give me the options I can provide the class. So there's an
obvious solution, to generate these classes:
```
class TensorFlowConfig(object):
optimizer: Optional[Optimizer] = None
# Or maybe a `Protocol`?
class Optimizer(object): pass
class AdamConfig(Optimizer):
learning_rate: float = 0.001
beta_1: float = 0.9
beta_2: float = 0.999
epsilon: float = 1e-07
amsgrad: bool = False
kwargs: dict = {}
TensorFlowConfig().optimizer = AdamConfig()
```
But, keeping in mind the goal to expose everything in all interfaces,
the question then becomes how to generate an argparse parser from
this. And how to generate a function from this. And how to ensure that
whatever code-completion interface one uses in Python, that it can
figure out what the correct parameters are.
So I should get an error about incorrect type when I:
```
# CLI
--optimizer 'Adam' 'learning_rate = True'
# class*
TensorFlowConfig().optimizer = tf.keras.optimizers.Adam(learning_rate=True)
TensorFlowConfig().optimizer = AdamConfig(learning_rate=True)
# Function*
MyTensorFlowClass.train(optimizer=tf.keras.optimizers.Adam(learning_rate=True))
MyTensorFlowClass.train(optimizer=AdamConfig(learning_rate=True))
* both of these next two lines—after the heading—should be equivalent
```
Syntax like this should also be valid
```
TensorFlowConfig().optimizer = 'tf.keras.optimizers.Adam'
TensorFlowConfig().optimizer = 'Adam'
MyTensorFlowClass.train(optimizer='tf.keras.optimizers.Adam')
MyTensorFlowClass.train(optimizer='Adam')
```
Do I have huge annotations like this*, with an `Any` lopped on for
further analysis down the line? - So it can easily generate into
`choices` for argparse? [which will need to be subclassed in order to
enable that "--optimizer 'Adam' 'learning_rate = True'" syntax]
*
https://github.com/SamuelMarks/ml-params-tensorflow/blob/1d48502/ml_params_tensorflow/ml_params/trainer.py#L213-L215
What makes sense?
Thanks for your insights,
Samuel Marks
Charity <https://sydneyscientific.org> | consultancy
<https://offscale.io> | open-source <https://github.com/offscale> |
LinkedIn <https://linkedin.com/in/samuelmarks>
PS: This is the `doctrans` project I referenced
https://github.com/SamuelMarks/doctrans
--
https://mail.python.org/mailman/listinfo/python-list
Dependencies from git aren't being installed? (setuptools, pip)
I'm sure I've just misconfigured my setup.py, requirements.txt and/or project structure, but I've tried: * install_requires * requirements.txt * install_requires + dependency_links With (for install_requires): * bettertutors_sql_models==dev * bettertutors_sql_models For requirements.txt, dependency_links: * git+https://github.com/bettertutors/sql-models#egg=bettertutors_sql_models * git+https://github.com/bettertutors/sql-models#egg=bettertutors_sql_models * https://github.com/bettertutors/sql-models/archive/master.zip#egg=bettertutors_sql_models * https://github.com/bettertutors/sql-models/archive/master.zip * https://github.com/bettertutors/sql-models * https://github.com/bettertutors/sql-models/tarball/master#egg=sql-models Also asked here: http://stackoverflow.com/q/28540839 At one point I even tried writing my own install_deps function() to be run before setup(): https://gist.github.com/SamuelMarks/45a998a83dd60ddbadbc But nothing--apart from `cd`ing to my sql-models dir and running `pip install .`--is working for me. I'm using Python 2.7.*, and have experienced identical issues on Windows and Linux (with/without virtualenv). How am I meant to write in this dependency? -- https://mail.python.org/mailman/listinfo/python-list
