On May 8, 12:07 am, MRAB wrote:
> def compound_filter(token_stream):
> stream = lowercase_token(token_stream)
> stream = remove_boring(stream)
> stream = remove_dupes(stream)
> for t in stream(t):
> yield t
The last loop is superfluous. You can just do::
def compoun
MRAB wrote:
James wrote:
Hello all,
I'm working on some NLP code - what I'm doing is passing a large
number of tokens through a number of filtering / processing steps.
The filters take a token as input, and may or may not yield a token as
a result. For example, I might have filters which lowerc
James wrote:
Hello all,
I'm working on some NLP code - what I'm doing is passing a large
number of tokens through a number of filtering / processing steps.
The filters take a token as input, and may or may not yield a token as
a result. For example, I might have filters which lowercases the
inpu
James wrote:
Hello all,
I'm working on some NLP code - what I'm doing is passing a large
number of tokens through a number of filtering / processing steps.
The filters take a token as input, and may or may not yield a token as
a result. For example, I might have filters which lowercases the
inpu
James writes:
> Hello all,
> I'm working on some NLP code - what I'm doing is passing a large
> number of tokens through a number of filtering / processing steps.
>
> The filters take a token as input, and may or may not yield a token as
> a result. For example, I might have filters which lowerca
Hello all,
I'm working on some NLP code - what I'm doing is passing a large
number of tokens through a number of filtering / processing steps.
The filters take a token as input, and may or may not yield a token as
a result. For example, I might have filters which lowercases the
input, filter out b
