On 2017-02-24 18:54, kar6...@gmail.com wrote:
I have a task to search for multiple patterns in incoming string and replace
with matched patterns, I'm storing all pattern as keys in dict and replacements
as values, I'm using regex for compiling all the pattern and using the sub
method on pattern object for replacement. But the problem I have a tens of
millions of rows, that I need to check for pattern which is about 1000 and this
is turns out to be a very expensive operation.
What can be done to optimize it. Also I have special characters for matching,
where can I specify raw string combinations.
for example is the search string is not a variable we can say
re.search(r"\$%^search_text", "replace_text", "some_text") but when I read from the dict where shd
I place the "r" keyword, unfortunately putting inside key doesnt work "r key" like this....
Pseudo code
for string in genobj_of_million_strings:
pattern = re.compile('|'.join(regex_map.keys()))
return pattern.sub(lambda x: regex_map[x], string)
Here's an example:
import re
# A dict of the replacements.
mapping = {'one': 'unu', 'two': 'du', 'three': 'tri', 'four': 'kvar',
'five': 'kvin'}
# The text that we're searching.
text = 'one two three four five six seven eight nine ten'
# It's best to put the strings we're looking for into reverse order in
# case one of the keys is the prefix of another.
ordered_keys = sorted(mapping.keys(), reverse=True)
ordered_values = [mapping[key] for key in ordered_keys]
# Build the pattern, putting each key in its own group.
# I'm assuming that the keys are all pure literals, that they don't
# contain anything that's treated specially by regex. You could escape
# the key (using re.escape(...)) if that's not the case.
pattern = re.compile('|'.join('(%s)' % key for key in ordered_keys))
# When we find a match, the match object's .lastindex attribute will
# tell us which group (i.e. key) matched. We can then look up the
# replacement. We also need to take into account that the groups are
# numbered from 1, whereas list items are numbered from 0.
new_text = pattern.sub(lambda m: ordered_values[m.lastindex - 1], text)
It might be faster (timing it would be a good idea) if you could put all
of the rows into a single string (or a number of rows into a single
srting), process that string, and then split up the result. If none of
the rows contain '\n', then you could join them together with that,
otherwise just pick some other character.
--
https://mail.python.org/mailman/listinfo/python-list
