Hi Martin, responses to your response below.
On Monday, October 24, 2016 at 10:07:53 PM UTC-4, Martin Blais wrote: > > On Sun, Oct 23, 2016 at 9:01 PM, <[email protected] <javascript:>> wrote: > >> Hi Martin, >> >> Just started using beancount to get away from Quicken. Your reasons for >> writing it and the documentation was my reason to give it a try. It should >> also help my Python skills. You sort of have an example in csv_test.py. >> It helped me determine what to put in my .import file. Strangely, csv.py >> and csv_test.py were not in my install, ofx.py and others were. I just >> copied from >> https://bitbucket.org/blais/beancount/src/b73da2b4f8e3182b1fb537a75385406c6d4fee90/src/python/beancount/ingest/importers/csv.py?at=booking&fileviewer=file-view-default >> >> and moved to the directory where ofx.py was and it worked, sort of. >> > > Strange. I just tested this now and the csv.py module does appear to > install. > Can you try running something like this? > > mkdir -p /tmp/b/lib/python3.5/site-packages > > PYTHONPATH=/tmp/b/lib/python3.5/site-packages:$PYTHONPATH python3 > setup.py install --prefix=/tmp/b > > ls -l > /tmp/b/lib/python3.5/site-packages/beancount-2.0b12-py3.5-macosx-10.11-x86_64.egg/beancount/ingest/importers > and share the output? > > I tried the above, but it didn't work. I broke up the second line, but still no go. Output below: Enter code here...JFSMini1:Beancount$ ll /tmp lrwxr-xr-x@ 1 root wheel 11B Aug 12 22:07 /tmp@ -> private/tmp JFSMini1:Beancount$ mkdir -p /tmp/b/lib/python3.5/site-packages JFSMini1:Beancount$ PYTHONPATH=/tmp/b/lib/python3.5/site-packages:$PYTHONPATH python3 setup.py install --prefix=/tmp/b /Library/Frameworks/Python.framework/Versions/3.5/Resources/Python.app/Contents/MacOS/Python: can't open file 'setup.py': [Errno 2] No such file or directory JFSMini1:Beancount$ echo $PYTHONPATH JFSMini1:Beancount$ PYTHONPATH=/tmp/b/lib/python3.5/site-packages JFSMini1:Beancount$ echo $PYTHONPATH /tmp/b/lib/python3.5/site-packages JFSMini1:Beancount$ echo $PYTHONPATH python3 setup.py install --prefix=/tmp/b /tmp/b/lib/python3.5/site-packages python3 setup.py install --prefix=/tmp/b JFSMini1:Beancount$ ls /tmp/b/lib/python3.5/site-packages/beancount ls: /tmp/b/lib/python3.5/site-packages/beancount: No such file or directory JFSMini1:Beancount$ ls /tmp/b/lib/python3.5/site-packages/ JFSMini1:Beancount$ I forgot to state this was installed on my MAC Mini running OS X 10.11.6, El Capitan. I then installed on my Macbook Pro running OS X 10.10.5, Yosemite. csv.py and csv_test.py were there. It must have been something with that install. > > I ran bean-identify, then the bean-extract as in the example, but I used a >> file, not a directory. However, the only thing in the output file was the >> header and the path/filename. No transactions made it into the file. >> Maybe the regexp input? >> > > Does bean-identify detect and associate your input file with the importer > you created below? > (If it does, it would clearly print out a few meaningful lines about that.) > I suspect your regexp may not match the file. > > Since bean-identify does print out **** /Users/jonathan/Documents/Beancount/20161018CapOne.csv I think it does associate it, but I must be missing something. > > >> I sort of guessed on that, I couldn't exactly see what to put in there >> except the headings. Here is the import file I created. >> >> #!/usr/bin/env python3 >> """Example import configuration.""" >> >> # Insert our custom importers path here. >> # (In practice you might just change your PYTHONPATH environment.) >> import sys >> from os import path >> sys.path.insert(0, path.join(path.dirname(__file__))) >> >> from beancount.ingest import extract >> from beancount.ingest.importers import ofx >> from beancount.ingest.importers import csv >> >> Col = csv.Col >> >> # Setting this variable provides a list of importer instances. >> CONFIG = [ >> csv.Importer({Col.DATE: 'Posted Date', >> Col.TXN_DATE: 'Transaction Date', >> Col.NARRATION1: 'Card No.', >> Col.NARRATION2: 'Description', >> Col.NARRATION3: 'Catagory', >> > > There's a typo here: Catagory -> Category > (I don't think it's what caused the problem though, just pointing it out.) > No, that wasn't the problem, but thanks for finding that. > > >> Col.AMOUNT_DEBIT: 'Debit', >> Col.AMOUNT_CREDIT: 'Credit'}, >> 'Liabilities:US:CapOne', >> 'USD', >> ('Transaction Date,Posted Date,Card No.,Description,' >> 'Category,Debit,Credit'), >> 'CapOne') >> ] >> >> >> # Override the header on extracted text (if desired). >> extract.HEADER = ';; -*- mode: org; mode: beancount; coding: utf-8; -*-\n' >> >> >> Am I correct in thinking I would need a separate import file for each >> different type of csv? >> > > I'm not sure what you mean by "a separate import file." > I think we are using different terminology, and I missed the 'dot' in front of the import. I call the example.import the import file, I think you are calling them the config file. > > - The source code for all the importers you create like this could very > well be in a one Python file, no problem with that. > > - You will need to instantiate a separate importer for each different type > of download, e.g. one for each source. > > BTW, here's an example configuration that ought to work on a US CapitalOne > CSV download: > > from beancount.ingest import regression > from beancount.ingest.importers import csv > > class Importer(csv.Importer): > > config = {csv.Col.DATE: 'Posted Date', > csv.Col.TXN_DATE: 'Transaction Date', > csv.Col.AMOUNT_DEBIT: 'Debit', > csv.Col.AMOUNT_CREDIT: 'Credit', > csv.Col.PAYEE: 'Description', > csv.Col.NARRATION: 'Category'} > > def __init__(self, account): > csv.Importer.__init__( > self, self.config, > account, 'USD', > ('Stage, Transaction Date, Posted Date, Card No., Description, > Category, ' > 'Debit, Credit'), > 'capitalone') > > def get_description(self, row): > payee, narration = super().get_description() > narration = '{} ({})'.format(narration, row.category) > return payee, narration > > You can place this in another .py file and import that from your importer > configuration source. > > I put the above code in a file called capitalone.import, and incurred the following error: JFSMini1:Beancount$ bean-identify capitalone.import 20161018CapOne.csv Traceback (most recent call last): File "/usr/local/bin/bean-identify", line 2, in <module> from beancount.ingest.identify import main; main() File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/beancount/ingest/identify.py" , line 92, in main _, config, downloads_directories = scripts_utils.parse_arguments(parser) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/beancount/ingest/scripts_utils.py" , line 56, in parse_arguments config = mod['CONFIG'] KeyError: 'CONFIG' Should I have put the code in a capitaloneimport.py and then made a separate capone.import? I have been looking at your code and reading some docs, but your Python coding is a lot more sophisticated than I am used to. It is a good learning experience though. > > > >> >> Thanks >> >> Jonathan >> >> On Friday, October 21, 2016 at 11:20:55 PM UTC-4, Martin Blais wrote: >>> >>> On Wed, Oct 19, 2016 at 4:19 AM, Saša Janiška <[email protected]> >>> wrote: >>> >>>> Hello, >>>> >>>> afaik, (h)ledger have some general CSV importer where one can define >>>> some mapping rules for the desired CSV format and then import? >>>> >>>> CSV is also the only format I can use since the bank just provides Excel >>>> file which can be converted to CSV. >>>> >>> >>> Beancount offers a framework to define your own custom importers, >>> documented here: >>> http://furius.ca/beancount/doc/ingest >>> >>> Unfortunately, you have to write some code to set that up. >>> >>> On the other hand, it also comes with a good example CSV importer which >>> is functional: >>> >>> https://bitbucket.org/blais/beancount/src/b73da2b4f8e3182b1fb537a75385406c6d4fee90/src/python/beancount/ingest/importers/csv.py?at=booking&fileviewer=file-view-default >>> >>> You just need to instantiate it and provide it with a dict that tells it >>> how to interpret each column of your CSV file. >>> (I should provide some example on how to do that, in the meantime you'd >>> have to read the source code.) >>> >>> HTH, >>> >>> >>> >>> >>> Sincerely, >>>> Gour >>>> >>>> -- >>>> While contemplating the objects of the senses, a person >>>> develops attachment for them, and from such attachment lust >>>> develops, and from lust anger arises. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Beancount" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/beancount/874m48rbsv.fsf%40atmarama.com >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Beancount" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/beancount/89102a4b-26aa-4c5e-82c6-0400d6268c38%40googlegroups.com >> >> <https://groups.google.com/d/msgid/beancount/89102a4b-26aa-4c5e-82c6-0400d6268c38%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > On Monday, October 24, 2016 at 10:07:53 PM UTC-4, Martin Blais wrote: > > On Sun, Oct 23, 2016 at 9:01 PM, <[email protected] <javascript:>> wrote: > >> Hi Martin, >> >> Just started using beancount to get away from Quicken. Your reasons for >> writing it and the documentation was my reason to give it a try. It should >> also help my Python skills. You sort of have an example in csv_test.py. >> It helped me determine what to put in my .import file. Strangely, csv.py >> and csv_test.py were not in my install, ofx.py and others were. I just >> copied from >> https://bitbucket.org/blais/beancount/src/b73da2b4f8e3182b1fb537a75385406c6d4fee90/src/python/beancount/ingest/importers/csv.py?at=booking&fileviewer=file-view-default >> >> and moved to the directory where ofx.py was and it worked, sort of. >> > > Strange. I just tested this now and the csv.py module does appear to > install. > Can you try running something like this? > > mkdir -p /tmp/b/lib/python3.5/site-packages > > PYTHONPATH=/tmp/b/lib/python3.5/site-packages:$PYTHONPATH python3 > setup.py install --prefix=/tmp/b > > ls -l > /tmp/b/lib/python3.5/site-packages/beancount-2.0b12-py3.5-macosx-10.11-x86_64.egg/beancount/ingest/importers > and share the output? > > > I ran bean-identify, then the bean-extract as in the example, but I used a >> file, not a directory. However, the only thing in the output file was the >> header and the path/filename. No transactions made it into the file. >> Maybe the regexp input? >> > > Does bean-identify detect and associate your input file with the importer > you created below? > (If it does, it would clearly print out a few meaningful lines about that.) > I suspect your regexp may not match the file. > > > >> I sort of guessed on that, I couldn't exactly see what to put in there >> except the headings. Here is the import file I created. >> >> #!/usr/bin/env python3 >> """Example import configuration.""" >> >> # Insert our custom importers path here. >> # (In practice you might just change your PYTHONPATH environment.) >> import sys >> from os import path >> sys.path.insert(0, path.join(path.dirname(__file__))) >> >> from beancount.ingest import extract >> from beancount.ingest.importers import ofx >> from beancount.ingest.importers import csv >> >> Col = csv.Col >> >> # Setting this variable provides a list of importer instances. >> CONFIG = [ >> csv.Importer({Col.DATE: 'Posted Date', >> Col.TXN_DATE: 'Transaction Date', >> Col.NARRATION1: 'Card No.', >> Col.NARRATION2: 'Description', >> Col.NARRATION3: 'Catagory', >> > > There's a typo here: Catagory -> Category > (I don't think it's what caused the problem though, just pointing it out.) > > >> Col.AMOUNT_DEBIT: 'Debit', >> Col.AMOUNT_CREDIT: 'Credit'}, >> 'Liabilities:US:CapOne', >> 'USD', >> ('Transaction Date,Posted Date,Card No.,Description,' >> 'Category,Debit,Credit'), >> 'CapOne') >> ] >> >> >> # Override the header on extracted text (if desired). >> extract.HEADER = ';; -*- mode: org; mode: beancount; coding: utf-8; -*-\n' >> >> >> Am I correct in thinking I would need a separate import file for each >> different type of csv? >> > > I'm not sure what you mean by "a separate import file." > > - The source code for all the importers you create like this could very > well be in a one Python file, no problem with that. > > - You will need to instantiate a separate importer for each different type > of download, e.g. one for each source. > > BTW, here's an example configuration that ought to work on a US CapitalOne > CSV download: > > from beancount.ingest import regression > from beancount.ingest.importers import csv > > class Importer(csv.Importer): > > config = {csv.Col.DATE: 'Posted Date', > csv.Col.TXN_DATE: 'Transaction Date', > csv.Col.AMOUNT_DEBIT: 'Debit', > csv.Col.AMOUNT_CREDIT: 'Credit', > csv.Col.PAYEE: 'Description', > csv.Col.NARRATION: 'Category'} > > def __init__(self, account): > csv.Importer.__init__( > self, self.config, > account, 'USD', > ('Stage, Transaction Date, Posted Date, Card No., Description, > Category, ' > 'Debit, Credit'), > 'capitalone') > > def get_description(self, row): > payee, narration = super().get_description() > narration = '{} ({})'.format(narration, row.category) > return payee, narration > > > You can place this in another .py file and import that from your importer > configuration source. > > > > >> >> Thanks >> >> Jonathan >> >> On Friday, October 21, 2016 at 11:20:55 PM UTC-4, Martin Blais wrote: >>> >>> On Wed, Oct 19, 2016 at 4:19 AM, Saša Janiška <[email protected]> >>> wrote: >>> >>>> Hello, >>>> >>>> afaik, (h)ledger have some general CSV importer where one can define >>>> some mapping rules for the desired CSV format and then import? >>>> >>>> CSV is also the only format I can use since the bank just provides Excel >>>> file which can be converted to CSV. >>>> >>> >>> Beancount offers a framework to define your own custom importers, >>> documented here: >>> http://furius.ca/beancount/doc/ingest >>> >>> Unfortunately, you have to write some code to set that up. >>> >>> On the other hand, it also comes with a good example CSV importer which >>> is functional: >>> >>> https://bitbucket.org/blais/beancount/src/b73da2b4f8e3182b1fb537a75385406c6d4fee90/src/python/beancount/ingest/importers/csv.py?at=booking&fileviewer=file-view-default >>> >>> You just need to instantiate it and provide it with a dict that tells it >>> how to interpret each column of your CSV file. >>> (I should provide some example on how to do that, in the meantime you'd >>> have to read the source code.) >>> >>> HTH, >>> >>> >>> >>> >>> Sincerely, >>>> Gour >>>> >>>> -- >>>> While contemplating the objects of the senses, a person >>>> develops attachment for them, and from such attachment lust >>>> develops, and from lust anger arises. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Beancount" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/beancount/874m48rbsv.fsf%40atmarama.com >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Beancount" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/beancount/89102a4b-26aa-4c5e-82c6-0400d6268c38%40googlegroups.com >> >> <https://groups.google.com/d/msgid/beancount/89102a4b-26aa-4c5e-82c6-0400d6268c38%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/3f76bc72-b536-4d90-8736-fd8a82e7cb6d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
