Thank you so much! Everything seems to be working now, so I just have to
tweak my.import and catch up on recording in my ledger.
Once I'm caught up, I'll come back to automating the download and extract
workflow.
Regarding the encoding issue, my first guess was it's because I'm doing all
this on Windows and using the Command Prompt shell, instead of bash. That
theory appeared to be incorrect after creating some test files from the
shell, which resulted in utf-8 encoding.
Next, I tried creating test files using the integrated terminal in VS Code
since technically that's where I've been running my commands, and it turns
out that was the culprit. ¯\_(ツ)_/¯
On Friday, September 20, 2024 at 9:42:32 PM UTC-5 Red S wrote:
> Thanks for the file! I can’t tell whether it was an artifact of how the
> ofx file you attached got created or whether the original ofx file suffers
> from this problem, but your ofx file is encoded in UTF-16 even though the
> header claims it’s UTF-8. ofxparse fails because of this. Running this
> allows it to work normally:
> iconv -f UTF-16 -t UTF-8 fidelity_X12345678.ofx -o a.ofx ofx-summarize
> a.ofx Total number of accounts: 1
> ----------------
> Account info: X12345678 fidelity.com
> Statement info: 2024-07-01 00:00:00 -- 2024-07-31 00:00:00. Bal: []
> Types: {'cash', 'check', 'dep', 'payment'}
>
> 2024-07-30 04:00:00 cash THIS IS FUN Note -15.0000
> 2024-07-30 04:00:00 dep THIS IS FUN Note -15.0000
> 2024-07-29 04:00:00 check THIS IS FUN Note -15.0000
> 2024-07-26 04:00:00 cash THIS IS FUN Note -15.0000
> 2024-07-26 04:00:00 cash THIS IS FUN Note -15.0000
>
> It's not the same stack trace as you got originally though. Either way,
> try this?
>
> On Friday, September 20, 2024 at 6:38:29 PM UTC-7 Joshua Cabrera wrote:
>
>> Thanks, Red. The file itself looks okay after another look.
>>
>> I tried the new line trick, but it didn't change the results.
>>
>> The simple parse threw the same unpacking value error on my ofx while
>> working on the sample ofx.
>>
>> I'm sharing a sanitized version of my ofx (replaced account number,
>> transaction details, etc.) and my import file.
>>
>> In the meantime, I'll try a different Fidelity account and smaller
>> date/transaction ranges.
>> On Friday, September 20, 2024 at 5:09:18 PM UTC-5 Red S wrote:
>>
>>> This is uncommon, but typically because there was an issue downloading
>>> the ofx. Usually because of authentication problems. Have you examined the
>>> ofx to see if it’s valid?
>>>
>>> Try this to make it more readable: sed 's/>/>\n/g' xyz.ofx
>>>
>>>
>>> You should see something like this on the top:
>>> <STATUS> <CODE> 0 </CODE> <SEVERITY> INFO </SEVERITY> <MESSAGE> Success
>>> </MESSAGE> </STATUS>
>>>
>>>
>>> If you see that, then try a really simple parse:
>>> #!/usr/bin/env python3 import ofxparse ofx_file_path =
>>> "your_ofx_file.ofx" with open(ofx_file_path, 'r') as ofx_file: ofx =
>>> ofxparse.OfxParser.parse(ofx_file) print("Account type:",
>>> ofx.account.account_type) print("Account ID:", ofx.account.account_id)
>>>
>>>
>>> On Friday, September 20, 2024 at 2:54:15 PM UTC-7 Joshua Cabrera wrote:
>>>
>>>> So, I've got my .ofx file and am not worried about the download task
>>>> anymore. That's the good news.
>>>>
>>>> The bad news is now I'm stuck on the import task.
>>>> bean-extract my.import fidelity_1234_july2024.ofx
>>>>
>>>>
>>>> Results
>>>> ERROR:root:Importer
>>>> beancount_reds_importers.importers.fidelity.Importer.identify() raised an
>>>> unexpected error: not enough values to unpack (expected 2, got 1)
>>>> Traceback (most recent call last):
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py"
>>>> , line 63, in find_imports
>>>> matched = importer.identify(file)
>>>> ^^^^^^^^^^^^^^^^^^^^^^^
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py"
>>>> , line 26, in identify
>>>> self.initialize_reader(file)
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
>>>> , line 26, in initialize_reader
>>>> self.ofx = self.read_file(file)
>>>> ^^^^^^^^^^^^^^^^^^^^
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
>>>> , line 57, in read_file
>>>> return ofxparse.OfxParser.parse(fh)
>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>>> , line 396, in parse
>>>> ofx_file = OfxPreprocessedFile(file_handle)
>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>>> , line 155, in __init__
>>>> super(OfxPreprocessedFile, self).__init__(fh)
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>>> , line 82, in __init__
>>>> self.read_headers()
>>>> File
>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>>> , line 95, in read_headers
>>>> header, value = line.split(six.b(":"))
>>>> ^^^^^^^^^^^^^
>>>> ValueError: not enough values to unpack (expected 2, got 1)
>>>> ;; -*- mode: beancount -*-
>>>>
>>>>
>>>> If I run ofx-summarize, it throws the same error, so I found a sample
>>>> .ofx online and tested ofx-summarize with it.
>>>>
>>>> And of course, the example works, so it looks like Fidelity is
>>>> providing an incorrectly formatted file? I noticed it doesn't have a
>>>> single
>>>> line break until the end of the file.
>>>>
>>>> On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote:
>>>>
>>>>> Glad you got it working with ofxget. bean-download is just a wrapper
>>>>> around ofxget or whatever command you have to parallelize all your
>>>>> downloads. If you don’t have several accounts you’re using it with, it’s
>>>>> of
>>>>> limited use. The built in template for fidelity is here, and lets you
>>>>> store your password securely in pass so it’s not stored in plaintext.
>>>>> You could simply paste your working ofxget command into the
>>>>> bean-download config file to get it to work if the parallelization is
>>>>> useful for you. Either way, that’s the last thing I’d do after I got
>>>>> everything else in the import workflow working first.
>>>>>
>>>>> And oh wow, I’d forgotten that Fidelity stopped
>>>>> <https://www.reddit.com/r/fidelityinvestments/comments/17tpgfr/why_doesnt_fidelity_support_ofx_downloads_anymore/>
>>>>> letting
>>>>> you download ofx via their website, and only let you do it via direct
>>>>> download.
>>>>>
>>>>>
>>>>> On Monday, September 16, 2024 at 7:31:34 PM UTC-7 Joshua Cabrera wrote:
>>>>>
>>>>>> Quick update:
>>>>>>
>>>>>> Installed ofxget and manage to get my ofxget.cfg updated using ofxget
>>>>>> acctinfo. Not sure if this required for bean-download.
>>>>>>
>>>>>> Also downloaded a .ofx file using:
>>>>>> ofxget stmt fidelity > c:\users\joshd\fidelity.ofx
>>>>>>
>>>>>> Struggling to my download.cfg and bean-download download working -
>>>>>> keep getting:
>>>>>> 1/1 sites failed: fidelity
>>>>>>
>>>>>> Will keep trying whenever I have time this week.
>>>>>> On Sunday, September 15, 2024 at 9:24:33 PM UTC-5 Red S wrote:
>>>>>>
>>>>>>> My reason for using .csv over .ofx is simply familiarity and ease of
>>>>>>> starting up.
>>>>>>>
>>>>>>> I have no experience with ofx and thought the learning curve would
>>>>>>> be much steeper, and I already feel like I'm drowning a bit as a
>>>>>>> layperson.
>>>>>>>
>>>>>>> I know it may seem like that, especially given ofx is not a human
>>>>>>> readable format. But might I suggest giving that a shot first, and you
>>>>>>> might be pleasantly surprised. Because there is nothing to do, no
>>>>>>> importer
>>>>>>> to write, since it is a machine format with an official spec, unlike
>>>>>>> csv.
>>>>>>> The learning curve should be close to zero. There are other benefits
>>>>>>> <https://reds-rants.netlify.app/personal-finance/a-word-about-input-formats-use-ofx-when-you-can/>
>>>>>>>
>>>>>>> too.
>>>>>>>
>>>>>>> I'll look into the csvreader changes you mentioned to try making a
>>>>>>> replacement for prepare_raw_columns() and also look further into ofx.
>>>>>>>
>>>>>>> If I were you, I’d definitely spend five minutes with ofx first. And
>>>>>>> reds-importers ships with ofx-summarize which you can use to peek
>>>>>>> inside and poke around an ofx file, which makes it a lot less opaque
>>>>>>> and
>>>>>>> easy on a layperson.
>>>>>>>
>>>>>>> Thanks for the help here and confirming I'm doing alright, and also
>>>>>>> thank you for maintaining it!
>>>>>>>
>>>>>>> Happy to, and glad its helpful!
>>>>>>>
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to beancount+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/beancount/b317aece-d9be-4d46-b39a-24b4584c3506n%40googlegroups.com.
