Re: Struggling to get my first importer working (using Red's Importers)

[Red S]

Thanks for the file! I can’t tell whether it was an artifact of how the ofx 
file you attached got created or whether the original ofx file suffers from 
this problem, but your ofx file is encoded in UTF-16 even though the header 
claims it’s UTF-8. ofxparse fails because of this. Running this allows it 
to work normally:
iconv -f UTF-16 -t UTF-8 fidelity_X12345678.ofx -o a.ofx ofx-summarize 
a.ofx Total number of accounts: 1

Account info:   X12345678 fidelity.com
Statement info: 2024-07-01 00:00:00 -- 2024-07-31 00:00:00. Bal: []
Types:  {'cash', 'check', 'dep', 'payment'}

2024-07-30 04:00:00 cash THIS IS FUN Note -15.
2024-07-30 04:00:00 dep THIS IS FUN Note -15.
2024-07-29 04:00:00 check THIS IS FUN Note -15.
2024-07-26 04:00:00 cash THIS IS FUN Note -15.
2024-07-26 04:00:00 cash THIS IS FUN Note -15.
​
It's not the same stack trace as you got originally though. Either way, try 
this?

On Friday, September 20, 2024 at 6:38:29 PM UTC-7 Joshua Cabrera wrote:

> Thanks, Red. The file itself looks okay after another look.
>
> I tried the new line trick, but it didn't change the results.
>
> The simple parse threw the same unpacking value error on my ofx while 
> working on the sample ofx.
>
> I'm sharing a sanitized version of my ofx (replaced account number, 
> transaction details, etc.) and my import file.
>
> In the meantime, I'll try a different Fidelity account and smaller 
> date/transaction ranges.
> On Friday, September 20, 2024 at 5:09:18 PM UTC-5 Red S wrote:
>
>> This is uncommon, but typically because there was an issue downloading 
>> the ofx. Usually because of authentication problems. Have you examined the 
>> ofx to see if it’s valid?
>>
>> Try this to make it more readable: sed 's/>/>\n/g' xyz.ofx
>> ​
>>
>> You should see something like this on the top:
>>   0   INFO   Success 
>>   
>> ​
>>
>> If you see that, then try a really simple parse:
>> #!/usr/bin/env python3 import ofxparse ofx_file_path = 
>> "your_ofx_file.ofx" with open(ofx_file_path, 'r') as ofx_file: ofx = 
>> ofxparse.OfxParser.parse(ofx_file) print("Account type:", 
>> ofx.account.account_type) print("Account ID:", ofx.account.account_id) 
>> ​
>>
>> On Friday, September 20, 2024 at 2:54:15 PM UTC-7 Joshua Cabrera wrote:
>>
>>> So, I've got my .ofx file and am not worried about the download task 
>>> anymore. That's the good news.
>>>
>>> The bad news is now I'm stuck on the import task.
>>> bean-extract my.import fidelity_1234_july2024.ofx
>>>
>>>
>>> Results
>>> ERROR:root:Importer 
>>> beancount_reds_importers.importers.fidelity.Importer.identify() raised an 
>>> unexpected error: not enough values to unpack (expected 2, got 1)
>>> Traceback (most recent call last):
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py"
>>> , line 63, in find_imports
>>> matched = importer.identify(file)
>>>   ^^^
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py"
>>> , line 26, in identify
>>> self.initialize_reader(file)
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
>>> , line 26, in initialize_reader
>>> self.ofx = self.read_file(file)
>>>
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
>>> , line 57, in read_file
>>> return ofxparse.OfxParser.parse(fh)
>>>
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>> , line 396, in parse
>>> ofx_file = OfxPreprocessedFile(file_handle)
>>>
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>> , line 155, in __init__
>>> super(OfxPreprocessedFile, self).__init__(fh)
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>> , line 82, in __init__
>>> self.read_headers()
>>>   File 
>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>>> , line 95, in read_headers
>>> header, value = line.split(six.b(":"))
>>> ^
>>> ValueError: not enough values to unpack (expected 2, got 1)
>>> ;; -*- mode: beancount -*-
>>>
>>>
>>> If I run ofx-summarize, it throws the same error, so I found a sample 
>>> .ofx online and tested ofx-summarize with it. 
>>>
>>> And of course, the example works, so it looks like Fidelity is providing 
>>> an incorrectly formatted file? I noticed it doesn't have a single line 
>>> break until the end of the file.
>>>
>>> On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote:
>>>

Beanquery - Select expenses tied to a specific asset

[Oscar & Alejandra]

Hello everyone,

I am using the latest beanquery from PyPI (v0.1.dev0) and I'm pretty new to SQL 
and beanquery's SQL-like syntax. I am trying to produce a report where I can 
see all of my expenses for a certain period, something like what my bank 
statement would show but grouped by my expense categories. I found this in the 
examples in the docs and it's pretty close to what I want but I'm not sure how 
to grab the 'transfers' from my asset account. For example when I make a 
payment towards my credit card or when I transfer into my savings.

FROM OPEN ON 2024-09-12 CLOSE ON 2024-09-26
SELECT account, SUM(position)
WHERE account~"Income|Expenses"
GROUP BY 1
ORDER BY 1

The above query produces something like the following:

---  
  account    SUM(position
Expenses:Auto:Fuel  50.00 USD
Expenses:Auto:Insurance    140.00 USD
Expenses:Clothes    20.00 USD
Expenses:Debt:Interest  83.00 USD
Expenses:Food:Dining   100.00 USD
Expenses:Food:Groceries    140.00 USD
Expenses:Home:Utilities:Electricity 80.00 USD
Expenses:Home:Utilities:Water   70.00 USD
Income:Salary    -3000.00 USD

Where I would really like something like this

---  
  account    SUM(position
Expenses:Auto:Fuel  50.00 USD
Expenses:Auto:Insurance    140.00 USD
Expenses:Clothes    20.00 USD
Expenses:Debt:Interest  83.00 USD
Expenses:Food:Dining   100.00 USD
Expenses:Food:Groceries    140.00 USD
Expenses:Home:Utilities:Electricity 80.00 USD
Expenses:Home:Utilities:Water   70.00 USD
Income:Salary    -3000.00 USD
Liabilities:CreditCard 100.00 USD
Assets:Savings 300.00 USD

Which would essentially show everything my 'paycheck' was spent on for the 
period selected. Is this possible using beanquery?

Thanks!

Oscar & Ale

m...@aleyoscar.com

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/ab4b949a2c34859d816728bb8a2c565143560715%40aleyoscar.com.

Re: Beanquery - Select expenses tied to a specific asset

[Chary Chary]

Hi,

I am not sure exactly what you want to achieve, but the following query 
will give you a sum of all changes to to all accounts (including Assets and 
Liabilities) over the period of time 

SELECT account, SUM(position)
WHERE date >=2024-09-12 and date <=2024-09-26

This will be all of your incomes, all expenses over the period, as well as 
changes to Assets and Liabilities over the same period.

Total will be 0.

On Friday, September 20, 2024 at 6:30:29 PM UTC+2 Oscar & Alejandra wrote:

> Hello everyone,
>
> I am using the latest beanquery from PyPI (v0.1.dev0) and I'm pretty new 
> to SQL and beanquery's SQL-like syntax. I am trying to produce a report 
> where I can see all of my expenses for a certain period, something like 
> what my bank statement would show but grouped by my expense categories. I 
> found this in the examples in the docs and it's pretty close to what I want 
> but I'm not sure how to grab the 'transfers' from my asset account. For 
> example when I make a payment towards my credit card or when I transfer 
> into my savings.
>
> FROM OPEN ON 2024-09-12 CLOSE ON 2024-09-26
> SELECT account, SUM(position)
> WHERE account~"Income|Expenses"
> GROUP BY 1
> ORDER BY 1
>
> The above query produces something like the following:
>
> ---  
>   accountSUM(position
> Expenses:Auto:Fuel  50.00 USD
> Expenses:Auto:Insurance140.00 USD
> Expenses:Clothes20.00 USD
> Expenses:Debt:Interest  83.00 USD
> Expenses:Food:Dining   100.00 USD
> Expenses:Food:Groceries140.00 USD
> Expenses:Home:Utilities:Electricity 80.00 USD
> Expenses:Home:Utilities:Water   70.00 USD
> Income:Salary-3000.00 USD
>
> Where I would really like something like this
>
> ---  
>   accountSUM(position
> Expenses:Auto:Fuel  50.00 USD
> Expenses:Auto:Insurance140.00 USD
> Expenses:Clothes20.00 USD
> Expenses:Debt:Interest  83.00 USD
> Expenses:Food:Dining   100.00 USD
> Expenses:Food:Groceries140.00 USD
> Expenses:Home:Utilities:Electricity 80.00 USD
> Expenses:Home:Utilities:Water   70.00 USD
> Income:Salary-3000.00 USD
> Liabilities:CreditCard 100.00 USD
> Assets:Savings 300.00 USD
>
> Which would essentially show everything my 'paycheck' was spent on for the 
> period selected. Is this possible using beanquery?
>
> Thanks!
>
>
> Oscar & Ale
>
> ma...@aleyoscar.com
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/32363a8a-d08e-4554-9941-b8ac89947a18n%40googlegroups.com.

Re: Beanquery - Select expenses tied to a specific asset

[Chary Chary]

P.S. changes to Equity will also be shown

On Friday, September 20, 2024 at 8:16:13 PM UTC+2 Chary Chary wrote:

> Hi,
>
> I am not sure exactly what you want to achieve, but the following query 
> will give you a sum of all changes to to all accounts (including Assets and 
> Liabilities) over the period of time 
>
> SELECT account, SUM(position)
> WHERE date >=2024-09-12 and date <=2024-09-26
>
> This will be all of your incomes, all expenses over the period, as well as 
> changes to Assets and Liabilities over the same period.
>
> Total will be 0.
>
> On Friday, September 20, 2024 at 6:30:29 PM UTC+2 Oscar & Alejandra wrote:
>
>> Hello everyone,
>>
>> I am using the latest beanquery from PyPI (v0.1.dev0) and I'm pretty new 
>> to SQL and beanquery's SQL-like syntax. I am trying to produce a report 
>> where I can see all of my expenses for a certain period, something like 
>> what my bank statement would show but grouped by my expense categories. I 
>> found this in the examples in the docs and it's pretty close to what I want 
>> but I'm not sure how to grab the 'transfers' from my asset account. For 
>> example when I make a payment towards my credit card or when I transfer 
>> into my savings.
>>
>> FROM OPEN ON 2024-09-12 CLOSE ON 2024-09-26
>> SELECT account, SUM(position)
>> WHERE account~"Income|Expenses"
>> GROUP BY 1
>> ORDER BY 1
>>
>> The above query produces something like the following:
>>
>> ---  
>>   accountSUM(position
>> Expenses:Auto:Fuel  50.00 USD
>> Expenses:Auto:Insurance140.00 USD
>> Expenses:Clothes20.00 USD
>> Expenses:Debt:Interest  83.00 USD
>> Expenses:Food:Dining   100.00 USD
>> Expenses:Food:Groceries140.00 USD
>> Expenses:Home:Utilities:Electricity 80.00 USD
>> Expenses:Home:Utilities:Water   70.00 USD
>> Income:Salary-3000.00 USD
>>
>> Where I would really like something like this
>>
>> ---  
>>   accountSUM(position
>> Expenses:Auto:Fuel  50.00 USD
>> Expenses:Auto:Insurance140.00 USD
>> Expenses:Clothes20.00 USD
>> Expenses:Debt:Interest  83.00 USD
>> Expenses:Food:Dining   100.00 USD
>> Expenses:Food:Groceries140.00 USD
>> Expenses:Home:Utilities:Electricity 80.00 USD
>> Expenses:Home:Utilities:Water   70.00 USD
>> Income:Salary-3000.00 USD
>> Liabilities:CreditCard 100.00 USD
>> Assets:Savings 300.00 USD
>>
>> Which would essentially show everything my 'paycheck' was spent on for 
>> the period selected. Is this possible using beanquery?
>>
>> Thanks!
>>
>>
>> Oscar & Ale
>>
>> ma...@aleyoscar.com
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/12877477-b999-45f6-aaba-042d4c101254n%40googlegroups.com.

Re: Beanquery - Select expenses tied to a specific asset

[Oscar & Alejandra]

Wow, so simple. That is pretty close to what I need, I'm just wondering if 
there is a way to filter out only transactions that have a posting with a 
specific account.

For example, take these transactions:

2024-01-01 * "Opening Balances"

    Liabilities:Credit  -1000.00 USD

    Equity:Opening   1000.00 USD



2024-09-01 * "Salary"

    Assets:Checking  3000.00 USD

    Income:Salary   -3000.00 USD



2024-09-02 * "Interest"

    Expenses:Interest  10.00 USD

    Liabilities:Credit    -10.00 USD



2024-09-03 * "Credit payment"

    Liabilities:Credit    100.00 USD

    Assets:Checking  -100.00 USD



2024-09-04 * "Groceries"

    Expenses:Food         200.00 USD

    Assets:Checking  -200.00 USD

Running that query will list all the accounts with their balances for that 
period, which is great:


-  --

       account   SUM(position



Assets:Checking           2700.00 USD

Expenses:Food  200.00 USD

Expenses:Interest   10.00 USD

Income:Salary            -3000.00 USD

Liabilities:Credit  90.00 USD

But is there a way to only select the transactions that have postings with 
'Assets:Checking'? So it would look like this

-  --

   account   SUM(position



Assets:Checking           2700.00 USD

Expenses:Food  200.00 USD

Income:Salary    -3000.00 USD

Liabilities:Credit 100.00 USD

So I can see everything that came out of my checking account in the given 
period? Note that the Liabilities:Credit balance is adjusted to no longer 
include the interest charge.



September 20, 2024 at 1:18 PM, "Chary Chary"  wrote:



> 
> P.S. changes to Equity will also be shown
> 
> On Friday, September 20, 2024 at 8:16:13 PM UTC+2 Chary Chary wrote:
> 
> > 
> > Hi,
> > 
> > I am not sure exactly what you want to achieve, but the following query 
> > will give you a sum of all changes to to all accounts (including Assets and 
> > Liabilities) over the period of time 
> > 
> > SELECT account, SUM(position)
> > WHERE date >=2024-09-12 and date <=2024-09-26
> > 
> > This will be all of your incomes, all expenses over the period, as well as 
> > changes to Assets and Liabilities over the same period.
> > 
> > Total will be 0.
> > 
> > On Friday, September 20, 2024 at 6:30:29 PM UTC+2 Oscar & Alejandra wrote:
> > 
> > > 
> > > Hello everyone,
> > > 
> > > I am using the latest beanquery from PyPI (v0.1.dev0) and I'm pretty new 
> > > to SQL and beanquery's SQL-like syntax. I am trying to produce a report 
> > > where I can see all of my expenses for a certain period, something like 
> > > what my bank statement would show but grouped by my expense categories. I 
> > > found this in the examples in the docs and it's pretty close to what I 
> > > want but I'm not sure how to grab the 'transfers' from my asset account. 
> > > For example when I make a payment towards my credit card or when I 
> > > transfer into my savings.
> > > 
> > > FROM OPEN ON 2024-09-12 CLOSE ON 2024-09-26
> > > SELECT account, SUM(position)
> > > WHERE account~"Income|Expenses"
> > > GROUP BY 1
> > > ORDER BY 1
> > > 
> > > The above query produces something like the following:
> > > 
> > > ---  
> > >   account    SUM(position
> > > Expenses:Auto:Fuel  50.00 USD
> > > Expenses:Auto:Insurance    140.00 USD
> > > Expenses:Clothes    20.00 USD
> > > Expenses:Debt:Interest  83.00 USD
> > > Expenses:Food:Dining   100.00 USD
> > > Expenses:Food:Groceries    140.00 USD
> > > Expenses:Home:Utilities:Electricity 80.00 USD
> > > Expenses:Home:Utilities:Water   70.00 USD
> > > Income:Salary    -3000.00 USD
> > > 
> > > Where I would really like something like this
> > > 
> > > ---  
> > >   account    SUM(position
> > > Expenses:Auto:Fuel  50.00 USD
> > > Expenses:Auto:Insurance    140.00 USD
> > > Expenses:Clothes    20.00 USD
> > > Expenses:Debt:Interest  83.00 USD
> > > Expenses:Food:Dining   100.00 USD
> > > Expenses:Food:Groceries    140.00 USD
> > > Expenses:Home:Utilities:Electricity 80.00 USD
> > > Expenses:Home:Utilities:Water   70.00 USD
> > > Income:Salary    -3000.00 USD
> > > Liabilities:CreditCard 100.00 USD
> > > Assets:Savings 300.00 USD
> > > 
> > > Which would essentially show everything my 'paycheck' was spent on for 
> > > the period selected. Is this possible using beanquery?
> > > 
> > > Thanks!
> > > 
> > > Oscar & Ale
> > > 
> > > ma...@aleyoscar.com
> > >
> > 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Beancoun

Re: Struggling to get my first importer working (using Red's Importers)

[Joshua Cabrera]

So, I've got my .ofx file and am not worried about the download task 
anymore. That's the good news.

The bad news is now I'm stuck on the import task.
bean-extract my.import fidelity_1234_july2024.ofx


Results
ERROR:root:Importer 
beancount_reds_importers.importers.fidelity.Importer.identify() raised an 
unexpected error: not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py"
, line 63, in find_imports
matched = importer.identify(file)
  ^^^
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py"
, line 26, in identify
self.initialize_reader(file)
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
, line 26, in initialize_reader
self.ofx = self.read_file(file)
   
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
, line 57, in read_file
return ofxparse.OfxParser.parse(fh)
   
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
, line 396, in parse
ofx_file = OfxPreprocessedFile(file_handle)
   
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
, line 155, in __init__
super(OfxPreprocessedFile, self).__init__(fh)
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
, line 82, in __init__
self.read_headers()
  File 
"C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
, line 95, in read_headers
header, value = line.split(six.b(":"))
^
ValueError: not enough values to unpack (expected 2, got 1)
;; -*- mode: beancount -*-


If I run ofx-summarize, it throws the same error, so I found a sample .ofx 
online and tested ofx-summarize with it. 

And of course, the example works, so it looks like Fidelity is providing an 
incorrectly formatted file? I noticed it doesn't have a single line break 
until the end of the file.

On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote:

> Glad you got it working with ofxget. bean-download is just a wrapper 
> around ofxget or whatever command you have to parallelize all your 
> downloads. If you don’t have several accounts you’re using it with, it’s of 
> limited use. The built in template for fidelity is here, and lets you 
> store your password securely in pass so it’s not stored in plaintext. You 
> could simply paste your working ofxget command into the bean-download 
> config file to get it to work if the parallelization is useful for you. 
> Either way, that’s the last thing I’d do after I got everything else in the 
> import workflow working first.
>
> And oh wow, I’d forgotten that Fidelity stopped 
> 
>  letting 
> you download ofx via their website, and only let you do it via direct 
> download.
> ​
>
> On Monday, September 16, 2024 at 7:31:34 PM UTC-7 Joshua Cabrera wrote:
>
>> Quick update:
>>
>> Installed ofxget and manage to get my ofxget.cfg updated using ofxget 
>> acctinfo. Not sure if this required for bean-download.
>>
>> Also downloaded a .ofx file using:
>> ofxget stmt fidelity > c:\users\joshd\fidelity.ofx
>>
>> Struggling to my download.cfg and bean-download download working - keep 
>> getting:
>> 1/1 sites failed:fidelity
>>
>> Will keep trying whenever I have time this week.
>> On Sunday, September 15, 2024 at 9:24:33 PM UTC-5 Red S wrote:
>>
>>> My reason for using .csv over .ofx is simply familiarity and ease of 
>>> starting up.
>>>
>>> I have no experience with ofx and thought the learning curve would be 
>>> much steeper, and I already feel like I'm drowning a bit as a layperson.
>>>
>>> I know it may seem like that, especially given ofx is not a human 
>>> readable format. But might I suggest giving that a shot first, and you 
>>> might be pleasantly surprised. Because there is nothing to do, no importer 
>>> to write, since it is a machine format with an official spec, unlike csv. 
>>> The learning curve should be close to zero. There are other benefits 
>>> 
>>>  
>>> too.
>>>
>>> I'll look into the csvreader changes you mentioned to try making a 
>>> replacement for prepare_raw_columns() and also look further into ofx.
>>>
>>> If I were you, I’d definitely spend five minutes with ofx first. And 
>>> reds-importers ships with ofx-summarize which you can use to peek 
>>>

Re: Struggling to get my first importer working (using Red's Importers)

[Red S]

This is uncommon, but typically because there was an issue downloading the 
ofx. Usually because of authentication problems. Have you examined the ofx 
to see if it's valid?

Try this to make it more readable:
```
sed 's/>/>\n/g' xyz.ofx
```

You should see something like this on the top:
```

  
0
  
  
INFO
  
  
Success
  

```

If you see that, then try a really simple parse:
```
#!/usr/bin/env python3
import ofxparse

ofx_file_path = "your_ofx_file.ofx"
with open(ofx_file_path, 'r') as ofx_file:
ofx = ofxparse.OfxParser.parse(ofx_file)

print("Account type:", ofx.account.account_type)
print("Account ID:", ofx.account.account_id)
```

On Friday, September 20, 2024 at 2:54:15 PM UTC-7 Joshua Cabrera wrote:

> So, I've got my .ofx file and am not worried about the download task 
> anymore. That's the good news.
>
> The bad news is now I'm stuck on the import task.
> bean-extract my.import fidelity_1234_july2024.ofx
>
>
> Results
> ERROR:root:Importer 
> beancount_reds_importers.importers.fidelity.Importer.identify() raised an 
> unexpected error: not enough values to unpack (expected 2, got 1)
> Traceback (most recent call last):
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py"
> , line 63, in find_imports
> matched = importer.identify(file)
>   ^^^
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py"
> , line 26, in identify
> self.initialize_reader(file)
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
> , line 26, in initialize_reader
> self.ofx = self.read_file(file)
>
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
> , line 57, in read_file
> return ofxparse.OfxParser.parse(fh)
>
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 396, in parse
> ofx_file = OfxPreprocessedFile(file_handle)
>
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 155, in __init__
> super(OfxPreprocessedFile, self).__init__(fh)
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 82, in __init__
> self.read_headers()
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 95, in read_headers
> header, value = line.split(six.b(":"))
> ^
> ValueError: not enough values to unpack (expected 2, got 1)
> ;; -*- mode: beancount -*-
>
>
> If I run ofx-summarize, it throws the same error, so I found a sample .ofx 
> online and tested ofx-summarize with it. 
>
> And of course, the example works, so it looks like Fidelity is providing 
> an incorrectly formatted file? I noticed it doesn't have a single line 
> break until the end of the file.
>
> On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote:
>
>> Glad you got it working with ofxget. bean-download is just a wrapper 
>> around ofxget or whatever command you have to parallelize all your 
>> downloads. If you don’t have several accounts you’re using it with, it’s of 
>> limited use. The built in template for fidelity is here, and lets you 
>> store your password securely in pass so it’s not stored in plaintext. 
>> You could simply paste your working ofxget command into the bean-download 
>> config file to get it to work if the parallelization is useful for you. 
>> Either way, that’s the last thing I’d do after I got everything else in the 
>> import workflow working first.
>>
>> And oh wow, I’d forgotten that Fidelity stopped 
>> 
>>  letting 
>> you download ofx via their website, and only let you do it via direct 
>> download.
>> ​
>>
>> On Monday, September 16, 2024 at 7:31:34 PM UTC-7 Joshua Cabrera wrote:
>>
>>> Quick update:
>>>
>>> Installed ofxget and manage to get my ofxget.cfg updated using ofxget 
>>> acctinfo. Not sure if this required for bean-download.
>>>
>>> Also downloaded a .ofx file using:
>>> ofxget stmt fidelity > c:\users\joshd\fidelity.ofx
>>>
>>> Struggling to my download.cfg and bean-download download working - keep 
>>> getting:
>>> 1/1 sites failed:fidelity
>>>
>>> Will keep trying whenever I have time this week.
>>> On Sunday, September 15, 2024 at 9:24:33 PM UTC-5 Red S wrote:
>>>
 My reason for using .csv over .ofx is simply familiarity and ease of 
 starting up.

 I have no experience with ofx and thought the learning curve w

Re: Struggling to get my first importer working (using Red's Importers)

[Red S]

This is uncommon, but typically because there was an issue downloading the 
ofx. Usually because of authentication problems. Have you examined the ofx 
to see if it’s valid?

Try this to make it more readable: sed 's/>/>\n/g' xyz.ofx
​

You should see something like this on the top:
  0   INFO   Success 
  
​

If you see that, then try a really simple parse:
#!/usr/bin/env python3 import ofxparse ofx_file_path = "your_ofx_file.ofx" 
with open(ofx_file_path, 'r') as ofx_file: ofx = 
ofxparse.OfxParser.parse(ofx_file) print("Account type:", 
ofx.account.account_type) print("Account ID:", ofx.account.account_id) 
​

On Friday, September 20, 2024 at 2:54:15 PM UTC-7 Joshua Cabrera wrote:

> So, I've got my .ofx file and am not worried about the download task 
> anymore. That's the good news.
>
> The bad news is now I'm stuck on the import task.
> bean-extract my.import fidelity_1234_july2024.ofx
>
>
> Results
> ERROR:root:Importer 
> beancount_reds_importers.importers.fidelity.Importer.identify() raised an 
> unexpected error: not enough values to unpack (expected 2, got 1)
> Traceback (most recent call last):
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py"
> , line 63, in find_imports
> matched = importer.identify(file)
>   ^^^
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py"
> , line 26, in identify
> self.initialize_reader(file)
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
> , line 26, in initialize_reader
> self.ofx = self.read_file(file)
>
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
> , line 57, in read_file
> return ofxparse.OfxParser.parse(fh)
>
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 396, in parse
> ofx_file = OfxPreprocessedFile(file_handle)
>
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 155, in __init__
> super(OfxPreprocessedFile, self).__init__(fh)
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 82, in __init__
> self.read_headers()
>   File 
> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
> , line 95, in read_headers
> header, value = line.split(six.b(":"))
> ^
> ValueError: not enough values to unpack (expected 2, got 1)
> ;; -*- mode: beancount -*-
>
>
> If I run ofx-summarize, it throws the same error, so I found a sample .ofx 
> online and tested ofx-summarize with it. 
>
> And of course, the example works, so it looks like Fidelity is providing 
> an incorrectly formatted file? I noticed it doesn't have a single line 
> break until the end of the file.
>
> On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote:
>
>> Glad you got it working with ofxget. bean-download is just a wrapper 
>> around ofxget or whatever command you have to parallelize all your 
>> downloads. If you don’t have several accounts you’re using it with, it’s of 
>> limited use. The built in template for fidelity is here, and lets you 
>> store your password securely in pass so it’s not stored in plaintext. 
>> You could simply paste your working ofxget command into the bean-download 
>> config file to get it to work if the parallelization is useful for you. 
>> Either way, that’s the last thing I’d do after I got everything else in the 
>> import workflow working first.
>>
>> And oh wow, I’d forgotten that Fidelity stopped 
>> 
>>  letting 
>> you download ofx via their website, and only let you do it via direct 
>> download.
>> ​
>>
>> On Monday, September 16, 2024 at 7:31:34 PM UTC-7 Joshua Cabrera wrote:
>>
>>> Quick update:
>>>
>>> Installed ofxget and manage to get my ofxget.cfg updated using ofxget 
>>> acctinfo. Not sure if this required for bean-download.
>>>
>>> Also downloaded a .ofx file using:
>>> ofxget stmt fidelity > c:\users\joshd\fidelity.ofx
>>>
>>> Struggling to my download.cfg and bean-download download working - keep 
>>> getting:
>>> 1/1 sites failed:fidelity
>>>
>>> Will keep trying whenever I have time this week.
>>> On Sunday, September 15, 2024 at 9:24:33 PM UTC-5 Red S wrote:
>>>
 My reason for using .csv over .ofx is simply familiarity and ease of 
 starting up.

 I have no experience with ofx and thought the learning curve would be 
 much steeper, and I alread

Re: Handling reverted transactions in a CVS importer

[Red S]

Glad to hear! If you haven’t seen them already, some of these articles 

 
might save you time too.


On Thursday, September 19, 2024 at 11:42:36 PM UTC-7 ddv...@gmail.com wrote:

> Thanks, that looks like what I need. In fact, your whole repo seems
> extremely useful.
>
> Cheers,
> Vlad
>
> On Thu, Sep 19, 2024 at 8:29 PM Red S  wrote:
> >
> > You could patch basic csv importer, but I find that frequently used 
> importers outgrow that pretty quickly on many other small requirements like 
> this. Writing your own importer can be a good idea if you find that 
> happening. Red’s importers has this. Here’s an example.
> >
> >
> >
> > On Thursday, September 19, 2024 at 7:20:12 AM UTC-7 ddv...@gmail.com 
> wrote:
> >>
> >> Hi everyone,
> >>
> >> Short version: How does one handle reverted transactions? They have 
> their own line in my CSV, listing -2 CHF, but the status is REVERTED.
> >>
> >> Longer version, to potentially avoid an XY problem:
> >>
> >> I'm very new to beancount and set about writing an importer for Revolut 
> based on beangulp's csvbase (which IIUC is the newer way to do it). I found 
> an older thread on the list, but it seems that the csv header has changed 
> since then, and it uses the older beancount.ingest. I also like building 
> things on my own, I suppose that's no surprise around here :)
> >>
> >> I got the basics working, but I wonder how to handle reverted 
> transactions. Google is not super helpful, so maybe Revolut is using the 
> wrong word for this, but AFAICT it's tiny transactions that merchants do to 
> figure out the card is legit:
> >>
> >> Type,Product,Started Date,Completed 
> Date,Description,Amount,Fee,Currency,State,Balance
> >> CARD_PAYMENT,Current,2023-09-09 
> 20:35:29,,Google,-2.00,0.00,CHF,REVERTED,
> >>
> >> I have a few ideas, but I wonder if this isn't already a solved problem:
> >>
> >> 1. Use `grep -v` (or awk, if we're being extra careful) to filter out 
> these lines prior to importing. But I'd like to see if I can do this 
> entirely with beancount
> >>
> >> 2. Override `finalize`, check the `State` column, and add a posting of 
> (-value) to reverted transactions. This works, but I don't like that it 
> pollutes the ledger. I also don't like that these reverted lines have an 
> empty `Balance` cell, which means I can't use csvbase's nifty automatic 
> balance handling.
> >>
> >> 3. Is it possible to add a `skip_line` customization point to the 
> csvbase importer? This would allow configuring my importer to skip reverted 
> lines, while keeping the balance handling (because lines will be skipped 
> before attempting to fetch the balance column). If you think that's a 
> sensible approach I'm happy to take a stab at contributing a patch.
> >>
> >> Thanks in advance,
> >> Vlad
> >
> > --
> > You received this message because you are subscribed to a topic in the 
> Google Groups "Beancount" group.
> > To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/beancount/jnAZSnzm1Ts/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to 
> beancount+...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/beancount/3b14aa08-0456-471e-bd6f-a3074392c543n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/af798bf1-8b37-4edf-a741-4a2479cf7dbfn%40googlegroups.com.

Re: Struggling to get my first importer working (using Red's Importers)

[Joshua Cabrera]

Thanks, Red. The file itself looks okay after another look.

I tried the new line trick, but it didn't change the results.

The simple parse threw the same unpacking value error on my ofx while 
working on the sample ofx.

I'm sharing a sanitized version of my ofx (replaced account number, 
transaction details, etc.) and my import file.

In the meantime, I'll try a different Fidelity account and smaller 
date/transaction ranges.
On Friday, September 20, 2024 at 5:09:18 PM UTC-5 Red S wrote:

> This is uncommon, but typically because there was an issue downloading the 
> ofx. Usually because of authentication problems. Have you examined the ofx 
> to see if it’s valid?
>
> Try this to make it more readable: sed 's/>/>\n/g' xyz.ofx
> ​
>
> You should see something like this on the top:
>   0   INFO   Success 
>   
> ​
>
> If you see that, then try a really simple parse:
> #!/usr/bin/env python3 import ofxparse ofx_file_path = "your_ofx_file.ofx" 
> with open(ofx_file_path, 'r') as ofx_file: ofx = 
> ofxparse.OfxParser.parse(ofx_file) print("Account type:", 
> ofx.account.account_type) print("Account ID:", ofx.account.account_id) 
> ​
>
> On Friday, September 20, 2024 at 2:54:15 PM UTC-7 Joshua Cabrera wrote:
>
>> So, I've got my .ofx file and am not worried about the download task 
>> anymore. That's the good news.
>>
>> The bad news is now I'm stuck on the import task.
>> bean-extract my.import fidelity_1234_july2024.ofx
>>
>>
>> Results
>> ERROR:root:Importer 
>> beancount_reds_importers.importers.fidelity.Importer.identify() raised an 
>> unexpected error: not enough values to unpack (expected 2, got 1)
>> Traceback (most recent call last):
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py"
>> , line 63, in find_imports
>> matched = importer.identify(file)
>>   ^^^
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py"
>> , line 26, in identify
>> self.initialize_reader(file)
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
>> , line 26, in initialize_reader
>> self.ofx = self.read_file(file)
>>
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py"
>> , line 57, in read_file
>> return ofxparse.OfxParser.parse(fh)
>>
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>> , line 396, in parse
>> ofx_file = OfxPreprocessedFile(file_handle)
>>
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>> , line 155, in __init__
>> super(OfxPreprocessedFile, self).__init__(fh)
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>> , line 82, in __init__
>> self.read_headers()
>>   File 
>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py"
>> , line 95, in read_headers
>> header, value = line.split(six.b(":"))
>> ^
>> ValueError: not enough values to unpack (expected 2, got 1)
>> ;; -*- mode: beancount -*-
>>
>>
>> If I run ofx-summarize, it throws the same error, so I found a sample 
>> .ofx online and tested ofx-summarize with it. 
>>
>> And of course, the example works, so it looks like Fidelity is providing 
>> an incorrectly formatted file? I noticed it doesn't have a single line 
>> break until the end of the file.
>>
>> On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote:
>>
>>> Glad you got it working with ofxget. bean-download is just a wrapper 
>>> around ofxget or whatever command you have to parallelize all your 
>>> downloads. If you don’t have several accounts you’re using it with, it’s of 
>>> limited use. The built in template for fidelity is here, and lets you 
>>> store your password securely in pass so it’s not stored in plaintext. 
>>> You could simply paste your working ofxget command into the 
>>> bean-download config file to get it to work if the parallelization is 
>>> useful for you. Either way, that’s the last thing I’d do after I got 
>>> everything else in the import workflow working first.
>>>
>>> And oh wow, I’d forgotten that Fidelity stopped 
>>> 
>>>  letting 
>>> you download ofx via their website, and only let you do it via direct 
>>> download.
>>> ​
>>>
>>> On Monday, September 16, 2024 at 7:31:34 PM UTC-7 Joshua Cabrera wrote:
>>>
 Quick update:

 Installed ofxget and manage to get my ofxget.cfg updated using ofxget 
>>