Re: [Question] Redirect Write Failures to Dead Letter Queue

2025-04-03 Thread Jonathan Hope
Of the supported Beam languages the one I know the best is Go so that's what I've been working in. What I ended up doing was forking the databaseio code and changing it so that any rows that had an error on a write are emitted as a pcollection, and the error in question is also emitted as a pcollec

Re: [Question] Redirect Write Failures to Dead Letter Queue

2025-04-03 Thread Radek Stankiewicz via user
hi Jonathan, Is there a specific IO in mind you would like to use? If I think about enforcing key constraints, then I think about sql databases like pgsql, mysql, alloydb. In Beam those databases are supported in JDBCIO. Problem with JDBCIO is that it doesn't yet have an error handler - https://gi

[Question] Redirect Write Failures to Dead Letter Queue

2025-04-01 Thread Jonathan Hope
Hello, I had a question and was hoping this was the right place to ask. Let's say I'm moving data from a NoSQL database to a SQL database. Here is an example document in the NoSQL database: { "id": "1234", "identifier": "5678" } The id is system generated, and the identifier is user provided.

Re: [Question] Redirect Write Failures to Dead Letter Queue

2025-04-01 Thread Robert Bradshaw via user
On Tue, Apr 1, 2025 at 12:01 PM Jonathan Hope wrote: > > Unfortunately both databases will be online during this so conflicts could > occur in either direction. I had previously dug up an answer around modifying > the JdbcIO here: > https://stackoverflow.com/questions/56398422/exception-handlin

Re: [Question] Redirect Write Failures to Dead Letter Queue

2025-04-01 Thread Robert Bradshaw via user
Good question. I think it depends on who else is modifying the SQL database. In the easy case (e.g. everything you want to write to your SQL database comes from the NoSQL source) you could group (e.g. via a GroupByKey) on your identifier, filter out duplicates with a subsequent DoFn, and then writ

Re: [Question] Redirect Write Failures to Dead Letter Queue

2025-04-01 Thread Jonathan Hope
Unfortunately both databases will be online during this so conflicts could occur in either direction. I had previously dug up an answer around modifying the JdbcIO here: https://stackoverflow.com/questions/56398422/exception-handling-in-apache-beam-pipelines-when-writing-to-database-using-java. But