AvroIO read SpecificRecord with custom reader schema?

Hi, Is it just me or is there no way for AvroIO to read SpecificRecords with a custom reader schema? AvroIO.read(Class recordClass) will use the schema of T and there's no way to override it. Cheers, Neville

Re: AvroIO read SpecificRecord with custom reader schema?

wrote: > Does AvroIO.readGenericRecords() work ? > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333 > > Thanks, > Cham > > On Thu, Jun 13, 2019 at 1:46 PM Neville Li wrote: > >> Hi, >> >> I

SDK support status clarification

Hi all, more specifically Googlers here, I want to clarify the Beam SDK support status w.r.t. Dataflow runner here: https://cloud.google.com/dataflow/docs/support/sdk-version-support-status When a Beam SDK is deprecated, what does it mean for users running it on Dataflow? The page mentions that

Scio 0.8.0-beta2 released

Hi all, We've just released Scio 0.8.0-beta2 (Scala API for Apache Beam for those on the Beam mailing list). This will likely be the last beta before 0.8.0 stable release. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.8.0-beta2 Features - Bump Beam to 2.16.0 (#2292

Scio 0.8.0 released

Hi all, We just released Scio 0.8.0. This is based on the most recent Beam 2.17.0 release and includes a lot of new features & bug fixes over the past 10 months. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.8.0 *"Amato Animo Animato Animagus"* Breaking changes & deprecations

Scio 0.8. 2 released

Hi all, We just released Scio 0.8.2. This is mainly a bug fix release with some small improvements. Thanks to all the contributors! Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.8.2 *"Capacious Extremis"* Features - Update Beam to 2.19.0 (#2665

Scio 0.8.3 released

Hi all, We just released Scio 0.8.3. This is mainly a bug fix release with some minor improvements. *Notes on JDK 11* JDK 11 is now supported starting with 0.8.2. To upgrade, you'll need to build your code and submit jobs to Dataflow with JDK 11. *Notes on Scala 2.11* Scio 0.8.x will be the last

Re: Scio 0.8.3 released

Iterable slightly lazy (#2749 <https://github.com/spotify/scio/pull/2749>)" #2791 <https://github.com/spotify/scio/pull/2791> On Fri, Mar 20, 2020 at 2:10 PM Neville Li wrote: > Hi all, > > We just released Scio 0.8.3. This is mainly a bug fix release with some >

Scio 0.9.0 released

Hi all, We just released Scio 0.9.0. The biggest change was dropping Scala 2.11 and adding 2.13 support. Also included are Guava/magnolia powered Bloom Filter, improved test messages and other improvements. For those using Featran for feature engineering, the

Scio 0.9.3 released

Hi all, We just released Scio 0.9.3. This bumps Beam SDK to 2.23.0 and includes a lot of improvements & bug fixes. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.9.3 *"Petrificus Totalus"* There are no breaking changes in this release, but some were introduced with v0.9.0: Se

Scio 0.9.4 released

Hi all, We just released Scio 0.9.4. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.9.4 *"Deletrius"* There are no breaking changes in this release, but some were introduced with v0.9.0: See v0.9.0 Migration Guide

Fwd: Scio community hangout

Cross posting here in case you're interested or already using Scio <https://github.com/spotify/scio>, the Scala API for Apache Beam. -- Forwarded message - From: Neville Li Date: Mon, Sep 14, 2020 at 1:49 PM Subject: Scio community hangout To: Scio Users Hi all

Re: Scio community hangout

So this is happening next Thu Oct 1 12PM US EDT/9AM PDT/16:00 UTC. Drop in if you want to chat or see what we're up to. See you there! meet.google.com/vze-suyd-kwd On Mon, Sep 14, 2020 at 1:51 PM Neville Li wrote: > Cross posting here in case you're interested or already using

Scio 0.9.5 released

Hi all, We just released Scio 0.9.5. This release upgrades Beam to the latest 2.24.0 and includes several improvements and bug fixes, including Parquet Avro dynamic destinations, Scalable Bloom Filter and many others. This will also likely be the last 0.9.x release before we start working on the n

Scio 0.10.0 released

Hi all, We just released Scio 0.10.0. Here's a short summary of the notable changes since 0.9.x: - Better decoupled Google Cloud Platform dependencies - Simplify coder implicits for faster compilation - Sort Merge Bucket performance improvements and bug fixes - Type-safe Parquet support - Parquet

Fwd: Scio 0.4.6 released

: 'Neville Li' via Scio Users Date: Tue, Dec 19, 2017 at 1:12 PM Subject: Scio 0.4.6 released To: Hi all, We just released Scio 0.4.6. It's based on Beam 2.2.0 now with a lot of improvements and bug fixes. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.4.6 *&

Scio 0.4.7 released

Hi all, We just release Scio 0.4.7. This release fixed a join performance regression and introduced several improvements. https://github.com/spotify/scio/releases/tag/v0.4.7 *"Hydrochoerus hydrochaeris"* Features - Add support for TFRecordSpec #990

Re: Scio 0.4.7 released

ems in Scio now ? > > Regards > JB > > On 01/04/2018 09:41 PM, Neville Li wrote: > > Hi all, > > > > We just release Scio 0.4.7. This release fixed a join performance > regression and > > introduced several improvements. > > > > https://github.com

Scio 0.5.0-alpha1 is out

Hi all, We just released Scio 0.5.0-alpha1. This release includes a typed BigQuery performance improvement by bypassing intermediate TableRow JSONs. It has shown a 2x speed up in some of our benchmarks. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.5.0-alpha1 *"Ia Io"* Breakin

Re: Trying to understand Unable to encode element exceptions

Not sure why it falls back to SerializableCoder. Can you file an GH issue with ideally a snippet that can reproduce the problem? On Fri, Jan 19, 2018, 7:43 AM Carlos Alonso wrote: > Hi everyone!! > > I'm building a pipeline to store items from a Google PubSub subscription > into GCS buckets. In

Re: Trying to understand Unable to encode element exceptions

s wondered. How does one specify a coder for a > case class? > > Regards > > On Fri, 19 Jan 2018 at 15:51, Neville Li wrote: > >> Not sure why it falls back to SerializableCoder. Can you file an GH issue >> with ideally a snippet that can reproduce the problem? >

Re: Trying to understand Unable to encode element exceptions

this exception arises: > https://pastebin.com/xxdDMXSf > > Thanks! > > On Fri, Jan 19, 2018 at 4:35 PM Neville Li wrote: > >> You shouldn't manually set coder in most cases. It defaults to >> KryoAtomicCoder for most Scala types. >> More details: >> https

Re: Trying to understand Unable to encode element exceptions

In this case it's probably easiest to map the scala `Map[K, V]` into a `java.util.Map` and explicitly set a `MapCoder` so you don't have to deal with internal coder inference. On Fri, Jan 19, 2018 at 11:03 AM Neville Li wrote: > That happens when you mix beam transforms into sci

Re: Trying to understand Unable to encode element exceptions

> > Thanks! > > On Fri, Jan 19, 2018 at 5:19 PM Neville Li wrote: > >> In this case it's probably easiest to map the scala `Map[K, V]` into a >> `java.util.Map` and explicitly set a `MapCoder` so you don't >> have to deal with internal coder inference. >

Re: Trying to understand Unable to encode element exceptions

your help! > > On Fri, Jan 19, 2018 at 5:04 PM Neville Li wrote: > >> That happens when you mix beam transforms into scio and defeats the >> safety we have in place. Map the values into something beam-serializable >> first or rewrite the transform with a scio built-in wh

Re: Trying to understand Unable to encode element exceptions

> > As a side note, I found this issue: > https://github.com/spotify/scio/issues/448 > > I can share/help there with our experience, as our job, with scio + > stateful + timely processing is working fine as of today > > Regards!! > > On Fri, Jan 19, 2018 at 6:21 P

Re: Trying to understand Unable to encode element exceptions

Here's a fix to #1020 https://github.com/spotify/scio/pull/1032 On Sun, Jan 21, 2018 at 4:36 PM Neville Li wrote: > Awesome! > We have't wrapped any stateful processing API in scala but if you have > working snippet or ideas it'd be great to share in that ticket. >

Fwd: Scio 0.5.0-alpha2 released

+user@beam.apache.org -- Forwarded message - From: Neville Li Date: Mon, Jan 29, 2018 at 6:54 PM Subject: Scio 0.5.0-alpha2 released To: d...@beam.apache.org Hi all, We just released Scio 0.5.0-alpha2. This is mostly a bug fix release. We'll probably have one or 2

Re: Scio 0.5.0 released

On Fri, Mar 9, 2018 at 4:31 PM 'Eugene Kirpichov' via Scio Users < scio-us...@googlegroups.com> wrote: > Hi! > > On Fri, Mar 9, 2018 at 1:22 PM Rafal Wojdyla wrote: > >> Hi all, >> >> We have just released Scio 0.5.0. This is a major/breaking release - make >> sure to read the breaking changes se

Scio 0.5.2 released

Hi all, We just released Scio 0.5.2 with a few enhancements and bug fixes. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.5.2 *"Kobus kob"* Features - Add Java Converters #1013 #1076

Scio 0.5.4 released

Hi all, We just released Scio 0.5.4. This release includes 2 important serialization fixes and a few new features. We recommend upgrading if you're on 0.5.x. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.5.4 *"Marmota monax"* Features - Add satisfySingleValue SCollection m

Scio 0.5.6 released

Hi all, We just released Scio 0.5.6. This release includes a lot of improvements, bug fixes, and is based on the latest Beam 2.5.0. https://github.com/spotify/scio/releases/tag/v0.5.6 *"Orcinus orca"* Breaking changes - Beam 2.5.0 introduced a breaking change in the metrics API. See e664dfc

Scio 0.6.1 released

Hi all, We just released Scio 0.6.1. This is mainly a bug fix release. Also just a heads up, we'll be releasing alpha/beta of 0.7.0 soon which will include some major breaking changes. Keep an eye on this section of t

Scio 0.7.4 released

Hi all, We just released Scio 0.7.4. The biggest change was upgrading to Apache Beam 2.11.0. https://github.com/spotify/scio/releases/tag/v0.7.4 *"Watsonula wautieri"* Features - Add sequence example support to scio-tensorflow (#1757 ) - Add c