Hi Rico!
We have finished the first part of the Window API reworks. You can find the
code here: https://github.com/apache/flink/pull/1175
It should fix the issues and offer vastly improved performance (up to 50x
faster). For now, it supports time windows, but we will support the other
cases in th
In the issue, it states that it should be sufficient to append the
hbase-protocol.jar file to the Hadoop classpath. Flink respects the Hadoop
classpath and will append it to its own classpath upon launching a cluster.
To do that, you need to modify the classpath with one of the commands
below. Not
I was just thinking that it would be nice to have an "Option" or "Nullable"
type in the Java API that the type extractor understands and that can be
used for exactly that case...
On Wed, Sep 23, 2015 at 5:26 PM, Stephan Ewen wrote:
> One part why the tuples are faster is actually the simpler ser
One part why the tuples are faster is actually the simpler serialization
format without null values...
On Wed, Sep 23, 2015 at 5:24 PM, Flavio Pompermaier
wrote:
> I could but I heard about the fact that the Tuple model is much faster
> than the Pojo one, isn't it?
> How big is such difference?
I could but I heard about the fact that the Tuple model is much faster than
the Pojo one, isn't it?
How big is such difference? Is there any blog post about that?
Thanks for the support,
Flavio
On Wed, Sep 23, 2015 at 4:00 PM, Fabian Hueske wrote:
> Flink's tuple data types do not support null
Flink's tuple data types do not support null fields.
A program with null-valued tuple fields will fail as soon as the tuple is
serialized (written into memory, to disk, or shipped over the network).
If you need support for null values, you can for example implement a custom
POJO data type.
Best, F
In general, Tuples may not have null fields - the tuples have no handling
for null.
It is just by chance that String actually handles itself with null.
In general the unspecified (but effective) behavior is that if the types
themselves support null handling, then you can use them. But I would not
Are you talking about a Tuple2 t for example?
So t.f0 == null doesn't cause any issues, but t.f1 == null is failing?
What's the error message?
On Wed, Sep 23, 2015 at 3:31 PM, Flavio Pompermaier
wrote:
> Hi to all,
>
> why it is possible to collect (in a flatMap function for example) a Tuple
> w
Hi to all,
why it is possible to collect (in a flatMap function for example) a Tuple
with a null String but it's not possible with a null Boolean value? Is that
normal?
Best,
Flavio
Hi Jack!
Interesting use case. I think the answer depends a bit on how you want to
make your setup:
Variant 1:
-
If you have a YARN setup and allocate a new Flink AppMaster and dedicated
workers, you can probably scale to as many topologies as you want (or as
many as YARN can handle).
Thanks Aljoscha, that works!
I tried passing values to base class constructor. Modifying the previous
example slightly:
class Base(var a: Int)
class X(b: Int) extends Base(b) {
this() { this(0) }
}
The code runs (even without Kryo) but compiler complains about b being
immutable.
> On 23 Sep
Hi Jack,
Stephan is right, this should work. Unfortunately the TypeAnalyzer does not
correctly detect that it cannot treat your Id class as a Pojo. I will add a
Jira issue for that. For the time being you can use this command to force
the system to use Kryo:
env.getConfig.enableForceKryo();
I hop
Hi Stephan!
Here's the trace (flink 0.9.1 + scala 2.10.5)
Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:314)
at
scala.runti
I don't think this is a critical issue. Its logged at DEBUG level.
I think you would have to specify the environment variable on all machines
in your cluster. BUT as far I know, HADOOP_HOME has been deprecated in
newer Hadoop (2.2+) versions.
On Wed, Sep 23, 2015 at 1:41 PM, Lydia Ickler
wrote:
Added the -Dhadoop.home.dir= to flink-conf.yaml but still get the same error.
Could it be that it somehow has to be added to HBase directly?
> Am 23.09.2015 um 13:27 schrieb Ufuk Celebi :
>
> Either specify the env variable HADOOP_HOME or set the JVM property
> ‘hadoop.home.dir’ (-Dhadoop.home.
Hi Jack!
This should be supported, there is no strict requirement for mutable types.
The POJO rules apply only if you want to use the "by-field-name" addressing
for keys. In Scala, you should be able to use case classes as well, even if
they are immutable.
Can you post the exception that you get
Either specify the env variable HADOOP_HOME or set the JVM property
‘hadoop.home.dir’ (-Dhadoop.home.dir=…)
– Ufuk
> On 23 Sep 2015, at 12:43, Lydia Ickler wrote:
>
> Hi all,
>
> I get the following error message that no valid hadoop home directory can be
> found when trying to initialize th
Hi,
I'm having trouble integrating existing Scala code with Flink, due to POJO-only
requirement.
We're using AnyVal heavily for type safety, and immutable classes as a default.
For example, the following does not work:
object Test {
class Id(val underlying: Int) extends AnyVal
class X(va
Hi all,
I get the following error message that no valid hadoop home directory can be
found when trying to initialize the HBase configuration.
Where would I specify that path?
12:41:02,043 INFO org.apache.flink.addons.hbase.TableInputFormat
- Initializing HBaseConfiguration
12:41
Hello,
I need to implement an engine for running a large number (several hundred) of
different kinds of topologies. The topologies are defined ad-hoc by customers,
hence combining the topologies is difficult.
How does Flink perform with so many topologies?
Thanks
It might me that this is causing the problem:
https://issues.apache.org/jira/browse/HBASE-10304
In your log I see the same exception. Anyone has any idea what we could do
about this?
On Tue, 22 Sep 2015 at 22:40 Lydia Ickler wrote:
> Hi,
>
> I am trying to get the HBaseReadExample to run. I hav
21 matches
Mail list logo
