退订
Unsubscribe
Unsubscribe
I run a flink job,when run one hour,there have a error:
ERROR org.apache.flink.runtime.webmonitor.handlers.TaskManagerLogHandler -
Fetching TaskManager log failed.
java.util.NoSuchElementException: None.get
Hi,all
in spark,the submit job can have --files,this means" Comma-separated list of
files to be placed in the working directory of each executor."
so,in flink,if there have the same method,i use --classpath file:///,but
the job run error,there has not the file.
today,i use flink-1.2.0,and run a job on yarn,the commend is:
flink-1.2.0/bin/flink run \-m yarn-cluster \-yn 2 \-ys 4 \-yjm 3072 \-ytm 2048
\--class statics.ComputevecSim \--classpath
file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar \--classpath
file:///opt/cloudera/parcels/CDH/lib/h
hi,all,i run a job,it is
:-val data =
env.readTextFile("hdfs:///")//DataSet[(String,Array[String])]val dataVec =
computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]val rescomm =
computeCosSims (dataVec)//DataSet[(String,Ar
yes,it is YARN single job,use the commend:
flink-1.1.1/bin/flink run -m yarn-cluster \-yn 2 \-ys 2 \-yjm 2048 \-ytm 2048
\--class statics.ComputeDocSim \--classpath
file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar \--classpath
file:///opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.
hi,i read file from hdfs,but there is error when run jon on yarn
clutster,---val dataSeg =
env.readTextFile("hdfs:///user/hadoop/text").filter(!_.startsWith("#")).map { x
=>
val values = x.split("\t")
(values.apply(0),values.appl
Hi ,alll,
i have a 36000 documents,and the document all transfer a vector , one doc is a
vector,and dimension is the same,so have DataSet
val data :DataSet[(String,SparseVector)]= //36000 record
val toData = data.collect()
val docSims = data.map{x=>
val fromId=x._
i have the same problem,but i put the flink job into yarn.
but i put the job into yarn on the computer 22,and the job can success run,and
the jobmanager is 79 and taskmanager is 69,they three different compu345ter,
however,on computer 22,the pid=3463,which is the job that put into yarn,is have
2.
Hi, i have a flink job,and abt assembly to get a jar file,so i put it to
yarn and run it,use the follow
commend:/home/www/flink-1.1.1/bin/flink
run \-m yarn-cluster \-yn 1 \-ys 2 \-yjm 4096 \-ytm 4096 \--class
skRecomm.
your message is very short,i can not read more.the follow is my guss,
in flink,the dataStream is not for iterative computation,the dataSet would
be more well.and fink suggest broadcast mini data,not large.
your can load your model data (it can be from file,or table),before main
function,a
think your anwser.
but i can not get your ideal."If all elements of "words2" have been processed,
the right side of your coGroup will always be empty no matter what is incoming
in your socketTextStream.",the mean i can not get.
the following is the ideal from me(it maybe error):
the coG
my data from a Hbase table ,it is like a List[rowkey,Map[String,String]],
class MySplittableIterator extends SplittableIterator[String]{
// Members declared in java.util.Iterator
def hasNext(): Boolean = {
}
def next(): Nothing = {
}
// Members decl
i try read data into a list or List[Map] to store the T2,but i think if use
list or List[Map],there is not parallelization,so i want to use coGroup.
other hand,the coGroup function is join the T1 and T2,and must have window and
trigger method,the window is cut the T1 and T2,
the trigger is trigg
Hi, the follow code:
val text = env.socketTextStream(hostName, port)val words1 = text.map {
x => val res = x.split(",") (res.apply(0)->res.apply(1))}
val words2 = env.fromElements(("a","w1"),("a","w2"),("c","w3"),("d","w4"))
val joinedStream = words1 .co
fromCollection is not parallelization,the data is huge,so i want to use
env.fromParallelCollection(data),but the data i do not know how to initialize,
- 原始邮件 -
发件人：Maximilian Michels
收件人："user@flink.apache.org" , rimin...@sina.cn
主题：Re: fromParallelCollection
日期：2016年09月05日 16点58分
Plea
Hi，val env = StreamExecutionEnvironment.getExecutionEnvironment val tr
= env.fromParallelCollection(data)
the data i do not know initialize,some one can tell me..
Hi, i have a problem,a dataStream read from rabbitMQ,and others data from a
hbase table,which is a dataSet.Those two data from follow:
val words=connectHelper.readFromRabbitMq(...) // words is
DataStream[String] val dataSet=HBaseWrite.fullScan() //dataSet is
DataSet[(int,Strin
Hi, in flink,the datastream have reduce Transformations,but the result do
not satisfy for me,for example,val pairs2 =
env.fromCollection((Array(("a", Map(3->"rt")),("a", Map(4->"yt")),("b",
Map(5->"dfs")val re=
pairs2.keyBy(0).reduce((x1,x2)=>(x1._1,x2._2++x1._2))re.map
Hi,every one , when i use scala version 2.10,and set the sbt project(add
those:flink-core,flink-scala,flink-streaming-scala,flink-kafka,flink-streaming-connectors,),the
result download the flink-shaded-hadoop1_2.10.jar,but use scala version 2.11,i
got flink-shaded-hadoop1_2.10.jar and flink-
Hi，today，I use flink to rewrite my spark project,in spark ,data is rdd,and
it have much transformations and actions,but in flink,the DataStream does not
have groupby and foreach, for example,val
env=StreamExecutionEnvironment.createLocalEnvironment() val
data=List(("1"->"a
23 matches
Mail list logo
