I don't recommend this kind of design because you loose physical data locality and you will be affected by "bad neighboors" that are also using the network storage... We have one similar design but restricted to small clusters (more for experiments than production)
2017-06-01 9:47 GMT+02:00 Mich Talebzadeh <mich.talebza...@gmail.com>: > Thanks Jorn, > > This was a proposal made by someone as the firm is already using this tool > on other SAN based storage and extend it to Big Data > > On paper it seems like a good idea, in practice it may be a Wandisco > scenario again.. Of course as ever one needs to EMC for reference calls > ans whether anyone is using this product in anger. > > > > At the end of the day it's not HDFS. It is OneFS with a HCFS API. > However that may suit our needs. But would need to PoC it and test it > thoroughly! > > > Cheers > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 1 June 2017 at 08:21, Jörn Franke <jornfra...@gmail.com> wrote: > >> Hi, >> >> I have done this (not Isilon, but another storage system). It can be >> efficient for small clusters and depending on how you design the network. >> >> What I have also seen is the microservice approach with object stores >> (e.g. In the cloud s3, on premise swift) which is somehow also similar. >> >> If you want additional performance you could fetch the data from the >> object stores and store it temporarily in a local HDFS. Not sure to what >> extent this affects regulatory requirements though. >> >> Best regards >> >> On 31. May 2017, at 18:07, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Hi, >> >> I realize this may not have direct relevance to Spark but has anyone >> tried to create virtualized HDFS clusters using tools like ISILON or >> similar? >> >> The prime motive behind this approach is to minimize the propagation or >> copy of data which has regulatory implication. In shoret you want your data >> to be in one place regardless of artefacts used against it such as Spark? >> >> Thanks, >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >
