Hi, Any design (in whatever framework) needs to consider both Functional and non-functional requirements. Functional requirements are those which are related to the technical functionality of the system that we cover daily in this forum. The non-functional requirement is a requirement that specifies criteria that can be used to judge the operation of a system conditions, rather than specific behaviours. From my experience the non-functional requirements are equally important and in some cases they are underestimated when systems go to production. Probably, most importantly they need to cover the following:
- *Processing time meeting a service-level agreement (SLA). * You can get some of this from Spark GUI. Are you comfortably satisfying the requirements? How about total delay, Back pressure etc. Are you within your SLA. In streaming applications, there is no such thing as an answer which is supposed to be late and correct. The timeliness is part of the application. If we get the right answer too slowly it becomes useless or wrong. We also need to be aware of latency trades off with throughput. - *Capacity, current and forecast. * What is the current capacity? Have you accounted for extra demands, sudden surge and loads such as year-end. Can your pipeline handle 1.6-2 times the current load - *Scalability* How does your application scale if you have to handle multiple topics or new topics added at later stages? Scalability also includes additional nodes, on-prem or having the ability to add more resources such as Google Dataproc compute engines etc - *Supportability and Maintainability* Have you updated docs and procedures in Confluence or equivalent or they are typically a year old :). Is there any single point of failure due to skill set? Can ops support the design and maintain BAU themselves. How about training and hand-over - *Disaster recovery and Fault tolerance* What provisions are made for disaster recovery. Is there any single point of failure in the design (end to end pipeline). Are you using standalone dockers or Google Kubernetes Engine (GKE or equivalent) HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 26 Apr 2021 at 17:16, ashok34...@yahoo.com.INVALID <ashok34...@yahoo.com.invalid> wrote: > Hello, > > When we design a typical spark streaming process, the focus is to get > functional requirements. > > However, I have been asked to provide non-functional requirements as well. > Likely things I can consider are Fault tolerance and Reliability (component > failures). Are there a standard list of non-functional requirements for > Spark streaming that one needs to consider and verify all? > > Thanking you >
