Hi Sana, I don't have much to offer. I haven't heard anyone doing any work directly towards energy efficiency per se, but indirectly yes. I have seen companies optimising performance of their workloads, with an ultimate goal of assigning fewer resources to a cluster in order to save up on a limited electricity budget in their data centers.
>From the Open Source perspective, we are trying to optimize Apache Flink, fix performance bottlenecks and fight against performance regressions. To this effect we primarily rely on our set of micro benchmarks [1] and occasional cluster level macro benchmarks, either with some artificial jobs, or TPC-DS benchmark suite for example. > What Linux kernel configurations are used? Has any OS tuning been done? > if anyone has tried to optimize the underlying OS/VM/container to achieve these outcomes. I don't remember those topics popping up in discussion around performance. My best guess is that the teams managing the hardware or containers are very far away from the teams that are actually touching Apache Flink in any way. Often for example teams using/touching Apache Flink don't have any guarantees or any knowledge about the environment. Also my best guess is that there are more lower hanging fruits to solve first before touching those lower layers. But I might be wrong and would be happy to learn something :) Do you maybe have some suggestions? What things would you expect us to try out in the future? Best, Piotrek [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847 wt., 23 sie 2022 o 17:45 Sharma, Sanskriti, Rakesh <sa...@bu.edu> napisał(a): > Hi everyone, > > > We are a team of researchers at Boston University investigating the energy > and performance behavior of open-source stream processing platforms. We > have started looking into Flink and we wanted to reach out to community to > see if anyone has tried to optimize the underlying OS/VM/container to > achieve these outcomes. > > > Some of the specific aspects we would like to explore include the > following: What Linux kernel configurations are used? Has any OS tuning > been done? What workloads are used to evaluate performance/efficiency, both > for turning and more generally to evaluate the impact of changes to either > the software or hardware? What is considered a baseline network setup, with > respect to both hardware and software? Has anyone investigated the policy > used in terms of the cpufreq governor ( > https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt)? > > > It would be especially helpful to hear from people running Flink in > production or offering it as a service. > > Thank you! > > Sana > >
