This is phenomenal work! Thanks for the hard work, and improving Beam's infra automation! Thank you too Pablo for mentoring!
Robert Burke On Fri, Sep 12, 2025, 9:58 PM Enrique Calderon <[email protected]> wrote: > Hi everyone, > > As my Google Summer of Code 2025 project with Apache Beam comes to a > close, I'm excited to share a summary of the work I've done. My project > focused on building a new set of tools to automate our infrastructure, > improve security, and make managing our GCP resources much easier. > > I developed a full suite of tools that work together to create a more > secure, efficient, and cost-effective infrastructure for the project. > What I Delivered > > - > > *Automated Resource Cleaner:* I built a tool that automatically finds > and removes old, unused GCP Pub/Sub topics and subscriptions. This will > help reduce clutter and save on costs without manual intervention. > - > > *Git-Based Access Control:* I implemented a more transparent way to > manage GCP permissions. All access control is now handled through a central > users.yml file using Terraform, which means all changes are managed > via pull requests, creating a clear audit trail. > - > > *Automatic Key Rotation & Compliance:* To boost security, I built a > framework that automatically rotates service account keys on a schedule. I > also created an *Infrastructure Enforcer* that runs regular checks to > ensure our configurations comply with these new standards. It's currently > in a dry-run mode, but the plan is for it to send email notifications to > this list for any issues it finds. > - > > *Security Monitoring:* Finally, I set up a security log analyzer to > monitor GCP audit logs for suspicious activity related to access control > and service account keys. It's designed to send weekly reports and is > currently in its final testing stage > > How This Affects You > > - > > *To Request GCP Access:* If you need to request or change GCP > permissions, the process is now managed entirely through a pull request. > Simply submit your changes to the infra/iam/users.yml configuration > file. > - > > *To Request a Service Account Key:* Similarly, if you need a service > account managed by this new system, please create a pull request with your > change in infra/keys/keys.yaml > > This has been an incredible learning experience, and I want to extend a > huge thank you to my mentor @pabloem and the entire Apache Beam community > for your support and guidance throughout the summer. > > If you’re interested in the technical details, you can find them in the > links below. I've also added a summary of this work to the latest community > draft report. > > - > > *GSoC Project Page:* > https://summerofcode.withgoogle.com/programs/2025/projects/QRKMhW67 > - > > *Final Work Report:* > https://gist.github.com/ksobrenat32/b028b8303393afbe73a8fc5e17daff90 > - > > *Community Draft Report Contribution:* > https://s.apache.org/beam-draft-report-2025-09 > > > > Best regards, > > Enrique Calderon > >
