**
*Event Summary*
*
Date of incident - 2019-05-14
Time event began - 2:13am PDT
Project services restored - 6:15AM PDT
Final resolution - Clean up the incursion and upgrade “Script Security”
Plugin for Jenkins to avoid repeat infection.
Causes
Root cause - shell escape exploit in the “Script Security” plugin for
Jenkins, remotely injected via a specially crafted URL. The exploit
caused the Jenkins server to crash, but succeeded in running the
payload. A bitcoin miner was installed under the Jenkins user for
*FD.io* Jenkins.
Other contributing causes:
*
Jenkins / plugin version 1.48 , latest is 1.58
Effects
Downtime for projects: FD.IO Jenkins
**
Minor downtime/interruptions for other project Jenkins systems as they
were updated.
Follow-up actions
*
Cleanup and update of the *FD.io* Jenkins VM.
*
Priority roll-out for other Jenkins to at least 1.56
*
Investigate better reporting of vulnerable Jenkins plugins
Timeline (PDT)
2:13am PDT: exploit is successfully performed on *FD.io* Jenkins via a
specially crafted URL containing the payload:
107.174.x.x - - [14/May/2019:09:13:26 +0000] "GET
/descriptorByName/org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript/checkScript?sandbox=True&value=public+class+x%7Bpublic+x%28%29%7Bnew+String%28%22[payload_omitted]%22.decodeHex%28%29%29.execute%28%29%7D%7D
HTTP/1.1" 200 6 "-" "python-requests/2.21.0"
The exploit caused Jenkins to crash, but the payload execution was
successful due to critical vulnerabilities in the “Script Security”
Jenkins plugin. The payload downloaded and executed the following shell
script:
https://paste.fedoraproject.org/paste/LjwXwoLyXnAI78kS0vhBBg
The script is not Jenkins-specific, so it does not attempt to modify any
content owned by Jenkins. Since the script runs as a non-privileged
user, most actions fail without any effect. The script does install a
cryptocoin miner (“cryptonight”) and puts itself into crontab for the
jenkins user in order to persist across reboots.
2:19am PDT: Monitoring recognizes that Jenkins is down and issues an
alert, which is received and acknowledged by C.Hoy Poy (sysops on call).
2:48am PDT: C.Hoy Poy identifies that there has been a security
incursion and shuts down the system per security first responder procedure.
2:56am PDT: The issue is escalated to K.Ryabitsev (Director of IT
Security), J.Conway (SysOps Team Lead).
4:00am PDT: The incursion is traced down to the Jenkins “Script
Security” plugin and payload is identified and analyzed. The team takes
a decision to clean up the affected system instead of reinstalling it,
since there is no indication that the payload succeeded to do anything
beyond installing the cryptocoin miner.
6:15am PDT: The system is cleaned up and all plugins are updated to
their latest security errata versions. Jenkins is brought back online
and all services restored.
~2:00pm PDT: All other Jenkins systems are analyzed for traces of the
same incursion and upgraded to the latest security errata. 2019-05-14
fd.io Jenkins security incident*
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13062): https://lists.fd.io/g/vpp-dev/message/13062
Mute This Topic: https://lists.fd.io/mt/31640510/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
