DevSecOps What Why and How?
A presentation at Black Hat USA in in Las Vegas, NV, USA by Anant Shrivastava
DevSecOps What Why and How? Anant Shrivastava @anantshri NotSoSecure Global Services
About: Anant Shrivastava Director NotSoSecure Global Services Sysadmin / Development / Security Trainer / Speaker: BlackHat, Nullcon, RootConf, RuxCon, IPExpo, C0c0n Project Owner: Android Tamer, Code Vigilant Contributor: null, G4H, OWASP and more https://anantshri.info (@anantshri on social platforms) 2
Agenda ● What is DevSecOps? ● Why do we need DevSecOps? ● How do we do DevSecOps? ● Integrate Security in DevOps Pipeline ● Tools of Trade ● Sample Implementation (On Prem and Cloud Native) ● Case Studies 3
Disclaimer ● I will be listing a lot of tools, It’s not an exhaustive list ● I don’t endorse or recommend any specific tool / vendor ● Every environment is different: Test and validate before implementing any ideas 4
What is DevSecOps? Effort to strive for “Secure by Default” ● Integrate Security via tools ● Create Security as Code culture ● Promote cross skilling 5
Why do we need DevSecOps? ● DevOps moves at rapid pace, traditional security just can’t keep up ● DevSecOps makes it easier to manage rapid pace of development & large scale secure deployments ● DevSecOps allows for much smoother scaling of process ● Security as part of process is the only way to ensure safety 6
Shifting Left saves cost & time Developer Source Code Repository Build CI/CD Server Staging/ QA Production Monitoring Suite of Security Test 7
Shifting Left saves cost & time Developer Source Code Repository Build Automated Source Code Review CI/CD Server Staging/QA Production Monitoring Suite of Security Test 1 SQL Injection Fewer Man Day Effort No New Deployments 8
How do we do DevSecOps? ● DevSecOps is Automation + Cultural Changes ● Integrate security tools into your DevOps Pipeline ● Enable cultural changes to embrace DevSecOps 9
Injecting Sec in DevOps Artifact Repository Build Artifacts versioning against code commits CI/CD Server Developer Pre-Commit Hooks IDE Plugins Code Repository Secrets Management Pre-Build Static Application Security Testing(SAST) Source Composition Analysis (SCA) Post-Build Dynamic Application Security Testing(DAST) QA/Staging Manual Web Application Pentesting Business Logic Flaws Production Security in Iaac Compliance as Code Alerting & Monitoring Vulnerability Management 10
DevOps —-> DevSecOps DevOps Pipeline DevSecOps Pipeline
Pre-Commit Hooks ● Sensitive information such as the access keys, access tokens, SSH keys etc. are often erroneously leaked due to accidental git commits ● Pre-commit hooks can be installed on developer’s workstations to avoid the same ● Work on pure Regex-based approach for filtering sensitive data ● If developers want they can circumvent this step hence use it like a defense in depth but don’t fully rely on it 12
IDE Security Plugin ● IDE Plugin’s provide quick actionable pointer to developers ● It is useful to stop silly security blunders ● Work on pure Regex-based approach ● If developers want they can circumvent this step hence use it like a defense in depth but don’t fully rely on it 13
Secrets Management ● Often credentials are stored in config files ● Leakage can result in abuse scenario ● Secrets Management allows you to tokenize the information 14
Software Composition Analysis ● We don’t write software’s, we build on frameworks ● Biggest portion of software is now third party libraries ● Major languages provide module managements ● PIP, NPM, Gems, go get, perl cpan, php packager and more ● Software Composition Analysis performs checks to identify vulnerable/outdated 3rd party libraries 15
Static Analysis Security Testing ● White-box security testing using automated tools ● Useful for weeding out low-hanging fruits like SQL Injection, Cross-Site Scripting, insecure libraries etc ● Tool by default configured with generic setting, needs manual oversight for managing false-positives 16
Dynamic Analysis Security Testing ● Black/Grey-box security testing using automated tools ● SAST may not get full picture without application deployment ● DAST will help in picking out deployment specific issues ● Results from DAST and SAST can be compared to weed out false-positives ● Tools may need prior set of configuration settings to give good results 17
Security in Infrastructure as Code ● Infrastructure as a code allows you to document and version control the infra ● It also allows you to perform audit on the infrastructure ● Docker / K8s infra relies on base images ● Environment is as secure as the base image ● Base images need to be minimal in nature and need to be assessed to identify inherited vulnerabilities 18
Compliance as Code ● Compliance could be industry standard (PCI DSS, HIPAA, SOX) or org specific ● Compliance is essentially a set of rules and hence can be converted into written test cases ● Having written code format this can again be version controlled 19
Vulnerability Management ● All the tools discussed above result in report fatigue ● Every tool has a different style of presentation ● A central dashboard is required to normalize the data ● Vulnerability Management System can then be integrated to bug tracking systems to allow devs to work on items 20
Alerting and Monitoring ● Monitoring is needed for two end goals ● Understand if our security controls are effective ● What and where we need to improve ● To test Security control effectiveness: ● ● ● ● When did an attack occur Was it blocked or not What level of access was achieved what data was bought in and bought out 21
Asset Monitoring ● With recent advancements assets now should include anything and everything where organization data resides ● With rapid development & provisioning the asset inventory can’t be a static inventory ● We need to monitor the assets constantly both on premise and Cloud Reference: https://redhuntlabs.com/blog/redifining-assets-a-modern-perspective.html 22
Sample Implementation - Java A simplistic flow of DevSecOps Pipeline incorporating various stages https://www.youtube.com/watch?v=7ILrzYLpr14
Tools of The Trade ThreatSpec Threat Modelling Tools Microsoft Threat Modeling Tool truffleHog Pre-Commit Hooks Software Composition Analysis Git Hound Retire.js Static Analysis Security Testing (SAST) IDE Plugins Secret Management CAT.net Keywhiz Preference given to open-source tools; we don’t endorse any tool 24
Tools of The Trade Vulnerability Management Dynamic Analysis Security Testing (DAST) Security in Infrastructure as Code Compliance as Code Docker Bench for Security WAF Preference given to open-source tools; we don’t endorse any tool 25
To be or Not to Be in Pipeline ● API / command line access ● Execution start to final output should be 15 minutes max ● Tools should be Containerizable / scriptable ● Minimal licensing limitations (parallel scans or threads) ● Output format parsable / machine readable (no to stdout, yes to json / xml) ● Configurable to counter false negatives / false positives 26
Pipeline Optimization ● Pipeline to be tweaked based on Milestone (Initiative/Epic/Story) ● Remember initial onboarding is tedious ● Ensure dataset dependent tool get frequent data refresh ● Sample optimization ● Only CSS Changes: no need for SCA ● Only pom.xml or gradle changes: no need of SAST ● If Infra as code has zero changes skip or fast track infra scan ● Ensure to run full (non optimized) pipeline periodically
Does Programming Language Matter ● Different programming languages need different tools for static analysis and software composition analysis ● Some tools support multiple languages like sonarqube ● Others are focused on one language 28
Language Specific Tools Languages Software Composition Analysis JAVA Source Code Static Analysis graudit ClearlyDefined graudit PHP graudit Python .NET Brakeman Ruby/Rails Node JS DotNet Security Guard DotNET Retire SafeNuGet ClearlyDefined Preference given to open-source tools; we don’t endorse any tool graudit npm-check NodeJsScan 29
DevSecOps Lab - Ruby
DevSecOps Lab - PHP
DevSecOps Lab - Python
DevSecOps Lab - NodeJS
What about Cloud ● The Threat Landscape changes - Identity and Access Management - Asset Inventory - Billing ● Infrastructure as Code allows quick audit / linting ● Focus more on: - Security groups - Permissions to resources - Rogue / shadow admins - Forgotten resources (compromises / billing) 34
Cloud Native Approach to Security ● Different Service Providers Approach Security Differently ● All of them provide some of the ingredient in-house ● Irrespective of Cloud provider some tools will still need to be sourced ● Static Code Analysis Tool ● Dynamic Code Analysis Tool ● Software Composition Analysis ● Vulnerability Management Tool 35
AWS Cloud Native DevSecOps https://youtu.be/i38-YQsnqfw 36
Cloud Native Dev[Sec]Ops Conventional Infra AWS Azure GCP Source Code Management Bitbucket, Github, Gitlab etc.. AWS CloudCommit Azure Repos Cloud Source Repositories Infrastructure As a Code Chef, Puppet, Ansible more.. Amazon CloudFormation Azure DevTest Labs Cloud Code CI/CD Server Jenkins, Bamboo, Gitlab, Travis CI, Circleci more AWS CodeBuild AWS CodeDeploy AWS CodePipeline Azure Pipelines, Azure Test Plans Cloud Build, Tekton Artifactory Repository jFrog Artifactory, Sonatype Nexus, more.. Amazon S3 Azure Artifacts Cloud Firestore Stg/Prod Servers VMWare, On-premises servers EC2 ECS (Elastic Containers) EKS (Elastic Kubernetes) Virtual Machines, Azure Lab Services, Azure Kubernetes Service (AKS) Compute Engine, App Engine, Shielded VMs Monitoring & Alert Nagios, Graphite, Grafana AWS CloudWatch Azure Monitor, Network Watcher Access Transparency Firewall Modsecurity AWS Firewall Manager, AWS WAF Azure Firewall Application Gateway DLP MyDLP, OpenDLP Amazon Macie Azure Information Protection Cloud Data Loss Prevention Threat Detection Snort, Kismet Amazon GuardDuty Azure Advanced Threat Protection Event Threat Detection (beta) Vulnerability Scanning OpenVAS, Nessus Amazon Inspector Azure Security Center Cloud Security Scanner Secrets Management Hashicorp Vault, Docker Secrets AWS Secrets Manager Azure Key Vault Secrets management 37
Cultural Aspect ● Automation alone will not solve the problems ● Encourage security mindset especially if outside sec team ● Cultivate/Identify common goals for greater good ● Build allies (security champions) in company ● Focus on collaboration and inclusive culture ● Avoid Blame Game Security team should try to eliminate the need of dedicated security team 38
Security Champion ● Bridge between Dev, Sec and Ops teams ● Single Person per team ● Everyone provided with similar cross skilling opportunities ● Incentivize other teams to collaborate with Sec team ○ Internal Bug bounties ○ Sponsor Interactions (Parties / get-togethers) ○ Sponsor cross skilling trainings for other teams 39
Security Enablers People Process Technology • Build relationships between teams, don’t isolate • Involve security from get-go (design or ideation phase) • Templatize scripts/tools per language/platform • Identify, nurture security conscious individuals • Fix by priority, don’t attempt to fix it all • • Security Controls must be programmable and automated wherever possible Adopt security to devops flow don’t expect others to adopt security • Keep an eye out for simpler and better options and be pragmatic to test and use new tools • • Empower Dev / ops to deliver better and faster and secure, instead of blocking. Focus on solutions instead of blaming • DevSecOps Feedback process must be smooth and governed 40
Generic Case Study 41
Case Studies – Fannie Mae https://www.slideshare.net/DevSecOpsDays/fannie-mae-devsecops-journey-with-chitra-elango-and-john-willis 42
Case Studies – Fannie Mae https://www.slideshare.net/DevSecOpsDays/fannie-mae-devsecops-journey-with-chitra-elango-and-john-willis 43
Case Studies – ABN Amro https://www.slideshare.net/derweeksglobal/abn-amro-devsecops-journey 44
Case Studies – ABN Amro https://www.slideshare.net/derweeksglobal/abn-amro-devsecops-journey 45
Negative Case Studies n o i t a r u g i f s t sse n o c s i M A d u o l C Prevention: Continuous monitoring and review of cloud assets and config 46
Is it Enough? ● Rite of passage by periodic pen test and continuous bug bounty ● It’s not just important to get feedback but to also action on them ● Risk Acceptance Documentation should be the worst case scenario not your first bet 47
Who Watches the Watcher ● Did we secure the security controls ● DevSecOops: If attacker controls security tools / build chain It has limitless power ● Ensure the same practice is followed back again for these tools ● Security role doesn’t means you get to circumvent the rules ● Follow basic security hygiene we always keep talking about ● Secure configuration ● Patching Policy 48
References • https://www.blackhat.com/docs/us-17/thursday/us-17-Lackey-Practical%20Tips-for-Defendin g-Web-Applications-in-the-Age-of-DevOps.pdf • https://www.sonatype.com/hubfs/2018%20State%20of%20the%20Software%20Supply%20 Chain%20Report.pdf • https://snyk.io/opensourcesecurity-2019/ • https://scaling-threat-detection.awssecworkshops.com/ • https://www.veracode.com/state-of-software-security-report 49
Key Takeaways ● Security is everyone responsibility ● Embrace security as an integral part of the process, use feedback to refine the process ● DevSecOps is not a one size fit all: your mileage will vary @anantshri anant@notsosecure.com 50
Security is often added towards the end, in a typical DevOps cycle through a manual/automated review. However, with DevSecOps, security can be injected at every stage of a DevOps pipeline in an automated fashion. Having a DevSecOps pipeline enables an organization to:
- Create a security culture amongst the already integrated “DevOps” team.
- Find and fix security bugs as early as possible in the SDLC .
- Promote the philosophy “security is everyone’s problem” by creating Security champions within the organization.
- Integrate all security software centrally and utilize the results more effectively.
- Measure and shrink the attack surface.
In this talk, we shall focus on how a DevOps pipeline can easily be metamorphosed into a DevSecOps and the benefits which can be achieved with this transformation. The talk (assisted with various demos) will focus on developing a DevSecOps pipeline using free/open-source tools in various deployment platforms, i.e. on-premise, cloud native and hybrid scenarios. We will then dive into cultural aspects of DevSecOps and the changes needed to get tangible benefits. The talk will also present various case studies on how critical bugs and security breaches affecting popular software and applications could have been prevented using a simple DevSecOps approach.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
Buzz and feedback
Here’s what was said about this presentation on social media.
