Lessons From Breweries and Security Teams: The Importance of Thinking Long-Term

In 1759, Arthur Guinness signed a 9,000-year lease on the disused St. James’s Gate brewery in Dublin.

The annual cost? £45.

Over the next two-and-a-half centuries, the company he founded became a global phenomenon that sells over 10 million glasses of stout beer every day. Thanks to its founder’s foresight, Guinness paid just £45 per year in rent — until the company bought out the brewery and surrounding land to expand operations.

Talk about an excellent long-term investment.

What Pentesting and Breweries Have in Common Speaking of long-term investments, brewing and pentesting have something in common: they both take time to get it right.

Most new breweries take years to settle on the perfect recipe. But once they have a winner, they stick with it because they don’t want to mess with the taste. That’s why many popular beers have been brewed in the same location — using the exact same recipe — for decades or even centuries.

When it comes to pentesting, the same principle applies. It can take a couple years to develop a robust pentest program and cement it into your security and engineering workflows. Once you have that ideal program in place, you can reap the benefits of your investment for years to come.

What is a Pentest Program?

While individual pentests are a point-in-time assessment of an asset’s security, a pentest program is a clearly defined series of pentests. Usually defined over several years — with testing completed periodically throughout — pentest programs aim to systematically find and fix vulnerabilities across a range of assets.

Pentest programs rely on the Pentest-as-a-Service (PtaaS) model — a platform-driven approach that utilizes a freelance talent pool where testers are selected for engagements based on their relevant expertise.

This approach has several benefits over one-off testing:

• High-risk vulnerabilities are more likely to be quickly identified and remediated.
• It’s much easier to ensure full asset coverage.
• Frequent testing stays on top of weekly or daily code updates.
• The availability of diverse testers via a PtaaS platform removes the need to rotate pentest vendors.

Most importantly, scheduling frequent pentests over a defined period enables the incremental improvement of security outcomes over time.

For a full rundown on pentest programs, download the Comprehensive Guide to Building a Pentest Program.

Multi-Year vs. Annual Pentesting

There’s no doubt planning pentests over a year-long timeline is a considerable improvement over ad-hoc testing. However, in the context of a broader security program, one year is a relatively short time to enact major changes. In our experience, it usually takes 1–2 years to fully develop and refine a pentest program and achieve optimal results. From there, most teams continue to see iterative improvements year-after-year.

For some organizations the process can take slightly longer, particularly if their pentest program is starting from scratch, or a multi-year program is needed to achieve full buy-in.

Of course, that’s not to say you have to wait years to see value from your pentest program. Compared to typical ad-hoc testing, a pentest program will deliver clear benefits within months. By learning from each engagement and refining the testing and remediation process, you can achieve ongoing improvements to security outcomes over the life of your program.

Where do these improvements come from? Simple:

More Time = More Data = A Better Understanding of Asset Security

The #1 benefit of a pentest program is the ability to improve continually. By conducting a postmortem after each pentest and engaging with pentesters and engineers over time, security teams can refine the process and embed continual testing into existing security and engineering workflows.

By analyzing the results of pentests over time, security and engineering teams can gain a deeper understanding of the types of vulnerabilities found. From there, changes to processes and technologies help to limit previously recurring classes of vulnerabilities in future code. After addressing this low-hanging fruit, pentesters will have more time to focus on finding deeper, more complex vulnerabilities in future tests.

It doesn’t stop there. Cobalt’s own multi-year pentest program has changed drastically since its inception. From the introduction of grey box pentesting to increased collaboration between engineers and pentesters, Cobalt’s results have improved incrementally over time — and continue to improve to this day.

“The more data we have, the better informed we are, which allows us to make better decisions on how to secure our application most effectively." — Eko Prasetya, VP Software Engineering at Carium

3 Things to Consider When Planning Your Multi-Year Pentest Program

1) Planning is essential

One of the main benefits of a multi-year pentest program is its predictability. By planning pentests over time, you can:

• Secure remediation resources in advance.
• Build communication, remediation, and post-pentest analysis into engineering workflows.
• Cement costs into your budget in a predictable cadence.

However, for all this to be possible, you must have an effective planning process in place. This process should involve thorough planning sessions ahead of program start and ongoing reviews to ensure the program runs smoothly and improves over time.

2) Align success metrics with engineering

Every engineering team has a process of tracking and resolving bugs. By aligning vulnerability tracking with your engineering team’s existing processes, you can ensure timely and effective remediation following each pentest.

However, it goes deeper than this. Engineering teams use established success metrics to keep operations on track. If you can align with (and expand) these metrics to support the pentest program, you can create a system that incentivizes engineers to treat vulnerabilities like any other bug.

Examples of metrics to consider include:

• Number of vulnerability classes
• Open findings by criticality
• Findings by status and criticality
• Findings by class
• Time to fix by criticality

As your pentest program evolves, each of these metrics should decrease. If they don’t, you’ll know where to focus on making improvements.

3) Align risk assessment with pentest results

Cyber risk management is a defining characteristic of modern security programs. To accurately calculate the risk associated with different threats, you must have a clear understanding of individual assets’ security — particularly those critical to your operations.

This is another area where multi-year pentest programs shine.

By carefully planning and tracking your program over time, you’ll gain a thorough understanding of where each asset is from a security standpoint. Not only will this improve your ability to assess cyber risk accurately, it will also help you track risk over time and make more informed security decisions.

The Only Thing that Really Matters

Ultimately, the end goal for any organization — whether it’s a brewery, a SaaS provider, or something else entirely — is to add value for customers. If an organization can’t achieve that consistently, it won’t be in business for very long.

And while it’s tempting to think of pentesting as an internal issue, the end goal is really to maximize value for customers by protecting their data and maintaining a strong customer experience.

So while planning your pentest program, always ask yourself: will this help us deliver (or protect) value for our customers?

If the answer is yes, it’s probably a good investment of your time and resources.

“We want to be able to give our customers the best solutions and by working with Cobalt we can leverage their cybersecurity knowledge and skills to more effectively secure our applications and efficiently anticipate emergent security threats.” — Flower Workman, Senior Director of Engineering Delivery at Higher Logic