Pentester Feedback to Product Updates

One of the opportunities Core members has while working for Cobalt is to give direct feedback of the Cobalt platform to the product team. There is a designated Slack channel where pentesters can provide suggestions and feedback on things they encounter during engagements. I talked with the Director of Product, Mark Hamill, about how his team handles these requests and their goals. 

The Process:

1. Pentester provides feedback or a suggestion for something in the Cobalt Platform
2. The product or Design team responds to the request acknowledging the feedback
3. The team asks some questions and tries to understand the problem that needs solving
4. The team tries to align with known objectives or respond if this is something that is already being considered or being worked on
5. The team writes a user story to capture the ask

How the team decides what to work on: 

"This is the tricky part," Mark Hamill said. "There are a few factors to consider."

Impact:

How many users want this?  How much of a game-changer is it?

Effort: 

How big is this feature?  Can we realistically tackle it?  Is it a quick win we can review and deliver in a day?

Alignment: 

How well does it fit with the company and team objectives we have set for ourselves (bearing in mind one of these is pentester happiness!)

Success Stories

Delete Draft Findings

Problem: Creating findings is the heart of pentester value for Cobalt’s clients. The creative process isn’t linear, though, and as is often the case with any kind of data add, there can be reasons to decide not to submit an entry after it has been started. Cobalt findings initially persisted forever once they had been added to the platform, even if they were only a draft. If the author decided the finding was not worthy of customer concern/review - they still have to flag and process the findings. They won’t be visible on reports but will be available in the system.

Solution: Pentesters now have the ability to delete draft findings during the editing process, allowing for information to be saved, but also removed prior to the customer reporting phase.

Clone Findings within a Pentest

Problem: Findings come and many shapes and sizes, but there are a limited number of types/categories. When writing up a finding, a previous finding in the same (or a different) pentest could easily be referenced to come up with the bulk of the text, but copying and pasting individual fields can be time-consuming.

Solution: Allow pentesters to copy/clone previous findings with the click of a button, saving time and energy in the editing process. Any irrelevant data could be stripped out automatically (e.g., Endpoint, HTTP Request, and prerequisite sections).

Why is this feedback-to-change loop important? 

"To build a fantastic product experience, you must understand your users.  This means getting to know their motivations, pain points, and everything in between.  Our community is incredibly engaged and highly technical in nature.  This is as tight as a feedback loop with a customer as I have ever experienced as a product manager, and it’s thrilling to see minor tweaks we launch have an immediate impact.

I not only want this feedback loop to continue but to evolve over time.  In the short time I have been in Cobalt, I have seen the engagement ramp up - and the more we announce (and more questions we ask), the more comments and suggestions we get from the Core.  Our product and design team is ramping up on a shadowing program to get even closer to our end users and see them in action to understand how they feel about the platform - both the good and the bad!"

Feeback from Core Pentesters

Core Pentester Arif is very vocal in our feedback channel and has been happy with how the product team responds to his and others' messages. 

"The product team always listens to our feedback, and the feedback is taken seriously, which is great," he said. "The issues are fixed efficiently based on the appropriate priority." 

Marcin Ogorzelski initially introduced the idea of cloning a vulnerability and was happy to see it implemented. 

"The application-feedback channel is a great place for anyone on the team to share their ideas for improving the Cobalt platform not only in terms of functionality but also in terms of improving processes regarding penetration testing," he said.  "Most importantly, it's a channel where everyone can feel heard, and every idea is taken into account and considered. A significant advantage of the channel is that not only members of the development team see the suggestions but also pentesters. This allows every idea to be immediately voted on, creatively critiqued, or simply improved through comments from others. Many of the functionalities proposed on the application-feedback channel are implemented in the platform, which gives great satisfaction and motivation to share more ideas. It's nice to add your contribution and impact the platform."