skip navigation
skip mega-menu

How Safe Are Your Secrets? The Importance of Data Security and Risk Mitigation

It is now widely accepted that data is a vital business asset, and with security features developing at a rapid rate, the importance of keeping up-to-speed with the best tools to use to protect an organisation’s data should not be understated.

Here, Informed Solutions’ Lead Consultant James Cruddas considers the importance of data security and risk mitigation, and how InformedENGINEERING© is cultivating an environment where developers can deliver solutions that make our world safer, smarter, healthier, greener, and more secure.

Importance of Data Security

Ask an organisation to tell you their most valued asset and I can almost guarantee that, shortly after its people, data will be one of the most common responses you hear.

The ability of a business to convert data into actionable insight – regardless of sector, size, age, or the number of customers you serve – continues to be ever more important in maintaining competitive advantage, increasing customer satisfaction, and driving operational efficiency gains.

This recognition, coupled with the legal need to safeguard personal information, has helped to remind people of the importance of data security. We see this evidenced by cloud providers enabling more secure configurations “by design”. They are also offering ever more sophisticated security features at more affordable price points than ever before. Take for example AWS, which by April 2023 will be enabling block public access settings by default on S3 buckets.

Risk and Mitigation

Data has become increasingly acknowledged as “business gold” and the digital vaults in which it is stored continue to grow stronger.

However, a vault built from titanium will open just as easily as one made of tin if you have its key. In data terms, when working with cloud technologies, physical keys are replaced with digital access tokens (be it for AWS, Azure, Google Cloud, or just about anything else). If you expose your access tokens you run a significant risk of exposing your data.

I choose the example of access tokens intentionally, although the same applies to several other forms of secret. Anyone that has read the marketing materials for technologies such as Azure KeyVault, or AWS Secrets Manager might quickly jump to the conclusion they solve just about any credential handling problem you could imagine. Whilst yes, their use is best-practice, and they avoid credentials needing to be stored in plaintext, as they are instead fetched on-demand over REST, you still need to authenticate against these service’s APIs. And to do that you will need an access token.

Multi-factor Authentication (MFA) is an excellent choice for mitigating the threat of compromise but concerningly, studies as recent as November 2022 suggest a mere 26% of enterprise cloud users from the sampled audience are taking advantage of the security benefits it brings. With such a low adoption rate for a relatively simple “switch-on” security feature, more intricate solutions such as carefully crafted identity policies or making use of short-lived tokens, combined with role-based (or attribute-based) controls, is probably even more scarce.

Back in 2019, a study at NCSU by Meli et al. highlighted that “thousands of new, unique secrets are leaked every day” in public GitHub repositories. If you’re one of the 74% of enterprise cloud users not implementing MFA as a secondary layer of protection – this poses a very real threat to your data security. With engineers being aware of how sensitive these credentials can be – how are they still ending up in source code?

The shortest possible answer I’ve found to this question is just three words long… “git add dot”. Whilst I strongly discourage use of this command, as even the best .gitignore files can’t catch everything, you will struggle to find an engineer who has not at some point used this to reduce the time needed to prepare commit contents. The command recursively adds all changes made in the current directory and below in just one fell swoop. This includes newly created files, updates, removals, and everything in-between. Compounding risks to its use, if you omit the “–untracked-files” flag when running a “git status” check, you’ll just be shown a printout of the folder names that are now destined for source control – not details of the individual files inside them. That “temp.txt” file an engineer put their access keys in whilst working on a series of complex CLI commands might never have been intended for source control… but “git add . [dot]” doesn’t care!

How Can Development Teams Protect Themselves?

I’m pleased to say several advances have been made in this space over recent years. In December 2022, GitHub made its secret scanning feature that previously sat behind an enterprise subscription paywall free for all public repositories. That said, whilst this early Christmas present might have brought a smile to the face of any security-conscious developer, an enterprise subscription is still needed if you want to keep your private repositories secret-free. I would also argue that, as generous this might be, only finding out your credentials have been exposed once they are out in the wild is a little too late! To underscore this argument; research published back in late 2020 suggests it can take as little as one minute for hackers to begin using exposed credentials for exploit attempts.

Fortunately, there are several tools that can help with stopping credentials going into source code in the first place. Snyk for example, when using its local CLI, supports, as part of its wider ecosystem, the ability to proactively check for credentials in source code from your local machine. Alongside commercial offerings, there are open-source alternatives such as GitLeaks, GittyLeaks, git-secrets, Trufflehog, and Whispers just to name a few.

At Informed, we use open-source a lot and have made use of TruffleHog OSS on several recent projects as part of our toolchain alongside other complementary provider-specific tools. After reaching out to members of our Service Engineering Community of Practice to gather feedback from our engineers (it was important that everyone be involved in the selection process as they will be the ones using tools), we felt this choice was the best fit for our use-case. At the risk of sounding like a footnote to a financial or legal newsletter – please be aware this does not represent formal advice – it’s simply what works for us!

In April 2022 TruffleHog overhauled its implementation under a new V3 re-design. Alongside other improvements, looking at the tool’s source code, it would seem the maintainers have taken the regex recommendations made by Meli et al. to heart. Combined with its Docker container support, and there being no need for license keys or server-side hub, we feel it provides a simple, portable solution.

Owing in large part to its container support, the tool can run both locally and in serverless CI contexts such as GitHub Actions. The validation feature which tests suspected credentials against 700+ provider types to see if they are genuine, rather than something that just looks like a credential, also goes a long way in preventing “alarm fatigue” by filtering out false positives.

Finally – by binding the tool’s CLI to git pre-commit hooks and instructing a scan to run each time files are staged it becomes straightforward to deploy a multi-layered credential scanning solution that can run on both engineer’s local machines and CI pipelines. This prevents secrets from entering source control in the first place, and double-checks on commits being pushed using CI. It’s always nice to have a fallback!

I’m not saying these types of automated tools are a replacement for understanding and practicing secure software development but as an engineer myself – it’s nice to know there’s a safety net there just in case! As the saying goes “I’d rather have it and not need it, than need it and not have it”.

I hope this article proves useful – whether it inspires you to improve your own development tooling and practices or simply filled 15 minutes of your lunchtime knowledge-quest with some food for thought. If you’re a like-minded individual who enjoys applying automation to help make engineering team’s lives more comfortable – feel free to connect with me on LinkedIn and let me know your thoughts on the topic.

Our team is also always open to discussing potential opportunities of a career at Informed if these are the types of things that get you out of bed in the morning!

Explore jobs at Informed Solutions

Lead Developers

About UsInformed Solutions delivers digital data and technology solutions that help make the world a smarter, safer, greener, and healthier place.Founded in 1992, we are a successful, growing International digital transformation consultancy. We deliver multi-Queen’s Award for innovation winning platforms and services that support large-scale digital transformation. Our digital, data and technology solutions are used by globally recognised public and private sector brands operating in a variety of sectors including Civil Defence, Healthcare, Sustainable Environment and Land Asset Management, and Digital Democracy.Our purpose is to create economic and social value by helping to build a more inclusive, fair, and safe society through ethical use of technology and data and investing in digital skills.The multi-Queen’s Award-winning data and AI-powered platforms and services we develop support large-scale transformation of services that make a difference.  They are used by millions of people every day in a connected, converged world.The OpportunityMake a difference and advance your career by helping deliver some of the UK’s most important #tech4good projects.You’ll join a talented team of dynamic and driven professional problem solvers; creative thinkers and solutions builders who thrive on helping clients meet the most exciting digital transformation challenges.At a certified Great Place to Work® you’ll experience a dynamic and nurturing environment that rewards initiative and flexibility and enjoy a career path tailored to your own aspirations.CultureWe are proud to nurture a workplace culture that is diverse, inclusive, rewarding, and egalitarian.We strive to live up to our values of Innovation, Excellence, and Integrity by thinking about things differently, always doing our best, and acting in good faith at all times.We’re a team of passionate problem solvers. We take pride in helping our clients accelerate and de-risk digital business change so that we can collaborate and co-design world class digital services that solve complex business and safety critical problems, particularly where place, location or geography are important. Our workplace culture reflects how we go about our work, the type of work that we choose to do, and our commitment and contribution to the sustainable social, environmental, and economic development aims of the communities that we are part of.  We focus both on technical skills and equally importantly, on the cultural fit of prospective new colleagues. Our success relies on fostering an environment where creativity and collaboration produces great outcomes for our people, our clients, and our partners.Why We Work at InformedLearn more about life at Informed Solutions by going to Informed.com/Careers

Informed Solutions
Live Service Manager

About UsInformed Solutions delivers digital data and technology solutions that help make the world a smarter, safer, greener, and healthier place.Founded in 1992, we are a successful, growing International digital transformation consultancy. We deliver multi-Queen’s Award for innovation winning platforms and services that support large-scale digital transformation. Our digital, data and technology solutions are used by globally recognised public and private sector brands operating in a variety of sectors including Civil Defence, Healthcare, Sustainable Environment and Land Asset Management, and Digital Democracy.Our purpose is to create economic and social value by helping to build a more inclusive, fair, and safe society through ethical use of technology and data and investing in digital skills.The multi-Queen’s Award-winning data and AI-powered platforms and services we develop support large-scale transformation of services that make a difference.  They are used by millions of people every day in a connected, converged world.The OpportunityMake a difference and advance your career by helping deliver some of the UK’s most important #tech4good projects.You’ll join a talented team of dynamic and driven professional problem solvers; creative thinkers and solutions builders who thrive on helping clients meet the most exciting digital transformation challenges.At a certified Great Place to Work® you’ll experience a dynamic and nurturing environment that rewards initiative and flexibility and enjoy a career path tailored to your own aspirations.CultureWe are proud to nurture a workplace culture that is diverse, inclusive, rewarding, and egalitarian.We strive to live up to our values of Innovation, Excellence, and Integrity by thinking about things differently, always doing our best, and acting in good faith at all times.We’re a team of passionate problem solvers. We take pride in helping our clients accelerate and de-risk digital business change so that we can collaborate and co-design world class digital services that solve complex business and safety critical problems, particularly where place, location or geography are important. Our workplace culture reflects how we go about our work, the type of work that we choose to do, and our commitment and contribution to the sustainable social, environmental, and economic development aims of the communities that we are part of.  We focus both on technical skills and equally importantly, on the cultural fit of prospective new colleagues. Our success relies on fostering an environment where creativity and collaboration produces great outcomes for our people, our clients, and our partners.Why We Work at InformedLearn more about life at Informed Solutions by going to Informed.com/Careers

Informed Solutions
User Researcher

About UsInformed Solutions delivers digital data and technology solutions that help make the world a smarter, safer, greener, and healthier place.Founded in 1992, we are a successful, growing International digital transformation consultancy. We deliver multi-Queen’s Award for innovation winning platforms and services that support large-scale digital transformation. Our digital, data and technology solutions are used by globally recognised public and private sector brands operating in a variety of sectors including Civil Defence, Healthcare, Sustainable Environment and Land Asset Management, and Digital Democracy.Our purpose is to create economic and social value by helping to build a more inclusive, fair, and safe society through ethical use of technology and data and investing in digital skills.The multi-Queen’s Award-winning data and AI-powered platforms and services we develop support large-scale transformation of services that make a difference.  They are used by millions of people every day in a connected, converged world.The OpportunityMake a difference and advance your career by helping deliver some of the UK’s most important #tech4good projects.You’ll join a talented team of dynamic and driven professional problem solvers; creative thinkers and solutions builders who thrive on helping clients meet the most exciting digital transformation challenges.At a certified Great Place to Work® you’ll experience a dynamic and nurturing environment that rewards initiative and flexibility and enjoy a career path tailored to your own aspirations.CultureWe are proud to nurture a workplace culture that is diverse, inclusive, rewarding, and egalitarian.We strive to live up to our values of Innovation, Excellence, and Integrity by thinking about things differently, always doing our best, and acting in good faith at all times.We’re a team of passionate problem solvers. We take pride in helping our clients accelerate and de-risk digital business change so that we can collaborate and co-design world class digital services that solve complex business and safety critical problems, particularly where place, location or geography are important. Our workplace culture reflects how we go about our work, the type of work that we choose to do, and our commitment and contribution to the sustainable social, environmental, and economic development aims of the communities that we are part of.  We focus both on technical skills and equally importantly, on the cultural fit of prospective new colleagues. Our success relies on fostering an environment where creativity and collaboration produces great outcomes for our people, our clients, and our partners.Why We Work at InformedLearn more about life at Informed Solutions by going to Informed.com/Careers

Informed Solutions
Service Designer

About UsInformed Solutions delivers digital data and technology solutions that help make the world a smarter, safer, greener, and healthier place.Founded in 1992, we are a successful, growing International digital transformation consultancy. We deliver multi-Queen’s Award for innovation winning platforms and services that support large-scale digital transformation. Our digital, data and technology solutions are used by globally recognised public and private sector brands operating in a variety of sectors including Civil Defence, Healthcare, Sustainable Environment and Land Asset Management, and Digital Democracy.Our purpose is to create economic and social value by helping to build a more inclusive, fair, and safe society through ethical use of technology and data and investing in digital skills.The multi-Queen’s Award-winning data and AI-powered platforms and services we develop support large-scale transformation of services that make a difference.  They are used by millions of people every day in a connected, converged world.The OpportunityMake a difference and advance your career by helping deliver some of the UK’s most important #tech4good projects.You’ll join a talented team of dynamic and driven professional problem solvers; creative thinkers and solutions builders who thrive on helping clients meet the most exciting digital transformation challenges.At a certified Great Place to Work® you’ll experience a dynamic and nurturing environment that rewards initiative and flexibility and enjoy a career path tailored to your own aspirations.CultureWe are proud to nurture a workplace culture that is diverse, inclusive, rewarding, and egalitarian.We strive to live up to our values of Innovation, Excellence, and Integrity by thinking about things differently, always doing our best, and acting in good faith at all times.We’re a team of passionate problem solvers. We take pride in helping our clients accelerate and de-risk digital business change so that we can collaborate and co-design world class digital services that solve complex business and safety critical problems, particularly where place, location or geography are important. Our workplace culture reflects how we go about our work, the type of work that we choose to do, and our commitment and contribution to the sustainable social, environmental, and economic development aims of the communities that we are part of.  We focus both on technical skills and equally importantly, on the cultural fit of prospective new colleagues. Our success relies on fostering an environment where creativity and collaboration produces great outcomes for our people, our clients, and our partners.Why We Work at InformedLearn more about life at Informed Solutions by going to Informed.com/Careers

Informed Solutions
Account Manager - Healthcare Sector

About UsInformed Solutions delivers digital data and technology solutions that help make the world a smarter, safer, greener, and healthier place.Founded in 1992, we are a successful, growing International digital transformation consultancy. We deliver multi-Queen’s Award for innovation winning platforms and services that support large-scale digital transformation. Our digital, data and technology solutions are used by globally recognised public and private sector brands operating in a variety of sectors including Civil Defence, Healthcare, Sustainable Environment and Land Asset Management, and Digital Democracy.Our purpose is to create economic and social value by helping to build a more inclusive, fair, and safe society through ethical use of technology and data and investing in digital skills.The multi-Queen’s Award-winning data and AI-powered platforms and services we develop support large-scale transformation of services that make a difference.  They are used by millions of people every day in a connected, converged world.The OpportunityMake a difference and advance your career by helping deliver some of the UK’s most important #tech4good projects.You’ll join a talented team of dynamic and driven professional problem solvers; creative thinkers and solutions builders who thrive on helping clients meet the most exciting digital transformation challenges.At a certified Great Place to Work® you’ll experience a dynamic and nurturing environment that rewards initiative and flexibility and enjoy a career path tailored to your own aspirations.CultureWe are proud to nurture a workplace culture that is diverse, inclusive, rewarding, and egalitarian.We strive to live up to our values of Innovation, Excellence, and Integrity by thinking about things differently, always doing our best, and acting in good faith at all times.We’re a team of passionate problem solvers. We take pride in helping our clients accelerate and de-risk digital business change so that we can collaborate and co-design world class digital services that solve complex business and safety critical problems, particularly where place, location or geography are important. Our workplace culture reflects how we go about our work, the type of work that we choose to do, and our commitment and contribution to the sustainable social, environmental, and economic development aims of the communities that we are part of.  We focus both on technical skills and equally importantly, on the cultural fit of prospective new colleagues. Our success relies on fostering an environment where creativity and collaboration produces great outcomes for our people, our clients, and our partners.Why We Work at InformedLearn more about life at Informed Solutions by going to Informed.com/Careers

Informed Solutions

Subscribe to our newsletter

Sign up here