Skip to main content

Append keys to existing Secret in AWS Secrets Manager

The AWS update-secret operation for Secrets Manager replaces all keys of a secret with the new value provided in the --secret-string.

But sometimes we want to add a few extra keys, without replacing values already present in a secret.

In this post we show how to use bash to add keys to a secret without replacing existing values.

What you need#

  • jq installed

1. Prepare the list of secrets you want to add the new keys#

This command generates a file named all-secrets. Run the command and update the file to reflect the list of secrets you want to add the new keys.

# Create `all-secrets` file
aws secretsmanager list-secrets | jq .SecretList | jq '.[]' | jq -r .ARN > all-secrets

Keep only the the secrets you want to add the new keys in the generated all-secrets file

2. Store the new keys in a json file named new-keys.json#

echo '
{
"AWS_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE"
"AWS_SECRET_ACCESS_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}' > new-keys.json

3. Run the script#

The script uses the all-secrets and new-keys.json files created in steps 1 and 2.

while read line; do
aws secretsmanager get-secret-value \
--secret-id $line | \
jq -c '.SecretString | fromjson' > current-keys.json
jq -s '.[0] * .[1]' current-keys.json new-keys.json > merge.json
aws secretsmanager update-secret --secret-id $line --secret-string file://merge.json > updated
done <all-secrets

For each line in the all-secrets file, the script:#

  • Gets current secret value and save to current-keys.json file
  • Merges current-keys.soj and new-keys.json into merge.json
  • Updates the secret value with the merge.json file contents as the secret-string

Runops Releases #21

Runops REPL!#

No more typing the same boilerplate when running Tasks. Targets without reviews return the logs of a Task right the way. We added a new command to the CLI:聽runops tasks repl聽 -, where you can start an interactive session with your Target. In the REPL session, you only type the script option and get the results. It's simple. The REPL also supports multiline scripts. Here is a preview of how it works:

You can read more here:聽https://runops.io/docs/user-guides/REPL

CSV output#

You can now configure different output formats for Mysql and Postgres database Targets. These types show tabular results by default, great for quick consumption in Slack or the CLI. However, for more advanced analysis, it's best to be able to open the results in a spreadsheet. So you can use the configuration聽FIELD_SEPARATOR聽to set different separators for the output. For instance: setting the separator to: ","聽- will generate a beautiful CSV ready to be open in Excel or other spreadsheet apps.

Custom type for Templates#

You can set the type of the Task for a Template using the file name. Runops will try to match the extension of the file to a valid type. For example, the file "my-query.MySQL"聽as a Template creates a Task of type聽MySQL聽even if the Target is of type "AWS"聽or "python". This is a great way to simplify Templates execution for Targets that support multiple integration types. Also, with the type attached to the Template, you don't have to set it when running a Template from Slack or the CLI.

New integrations#

Hashicorp Vault You can now use Runops to interact with Hashicorp Vault. This is great for automating one-off tasks such as creating database users for new applications. Using a Template, you enable everyone to run sensitive operations in Vault that only executes after review. So, instead of doing everything yourself, you become a reviewer, taking way less of your time while still keeping security and compliance in place.

Kubernetes + Rails Use the k8s-rails type to enable access to the Rails console of applications running on Kubernetes. Then, using the Runops REPL, you authenticate with Google and get an experience similar to the Rails console. Zero hassle and risks of distributing SSH keys, Kubernetes credentials, VPNs, and other things required to get a shell running side-by-side with your production application.

Version checker in the CLI#

You will get a message when a new version of the CLI is available. This is because the CLI checks the latest version in the most used commands. The check happens in parallel, so there is no impact on the execution latency of your commands. The message stops showing up after you update the CLI.

PII data redact configuration#

You can now disable the automatic redacting of PII data for specific Targets. Set the聽redact聽option in the Target to "none"聽to remove redacting, or聽"all"聽to remove all sensitive data. The "all" option聽is the default. For now, only聽all聽and聽none聽are available. You will be able to fine-tune redact specific data types in the future.

Other Enhancements#

We removed the tutorial from "runops login"聽command. It only shows up in the聽runops signup聽option now. You can use the PUT users API endpoint using the email to identify the user. This is useful for integrations to activate/deactivate users from your internal systems without the Runops user id. The new Runops agent is now the default for the hosted and self-hosted options when creating new Targets.

Runops Releases #20

We are excited to announce many of the top requested features that we released over the last few weeks. Let's dive in!

Runops API is out#

You can start using the Runops API to build automations and integrate them with your existing tools. The list of endpoints and their specification is available in the docs (https://runops.io/docs/api). One use case requested by many of you was the user management API to integrate existing employee credentials provisioning systems, and we released it!

User management#

You can now add, update, and remove users using the Runops CLI. The create users option is an additional way to bring new members to Runops, in addition to the self-register option from Slack and CLI. You can also update user's status to activate/deactivate accounts and change users' teams and roles.

Role-based access control#

Control who can access what Targets. You can now group Targets into roles and attach these roles to users. A user only sees the Targets allowed by their roles.

Kubernetes Secrets support#

You can now store secrets and configurations in your Kubernetes Cluster secrets. In addition to Hashicorp Vault and AWS Secrets Manager, this new secrets storage mechanism makes it easier to get started before you move to a proper Secrets Manager.

馃帹 Improved outputs in the CLI#

You won't see JSON outputs as the result of commands anymore. The CLI now displays all outputs as tables. When creating Targets or running Tasks with reviews, the CLI displays all the information you need in an easy-to-read table.

鉁旓笍Improved validations for Tasks execution#

You will get a detailed error message when a specific secret or configuration is missing for a given Target type. The agent now performs an extensive set of validations before running a Task. Works both when creating a Target or running Tasks.

馃攲 New integrations#

You can use a range of new integrations in Runops. Use bash for running arbitrary shell commands. With k8s-apply you provide a YAML, and Runops applies it to Kubernetes clusters (we are using this one internally). Use rails to access a Rails application and rails-console get a Rails console experience, but with all the controls of Runops. You can check the complete list of integration in the docs: https://runops.io/docs/concepts/integrations

馃搼 Revamped docs#

You will have a better navigation experience in the documentation. Find dedicated sections to end-users and operators in the menu. The docs also got additional guides detailing the Kubernetes Secrets support. Check it out:聽https://runops.io/docs/quickstart/kubernetes

馃悰 Bug fixes#

Fixed bug in redact logic that broke emails format.

Runops Releases #19

We have been working hard the past couple of weeks to get this release out, and it's finally here! The new Runops agent is available, and it brings huge improvements to the Runops experience. We also launched a new version of the web app to get metrics and graphs on the Tasks executed and manage users' status. Let's dive in!

馃弾 10x improvement to execution time: from 50 to 5 seconds#

The new Runops agent brings the execution time from 50 to 5 seconds. The experience in the CLI got closer to the interactive experience of directly accessing a database or container. Slack users won't even notice a delay anymore.

馃洶 Simplified deployment model#

One instance of the new agent can execute Tasks from N Targets. This means that there is no need to set up a new agent for every new Target you add to Runops. Using a tagging system, you can configure which Targets run on which agents. A typical pattern is setting up one agent per environment and then using these same agents every time you need to add a new Target to Runops.

馃攼 Secret Managers integrations#

You can now keep your secrets in the Secrets Management solution of your choice. There is no need to add Secrets to Runops anymore. Your secrets stay in your vault, and the new Runops agent will pull a temporary credential to execute Tasks. We already support Hashicorp Vault and AWS Secrets Managers, with GCP Secrets Manager and other providers coming soon.

馃搱 Metrics, Graphs, and Users Management in the web app#

The revamped admin app has metrics and graphs that enable you to understand how your company uses Runops to access infrastructure. You get average review times and which teams are more active, besides the complete picture of Tasks and their details. Now you can also activate or deactivate users in the Users Management section.

馃悶 Error logs in Slack#

We fixed a bug that prevented you from seeing the result logs of Tasks that failed in Slack. The logs were sent as binary, and you had to download them and open them on your computer to see why the Task failed. Now we send results as text. You will see what happened in case of success or failure directly in Slack.

Hands Tied Engineers

The worst thing that can happen to an engineer is to get paged out-of-hours only to realize they can't fix the problem on their own. They did all the hard work of waking up, debugging the problem, and finding a solution. But when it's time to apply the fix, they don't have access. Time to call a DevOps engineer to get permission or have them run the patch.

DevOps means developers run their own code, but how can a developer operate a piece of software if she can't access the database, the cloud provider, or the Kubernetes cluster? Only a handful of people has access to these resources at most companies today.

This problem is not just bad on out-of-hour pages. DevOps teams centralizing raw access to production are bottlenecks to the whole engineering team. A simple query in the database to troubleshoot a problem can take hours for the busy and sad DevOps team to process the request in their queue.

It's not ok to keep making direct updates to the database or change things in the AWS console all the time. CI/CD and infrastructure as code are great tools. But direct access will happen no matter how much automation a company has. Restricting raw access to a few engineers results in bad culture incentives and an environment with low trust and autonomy.

Runops democratize access to production to enable DevOps. We fix this problem by adding security and compliance into easy-to-use clients. Runops enable any engineer to make production access with security and reliability.

Runops is a client to databases, AWS, Kubernetes, and others. Users access Runops from Slack and a CLI. We make accessing cloud resources easier than installing multiple clients and using VPNs. Runops also improves security and GDPR compliance.

Runops Releases #18

馃摎 Revamped documentation website#

We completely revamped the Runops documentation website. It's now easier to navigate and has a better design. We also enabled the chat support widget to the docs. Now you can reach out anywhere in the docs to talk to us. The new website is also hosting our blog, where the Releases and other articles are shared.

馃檹 Thanks#

That is it for this week, we'll see you next week with more Releases.

As always, please let us know of any feedback or comments you might have on Runops.

Thanks!

Runops Releases #17

馃幆 Select Task type in Slack#

You can now override the type of a Task in Slack. We added a new menu option that you can use to change the type from what is set in the Target when creating a Task in Slack. It's an optional field, so if you don't provide it, the type set in the Target is used.

馃檹 Thanks#

That is it for this week, we'll see you next week with more Releases.

As always, please let us know of any feedback or comments you might have on Runops.

Thanks!

Runops Releases #16

Andrios

Andrios

Engineering

Improved logs experience in Slack, Targets pagination in Slack, improved analytics and security#

馃搥 Logs in Slack#

We improved the delivery of logs in Slack. The Task logs are now delivered as an attached file to Slack for logs with less than 5MB in size. You can see the results directly in the message instead of having to download a file to your computer and open it. The download link continues to work for logs with more than 5MB in size.

馃幆 Targets pagination in Slack#

We added a button to the Slack Tasks execution UI that let you paginate Targets. This is useful for organizations with more than 100 Targets. By default the UI will load the first 100 Targets and you can use this button to load the next 100 Targets in the org.

馃И Improved analytics and security#

This task has no direct impact to how you use Runops today. We worked on improving the measurements of the app to further improve your experience and had an external party perform a penetration test to the Runops API. The results were great, let us know if you want to get access to the report.

馃檹 Thanks#

That is it for this week, we'll see you next week with more Releases.

As always, please let us know of any feedback or comments you might have on Runops.

Thanks!

Runops Releases #15

Andrios

Andrios

Engineering

馃獩 Enhanced Templates execution#

We fixed a bug that prevented Templates from running when the number of Templates in the Github repository increased. Templates now run faster and there is no limit to the number of Templates in the repository. We also updated the CLI experience: the Templates list now return just the names and to get the parameters of the Template we created a new command: runops templates get --name {template-name}. This makes it easier to read the output in the terminal and also helps for a faster Templates experience.

馃檹 Thanks#

That is it for this week, we'll see you next week with more Releases.

As always, please let us know of any feedback or comments you might have on Runops.

Thanks!

Runops Releases #14

Andrios

Andrios

Engineering

ECS exec Tasks, CLI Sign Up, Compliance enforcement, and major CLI experience improvements.#

馃搰 ECS Exec commands integration#

You can now use Runops to access running containers in AWS ECS, using the ECS exec feature recently launched by AWS. This means that you can get a shell inside ECS containers without bastion hosts or exposing your private network to the internet. The access happens trough the AWS internal infrastructure. You get the all the auditing capabilities of Runops along with the AWS Cloud Watch trails for every command. It's a great alternative to manage and control access to the Rails and Elixir consoles in production.

馃拡 Sign up from the CLI#

We are really excited about this feature! Anyone can Sign Up for Runops directly from our CLI. New organizations get created when the first user in the company signs up with a company email. Users joining existing organizations can also use the CLI to do so. The runops signup command in the CLI will route an user to the right workflow. We will identify the organization from the company's email domain. The approval mechanism in Slack for users joining an existing organization stays the same.

馃崶 Major CLI experience improvements#

The admin experience is way better in the CLI. The CLI now requires the minimum amount of parameter for any operation. You can do things like add new secrets, update runner, change the type, all providing only what you meant to change. Before this update you had to provide parameters you were not changing like the type and review mode for every command. We also cleaned up null values present in json-formated responses.

Compliance-enforced reviews#

The teams review mode got a new compliance enforcement feature: the creator of a Task can't review their own Tasks. This is important for companies that have resources with shared responsability across teams and require this type of locking mechanism.

馃檹 Thanks#

That is it for this week, we'll see you next week with more Releases.

As always, please let us know of any feedback or comments you might have on Runops.

Thanks!