Python/DevOps

Some time ago, I wanted to debug messages on a specific topic in Kafka , but I did not want to do that on production (we set up Kafka on Kubernetes using bitnami chart ). So, the question was, how can I do that? How can I debug something on production? It almost always a bad idea, so I wanted to "dump" a topic on my local machine. Something like "dump of topic" does not exist of course, because what does it mean in Kafka world? 🤔 But, you can create a consumer which will consume the topic from the beginning and pipe messages into a file; then using producer you can put these messages onto a specific topic on your local machine. Remember that my Kafka Cluster is deployed on Kubernetes Cluster. I can "ssh" onto Kafka pod, run command and then copy the file on my local machine. Or, I can just exec a command on a Kafka pod and pipe the output (messages) to my local machine and save the messages locally. Here is the comamnd: You tell kafka-0 pod that you ...

Pipeline Editor has been introduced in 13.8 version. What can you do with this tool? Normally, when you edit your .gitlab-ci.yml file, you will find that something is wrong with the code only after you push your changes. It would be great to know that there is a typo or something extremely wrong with this config in advance, right? And for this, you can use Pipeline Editor . There are three things you can do with that tool: Validate pipeline configuration Visualise the configuration Lint the configuration Validation is done automatically when you're editing the file. Okay, so how to start? 🤔 Go to CI / CD section and click the Editor link: Validation After that, you should see the Pipeline Editor: Let's break something! Hmm, what's wrong with this code? 🤔 I've received a message that my CI configuration is invalid because chosen stage does not exist. This make sense! I have a build  job and I inherit from hidden job .default , and I had not set a stage in any of th...

Hi, today I am going to introduce Pluto to you. So, what is it? 🤔 Pluto is a utility to help users find deprecated Kubernetes apiVersions in their code repositories and their helm releases. Sounds great! You use Kubernetes right? So, you know that Kubernetes has Deprecation Policy . It is a large system, so many features evolve over time and many of them might be removed. Thus, to avoid problems, Kubernetes provides a deprecation policy. So, you know in advance that something is going to be deprecated and you should act! There are 3 main " API Groups " in Kubernetes: Alpha (v1alpha1) Beta (v1beta1) Stable (v1) From time to time, some objects (or their versions) in Kubernetes can be removed or deprecated and it would be nice to know about it in advance. And for that job you can use: Let's see this guy in an action! You need to install it firstly of course. You can do that by this command: I used minikube ( minikube start --kubernetes-version=v1.16.1 ) to set up a...

I wrote a blog post about managing secrets in GitLab / Git some time ago, where I touched sops . Today, I am going to write more about this tool. sops  is useful when you want to encrypt your data and keep it somewhere securely. Why is it so secure? Because it uses envelope encryption . This way you can keep your encrypted data and encrypted data key  (which is needed to decrypt your data ) in the same file. So, when everything is encrypted you can store it anywhere, for example in your git repository. How it works? sops  generates a data key and this data key is used to encrypt and decrypt your data . So, how then is your data key encrypted? 🤔 By your KMS or PGP master key (or both of them, or even more...  sops  supports AWS KMS , GCP KMS, Azure Key Vault and PGP). As you can see, sops only touches your data key and your data . With your master/wrapping key you encrypt and decrypt your data key . By default, you can encrypt and decrypt your data...

Firstly what is it? AWS Key Management Service (AWS KMS) is a managed service that makes it easy for you to create and control customer master keys (CMKs), the encryption keys used to encrypt your data. AWS KMS CMKs are protected by hardware security modules (HSMs) that are validated by the FIPS 140-2 Cryptographic Module Validation Program except in the China (Beijing) and China (Ningxia) Regions. So, with AWS KMS you can store your customer master keys securely. What are customer master keys (CMKs) then? Customer master key is the primary resource in AWS KMS (so, it has own ARN). It is a logical representation of a master key. You can create  symmetric and asymmetric CMKs. CMKs never leave AWS infrastructure unencrypted. No one from AWS has access to these guys, only you. Your master keys are stored in such devices (hardware security module ( HSM )): You can read more about the cryptographic details here . Okay, so what can you do with AWS KMS? AWS managed CMKs U...

Let's say that you have to log in via ssh into an instance, and you work with GitLab, so you want to keep the private key in GitLab somewhere. Is it secure? Let's see! Custom environment variables You can use custom environment variables. Here you can read more about them (Developers cannot change them, only Maintainers and Owners can). There are two types of variables: Variable (the runner creates an environment variable that uses the key for the name and the value for the value) File (the runner creates an environment variable that uses the key for the name. For the value, the runner writes the variable value to a temporary file and uses this path) It seems that we can use File type for our purpose. We can set up it via API or UI . So, let's do that! Go to project's Settings > CI/CD . There will be Variables section (btw, you can specify variables also per group and even for all projects (in admin panel)). Click Add Variable button and add a variable: Key: ...