GOA Systems

Month: January 2022

  • Send passwords more secure

    Sometimes it is inevitable to send passwords and credentials to a counterpart who needs access to some kind of management tool because there is not extended and secure user management available.

    I always just sent the credentials with the login link. Now imagine what this means: A potential scanner or crawler could find my the credentials combined with the login link in one email or if the email gets accidentally forwarded there is also the full information attached including who sent it: Me.

    Now I started to use some sort of 2 factor communication:

    1. Write the credentials in a Zip file and save it with a password.
    2. Send the protected Zip file via Mail to the counterpart.
    3. Send the password over a different channel, e.g. WhatsApp, another mail, a phone call or a small piece of paper.
    4. Do not mention the original Mail with the Zip file in the password message.

    What does this mean?

    1. Somebody unauthorized would not know what the password is for.
    2. The password for the Zip file (sent via WhatsApp) is in a different system than the Zip file itself (sent via Mail), thus hacking one system does not bring anything.
    3. The password is not stored in plain text and does not get multiplied by every mail forward.
    4. This method adds another layer of security because now two systems would need to be compromised.
    5. It pushes others to also use this system and acts as a reminder to not just forward mails with passwords but to also use the second communication channel because one would have to rewrite or at least edit the mail with the password in order to get everything into one insecure mail.

    I got on this topic last time when I received credentials via mail and the password via WhatsApp. In the first moment I took this for granted and realized only a few days later that this is pretty genius. It is not the most secure or absolute perfect solution but it does add a layer of security to communication and this is what security is about for me.

  • Example for automating a time consuming manual process

    My recent entries were quite technical and were meant to solve very specific problems or show how to overcome certain issues on the last mile that are very annoying in a per se well defined concept. Today I’d like to give a overview from a higher level to put my other posts into perspective.

    The premise: You are a software/service provider

    Imagine you have a program or software you want to offer to your customer but the customer, often technically inexperienced, should not have to deal with servers, providers, DNS entries, certificates, maintenance, the cloud and all the technical stuff that comes with this. He should just fill out a form and afterwards it “should just work”.

    The start: A manual process

    As soon as the order comes in a manual process takes place on the provider side:

    • Checking the provided data (are the VAT number and contact details correct, is it a real customer, etc.)
    • Create the virtual machine. Each customer gets one because problems on one machine should not affect the others.
    • Install and configure the software.
    • Setup DNS entries so that the page can be found by others (i.e. use “myshopname.com” instead of “172.16.47.11”)
    • Acquire certificates for state of the art secure communication. Under certain circumstances this has to be repeated every three month.
    • Register the page with tags in search engines.
    • Integrate the server into a monitoring environment so that you can call the customer that a problem has been solved instead of the customer calling you that he has a problem.

    As you can see: There are quite a lot of steps that need to be done and all starts with a potential customer filling out a form. Now customers are volatile and if this takes longer than the “accepted amount of time” the customer will not use your service but maybe the competitors. I don’t have to go into detail what this means. The technical process on the other hand takes a certain amount of time and it has to be done by a person. While this person is doing it all new requests that come in during the process have to wait and get scheduled back which increases the duration even more. This means this is not very effective. Thus it should only be done during development of an automated process: Fist get something running, then make it better and more efficient. The next part will describe how such a process could look like.

    The goal: Automation

    The final goal is a process where the only thing indicating that a new customer has subscribed to the service is a payment at the end of the month but just sitting down and saying “I am going to do this!” is like standing in front of a mountain and trying to take it all in one step: Impossible. What is possible is automating and testing small parts on the way and putting them together to get a automated process.

    First let’s analyze the mentioned manual steps:

    As you can see there are several layers and “partners” where to get information from or where to send information to. Often these partners offer APIs (interfaces working with machine language that can be easily accessed from a program). So now I’d estimate where I could get which information from which leads to this annotated diagram:

    Annotated process

    Now I would have a basic understanding what needs to be done and where I could get my information from. This steps then translate directly into stages of two pipelines (automated and via trigger launchable processes). The first trigger (for the validation pipeline) would be the customer filling out the form and sending it and the second trigger (for the setup pipeline) would be the customer clicking on the confirmation link in the email he gets from the system.

    Pipeline view of the processes

    And in the end the customer would get a notification mail with generated access credentials and when he opens the page for the first time he would have to change the password to his own.

    This is of course a massive oversimplification. In this process there are much more components involved: You need a application hosting the pipeline and listening to triggers, a server is required to provide software components, a monitoring server has to be set up and so on and so forth. Bringing everything into these diagrams would have been way too much for this example. What I wanted to show is that even very complex processes can be broken down to simple, manageable steps. Or as said in terms of programming: Divide and conquer

    Thanks for reading and as always: I hope this helps somebody out there.

    PS: If you want to modify the diagrams for yourself you can open the PNG files in diagrams.net just save them and open them on the page like this:

    Just select the PNG and it should open as editable diagram.

  • Zabbix agent deployment and configuration

    Installation and configuration of the Zabbix agent, the counterpart of the Zabbix server, that collects and sends data to the server to be shown, is quite easy and can be done with a simple bash script.

    #!/bin/bash
# File: client.sh

SERVER=$1
HOSTNAME=$2

# Zabbix version
ZV=5.4

# File name version
FNV=${ZV}-1

if [ -z $HOSTNAME ]
then
        HOSTNAME=$(cat /etc/hostname)
fi

if [ -z $SERVER ]
then
        SERVER=zabbix.example.com
fi

wget https://repo.zabbix.com/zabbix/${ZV}/ubuntu/pool/main/z/zabbix-release/zabbix-release_${FNV}+ubuntu20.04_all.deb
dpkg -i zabbix-release_${FNV}+ubuntu20.04_all.deb
apt update
apt install -y zabbix-agent

sed -i "s/^Server=127\.0\.0\.1$/Server=${SERVER}/g" /etc/zabbix/zabbix_agentd.conf
sed -i "s/^Hostname=Zabbix server$/Hostname=${HOSTNAME}/g" /etc/zabbix/zabbix_agentd.conf

systemctl restart zabbix-agent.service

    Now let’s imagine you have 1000 hosts and would have to do that on each and every one of them. That’s going to be a long day of work.

    My assumption here is, that you as the administrator have ssh root access to all these machines via a ssh public key and each host has a individual hostname (For example: vm1, vm2, vm3, …, vmN or GUIDs).

    If this is given the shown script can be transferred to the host via “scp” and executed via “ssh”. I will show this with 4 hosts but with a little bash magic this can be extended to a lot more.

    #!/bin/bash
# File: server.sh

SERVER=zabbix.example.com

for HOST in vm1 vm2 vm3 vm4
do
        scp client.sh root@$host:/root/client.sh
        ssh root@$host "chmod u+x ./client.sh; ./client.sh $SERVER $HOST; rm ./client.sh;"
done

    In this script the server securely copies the file “client.sh” shown above to the hosts one by one and executes it to install and configure Zabbix agent. The agent is restarted afterwards and can be added to the Zabbix server. I am planing to write another tutorial on how to do this via API. This can then be integrated into the server script.

    The client script can also be used to add the Zabbix agent manually. Just check the current version and modify the script if necessary.

    As always: I hope this saves somebody some time and makes administration a little bit easier.

  • Parse JSON output from an API with Python

    As great as PowerShell is with JSON objects you can’t always use it because it is not available on each and every machine with Linux out there. And I don’t know a way of parsing JSON code with plain bash properly. Fortunately a lot of servers come with “python” installed and it can be used to parse JSON and retrieve values from the object with a comprehensible notation.

    For example, this is the response from a API call to create a new DNS record in a self managed zone:

    {
    "record": {
        "id": "4711",
        "type": "A",
        "name": "testentry",
        "value": "0.0.0.0",
        "ttl": 3600,
        "zone_id": "2",
        "created": "1970-01-01 00:00:00.00 +0000 UTC",
        "modified": "1970-01-01 00:00:00.00 +0000 UTC"
    }
}

    What I would need from this call is the record id. To show how this works, I have to explain and extend the example: Such a response comes most of the times when calling an API via “curl” for example. It is not just lying around on the disk. This in mind let’s assume that the shown JSON code is the result of the following call:

    curl -s https://exampleapi.goa.systems/path/to/endpoint

    Now curl just puts out the response on stdout and we can pipe it in the following command and read it there via stdin:

    curl -s https://exampleapi.goa.systems/path/to/endpoint | python3 -c "import sys,json; print(json.load(sys.stdin)['record']['id'])"

    With this code the value of “id” (['record']['id'] – compare with the source JSON, in the example “4711”) is printed (println) on the command line. Now, to store it in a variable one could do this:

    $RECORDID=$(curl -s https://exampleapi.goa.systems/path/to/endpoint | python3 -c "import sys,json; print(json.load(sys.stdin)['record']['id'])")

    And then save it on the disk and use it for your purposes. In my case for example I’d use it for deleting the record later in a process I’ll describe in another tutorial.

  • Redirecting with mod_rewrite

    Assume you have a shiny new site with great content, a new URL (or http address) and are expecting customers. Unfortunately your old site is not yet really gone in the search results but luckily you still have access to your old address. There is an easy way to redirect the users, who click on the search results pointing to the old page.

    1. Setup a vhost

    You need to answer to those requests. For this step it is assumed that the old domain is pointing to a web space or server managed by you. Basically all you need is a folder and a configured host (vhost):

    <VirtualHost *:80>
  ServerName my.old.domain
  DocumentRoot /var/www/myoldsite
  <Directory /var/www/myoldsite/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>
<VirtualHost *:443>
  ServerName my.old.domain
  DocumentRoot /var/www/myoldsite
  <Directory /var/www/myoldsite/>
    Options Indexes FollowSymLinks MultiViews
    AllowOverride All
    Require all granted
  </Directory>
  SSLEngine on
  # SSL configuration
</VirtualHost>

    Let’s just answer on both ports. Just in case. And don’t forget to enable mod_rewrite:

    a2enmod rewrite

    Now if the page is opened, there is no error but a empty directory listing. Better than nothing, but the goal is to redirect to the new page.

    Hint: I am realizing this with a “.htaccess” file because this way I can reconfigure “on the fly” and do not have to reload the web server after every change.

    2. Setup “.htaccess” for redirection

    Create a new file in the directory ” /var/www/myoldsite” named “.htaccess”. Don’t forget the leading “.”!

    RewriteEngine on
RewriteCond %{QUERY_STRING} ^(.*)$
RewriteRule .* https://my.new.site

    This means: If the url contains a query string (the part after the top level domain) rewrite the whole URL to ” https://my.new.site”.

    Why?

    You don’t know which sub pages are cached and shown in the search results. So: It’s better to just forward all to the new page instead of showing a “404 – not found”. Plus: You can define multiple rules and forward specific matches to possible new sub pages. But that would be beyond the scope of this short tutorial.

    Why just sub pages?

    On the main page I show a hint, that the URL has changed and that the user should be so kind as to change his bookmarks if the old URL is stored there. He is automatically redirected to the new page after 7 seconds:

    <!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="refresh" content="7; url='https://my.new.site'" />
    <style type="text/css">
        * { font-family: sans-serif; }
    </style>
  </head>
  <body>
    <p>The domain has changed to 'https://my.new.site'. If you have bookmarks, please save them under the new domain. You will be forwarded in a few seconds. Alternatively you can use <a href="https://my.new.site">this link</a>.</p>
  </body>
</html>

    I hope this helps somebody to guide users and customers to a new and improved web presence without having to deal with ugly error messages and confusion when clicking on search results.

  • Why DevOps?

    Why not just developers, like Steve Balmer once said? Why not just managers? Why do you need someone who understands how software is written and who also understands how it can be tested, packaged, deployed and maintained in an automated manner? I think this question answers itself, doesn’t it?

    I’d like to bring up an example that I have experienced first hand: Deployment of a PHP based software, developed by an external developer and originally published over a deployment tool provided by the framework developers.

    The situation was, that the developers had access to the deployed instances of the software and applied patches manually. This led to the situation that no one knew which patch level was installed on which machine. To be honest it wasn’t even possible to talk about patch levels because hot fixes were applied directly and sometimes not integrated into other installations or the distribution itself. So how do you maintain this? That’s the neat part: You don’t!

    So what can you do about this?

    1. Draw a line

    The development was separated from the deployment and the interface was a versioned zip distribution of the software in the developer to customer direction. Customer to developer direction was implemented as bug tickets and development sprints with a versioned outcome.

    2. Split up, define and clean up

    All parties, developers, administrators and the DevOps Engineer in between, use the same environments and use the same point of origin: A clean machine with a defined operating system and a clear and not deviating setup procedure. Required manual changes are reported back to the DevOps engineer in order to integrate them into the machine setup process and every one in the team knows how the machines are set up and how they are supposed to work. Log files are in the same location, the same technologies are used on all machines and there are no manual fixes on production instances.

    3. Automate

    As soon as things are clear: automate. Set up pipelines (in GitLab, Azure DevOps or Jenkins for example), define access (API, SSH or other remote protocols) and specify the installation procedure. This means where to gather your software, what to install, how to configure installed tools and what external sources, like DNS entries and certificates, to use.

    Conclusion

    The basic tasks of DevOps, like compiling (gcc, javac, csc, tsc …), packaging (Nsis, zip, tar, …), deploying (FTP, SSH, SMB, …), patching and archiving (Maven, NuGet, npm, …) are similar if not the same in most of the cases. One has to know the protocols, how they act together and what it means to “setup” software. A software product is a conglomerate of multiple components: Application, configuration, environment, database and many more things. As a DevOps engineer you have to know that these things exist, that they can be set up and you need the drive to dig into the configuration and not just be satisfied with “yeah it works now”. What drives me is the “How does it work?”, “How is it done in best practice?” and “How can I automate it reproducible?” Yeah, maybe sometimes I invest a little too much time into digging into the topic but I think after 100.000 iteration where my script, I spent a few hours on, works I think it has paid off …

  • Improve Gimp UI integration

    Okay, this is nothing functional and really just something for the eye. I am using Gimp on a regular basis and think it is a really great piece of software. There was just something that annoyed me every time I used the software: The font integration which looked to me somehow “out of place”. Let me give an example:

    The font here just does not match the default Windows font “Segoe UI” and the scaling is also different. This is how it should look like:

    You can clearly see the difference in the font presentation. Good thing is, this can easily be changed in the file “gtkrc” in the theme directory:

    %ProgramFiles%\Gimp\share\gimp\2.0\themes\Dark\gtkrc

    Replace “Dark” with the theme you are using on your system. There are a few of them in this folder.

    1. Uncomment the line starting with “#gtk-font-name” and change it to:
      gtk-font-name = "Segoe UI 9"
    2. Find the line
      GimpRuler::font-scale = 0.6444
      and change it to
      GimpRuler::font-scale = 1
    3. Find the line
      GimpDock::font-scale = 0.6444
      and change it to
      GimpDock::font-scale = 1

    With these changes Gimp should look like a regular Windows application using the default font.

  • GitLab: Persisting data between jobs

    When I first tried to write and run pipelines in GitLab first thing I got were errors. I tried to create a json file containing data that needed to be accessible in each job of the pipeline. This was an id generated randomly in the first step and each following step would rely on this id to produce additional data and store it. I found out, that my generated id file was not accessible in the follow up steps and thus my pipeline would not work. Fortunately there is a solution for this and I’d like to share in a simple example of three files how to make this work. This can be adopted to all sorts of data like binary files, test results, compiler outputs, generated PDFs and all sorts pipeline output. This documented in detail on the official GitLab documentation page but because I prefer quick and direct tutorials handling just my core problem and leave modifications and extensions to me instead of blown up documentations covering every single detail I want to share my solution here.

    My solution consists of three files. The first two are the two scripts for the steps. Step one generates the JSON file and stores it and is called “step1.ps1”:

    $MyObject = @{
	guid = (New-Guid).ToString()
}
$Json = ConvertTo-Json -InputObject $MyObject
Set-Content -Path "guid.json" -Value $Json
Write-Host -Object "Generated Guid: $($MyObject.guid)"

    The second file is called “step2.json” and reads the generated id from the JSON file and thus needs access to the file “guid.json”:

    $Json = ConvertFrom-Json -InputObject (Get-Content -Raw -Path "guid.json")
Write-Host -Object "Loaded Guid is $($Json.guid)"

    Why is the generated UUID printed to stdout via “Write-Host”? To check if they are the same in the pipeline.

    Finally the “.gitlab-ci.yml” file defines the pipeline:

    stages:
  - create
  - read

create-json:
  stage: create
  script:
    - pwsh -File step1.ps1
  artifacts:
    paths:
      - guid.json

read-json:
  stage: read
  script:
    - pwsh -File step2.ps1

    So now your folder should consist of three files containing the shown code:

    And running the pipeline in GitLab will show this:

    The jobs can then be opened and should show …

    $ pwsh -File step1.ps1
Generated Guid: deb4e6cc-c968-4c86-95f5-fd72ca0f64c9

    … in the “create-json” job and …

    $ pwsh -File step2.ps1
Loaded Guid is deb4e6cc-c968-4c86-95f5-fd72ca0f64c9

    … in the “read-json” job in the log output.

    Now one has the basic principle implemented to share data and artifacts between jobs.

    But wait … there’s more!

    When opening the job the generated artifacts can be downloaded to the local disk:

    This zip file would now contain the file “guid.json” generated by “step1.ps1” and defined in the file “.gitlab-ci.yml” with the lines

    artifacts:
  paths:
    - guid.json