Lesegruppen / Buchclubs in Unternehmen

Buchbesprechungen als Werkzeug für Organisationales Lernen

Auf den devopsdays 2015 in Berlin hatten wir eine Open Space Gruppe, die sich mit dem Thema beschäftigte: “Ich lese viele Bücher, Blogs etc, und würde dieses Wissen gerne in das Unternehmen tragen, in dem ich gerade arbeite”. Dabei kam die Idee auf, Lesegruppen zu bilden. Diese Idee fand ich so gut, dass ich sie direkt einmal ausprobiert habe.

Seitdem habe ich mehrere Buchlesegruppen in Unternehmen (mit-)gegründet und diese “Buchclubs” als als sehr effektives Werkzeug des strukturierten Organisationales Lernens schätzen gelernt.

Was ist ein Buchclub / eine Lesegruppe?

Ein Buchclub ist eine Gruppe von Leuten, die ein Buch durcharbeiten und dann (z. B.) kapitelweise in regelmäßigen Treffen besprechen. Das eigentliche Lesen geschieht vor den Treffen durch jedes Mitglied in Eigenverantwortung. Als Bücher kommen Fachbücher in Frage, die einen Bezug zum Unternehmen haben.

Vorteile von Buchclubs - was bringt uns das?

Buchclubs bringen eine Reihe von positiven Eigenschaften:

  • Besprechung in der Gruppe: Das Themengebiet des Buches wird in der Gruppe besprochen und bearbeitet. Dabei entstehen für das Unternehmen sehr wertvolle (wenn nicht die wertvollsten) Diskussionen. (Zumeist) neue Konzepte werden zuerst von jeder teilnehmenden Person einzeln gelesen und danach in der Gruppe besprochen.
    Es sind also nicht die Ideen einer einzelnen Person, sondern jedes Gruppenmitglied hat das Thema selbst durch Lesen erarbeitet. Menschen neigen dazu, “selbst aufgenommene” Informationen besser aufzunehmen und in ihre mentalen Modelle einzuarbeiten (die ggf. stark zu dem Gelesenen divergieren), als wenn z.B. eine Einzelperson versucht, neue Ideen in eine Gruppe zu bringen.
  • Angleichen von unterschiedlichen mentalen Modellen: Wie wir die Wirklichkeit sehen, ist immer nur ein Ausschnitt. Es kann zum Beispiel sein, dass völlig unterschiedliche Arbeitsweisen oder Begriffsdefinitionen vorliegen innerhalb der Gruppe. Werden diese im Buch angesprochen, so entstehen oft Diskussionen wie “Ach, ihr macht das so?”, oder “Ach jetzt verstehe ich, was ihr mit X meint!”, aber auch “Dann lass uns doch auf X einigen, ich spreche das einmal in meinem Team an!”. Das sind genau die richtigen Diskussionen, weil Missverständnisse und Unverständnisse untereinander sachlich(er) geklärt werden, und dadurch wahrscheinlich auch die Gruppenkohäsion im Unternehmen gestärkt wird.
  • Reflektion der eigenen Arbeit: Fachbücher bieten eine gute Grundlage, um einen Realitätsabgleich von aktuell vorherrschenden Arbeitsweisen oder -mustern zu machen. Natürlich steht in Büchern auch immer nicht die ganze Wahrheit, oder die vorgestellte Welt ist zu perfekt, aber trotzdem bietet die Literatur meistens eine gute Indikation, auf welchem Level eine Person oder eine Gruppe sich befindet. Aktuell zum Beispiel lesen wir das Buch “Site Reliability Engineering”, welches beschreibt, wie Google intern arbeitet - und daher sind einige Konzepte aus dem Buch auch erst anzuwenden, wenn man eine Größe wie Google erreicht hat. Im Großen und Ganzen sind die Konzepte aber übertragbar bzw. sie regen zumindest wertvolle Diskussionen an.
  • Direkte Anwendung von Gelerntem: Ich erinnere mich an einen Buchclub, in dem wir “Implementing Domain-Driven-Design” durchgearbeitet haben, und Leute aus unterschiedlichen Teams dabei waren. Herausgekommen sind sehr konstruktive Diskussionen über die Gesamtarchitektur der Software, die das Unternehmen entwickelt, und zwar dieses mal geleitet von der Theorie aus dem Buch und nicht von unterschiedlichen mentalen Modellen oder Wissensständen (was für mich gefühlt in vorherigen Meetings immer der Fall war).
    Ein weiteres Beispiel war ein Buchclub, in dem wir “Toyota Kata” besprochen haben, und dann angefangen haben, eine Value-Stream-Map für das gesamte Unternehmen aufzustellen. Das war ein spannender Einblick in andere Unternehmensbereiche und ich habe gemerkt, wieviel Spaß es macht, mit der Gruppe erst die Theorie zu besprechen und dann darüber zu philosophieren, wo unser Unternehmen eigentlich wertschöpfend ist - so qualitativ hochwertige Diskussionen habe ich selten erlebt.
  • Gruppendruck: Wie so viele Dinge, die wichtig, aber nicht dringend sind, geht häufig auch das disziplinierte Zu-Ende-Lesen von Büchern im Alltag unter. Hier kann der Druck der Gruppe helfen: Wenn morgen der nächste Besprechungstermin ist, dann steht man mitunter schon einmal eine Stunde früher auf, um das Kapitel durchzulesen.
  • Günstig: Es gibt viele Wege der Mitarbeiter- und Teamentwicklung: Workshops, Schulungen, Konferenzen etc. - diese sind oft sehr teuer. Und der Erfolg ist ggf. auch noch fragwürdig: Meistens holt uns nach einer Schulung oder Konferenz schnell wieder das Tagesgeschäft ein - und die frischen Ideen sowie der Elan verpuffen. Die direkten Kosten von Buchclubs beschränken sich normalerweise auf Lese- und Besprechungszeit sowie die Anschaffung des Buchs.

Wie fange ich an?

In der Gruppe

Thema, Buch und Lesegruppe finden

Zuerst muss “jemand” ein Buch vorschlagen und dann dafür eine Lesegruppe finden. Meistens wird diese Person dann auch direkt Organisator der Gruppe. Das Finden von Mitgliedern kann z. B. durch Vorstellung des Buchclub-Konzepts in Meetings oder einfach an der Kaffeemaschine passieren. Wichtig ist, dass es immer ein freiwilliges Angebot ist.

Hier hilft es auch, dass man selbst “Schwäche zeigt”, z. B. “Ich würde mich gerne in Thema XYZ einarbeiten, da ich auf dem Gebiet Wissenslücken habe. Dazu habe ich Buch XYZ gefunden und würde dies gerne mit mehreren Leuten besprechen können, um sicherzustellen, dass ich die Konzepte wirklich verstanden habe”. Durch dieses “Verletzlich machen” (Zugeben, dass man Wissenslücken hat) erhöhen sich die Chancen, dass man mehr Menschen zum Mitmachen bewegen kann: Entweder merken sie, dass es nicht schlimm ist, mit Unwissen in diese Gruppe zu gehen, und dass es nicht darum geht, einzelne Mitarbeiter blosszustellen - oder sie sind schon versiert auf dem Thema und können dann in den Kapitel-Besprechungen ihr bestehendes Wissen gezielt einstreuen.

Lesegruppen-Setup

Hat man seine Lesegruppe gefunden, kann es losgehen! Zuerst sollte ein regelmäßiger Termin, z.B. jede Woche eine Stunde, festgelegt werden. Es ist auch hilfreich, direkt eine Mailingliste oder einen Chat (je nachdem, was im Unternehmen da ist) einzurichten und die Mitglieder einzuladen. Diese können dann für Ankündigungen genutzt werden.

Als nächstes einigt sich die Gruppe darauf, dass jedes einzelne Mitglied bis zum ersten Termin ein oder mehrere Kapitel durchgelesen hat. Als Empfehlung für den ersten Termin: Lieber mit dem ersten Kapitel oder der Einleitung starten, also nicht zuviel auf einmal am Anfang, denn beim ersten Treffen gibt es bestimmt viel auszutauschen.

Im Treffen geht die Gruppe dann das Kapitel durch, z. B. werden markierte Stellen besprochen. Weiterhin lohnt es sich oft, auch hinter die Referenzen und Fußnoten zu schauen. Es gibt auch die Möglichkeit, dass ein_e Moderator_in für das Treffen ausgemacht wird, diese_r dann das Kapitel vorstellt und hindurch leitet. Mit den entstehenden Diskussionen (siehe oben) ist die Zeit dann meistens auch schneller herum als erwartet.

Abschluss des Buches

Ist das Buch zuende gelesen, kann sich die Gruppe entweder auflösen oder direkt ein weiteres Buch finden und somit bestehen bleiben. Bleibt die Gruppe bestehen, ist es aber hilfreich, weitere potentielle Mitglieder in die Gruppe aufzunehmen, um die Diversität zu erhöhen. Denn auch Buchclubs sind nicht vor Gruppendynamiken wie Groupthink gefeit. Man muss auch aufpassen, dass sich keine In- und Out-Gruppen bilden (z. B. Mitglieder des Buchclubs, die sich dann “elitärer” fühlen als Nichtmitglieder).

Als Unternehmen

  • Arbeitszeit explizit freigeben: Als Zeichen, dass das Unternehmen die Weiterbildung seiner Mitarbeiter_innen unterstützt, sollte ein gewisser Anteil der Arbeitszeit “freigegeben” werden für explizite Weiterbildungsmaßnahmen wie z. B. Buchclubs. Ein Beispiel für ein Modell wären z. B. die Besprechungszeit übernimmt das Unternehmen, die Lesezeit die Mitarbeiter_innen.
  • Teamleiter_in mit einbinden: Häufig kommen in Buchbesprechungen dann “Man müsste mal” Themen auf. Hilfreich ist es hier immer, wenn Teamleiter_innen direkt mit dabei sind, so dass Änderungen an Arbeitsprozessen oder Experimente schneller umgesetzt werden können. Häufig werden dann auch Dinge angesprochen, die sonst im Alltag untergehen würden.

Zusammenfassung

Buchclubs bieten eine für Unternehmen sehr kosteneffektive Möglichkeit, die Weiterbildung seiner Mitarbeiter_innen zu fördern. Weiterhin sind sie ein Werkzeug für organisationales Lernen.

AWS Continuous Infrastructure Delivery with CodePipeline and CloudFormation: How to pass Stack Parameters

When deploying CloudFormation stacks in a “Continuous Delivery” manner with CodePipeline, one might encounter the challenge to pass many parameters from the CloudFormation stack describing the pipeline to another stack describing the infrastructure to be deployed (in this example a stack named application).

Consider a CloudFormation snippet describing CodePipeline which deploys another CloudFormation stack:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# pipeline.yaml
...
Resources:
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
...
Stages:
...
- Name: Application
Actions:
- Name: DeployApplication
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
StackName: application
TemplatePath: Source::application.yaml

Now when you want to pass parameters from the pipeline stack to the application stack, you could use the ParameterOverrides option offered by the CodePipeline CloudFormation integration, which might look like this:

1
2
3
4
5
6
7
8
# pipeline.yaml
...
- Name: DeployApplication
...
Configuration:
StackName: application
TemplatePath: Source::application.yaml
ParameterOverrides: '{"ApplicationParameterA": "foo", "ApplicationParameterB": "bar"}'

This would pass the parameters ApplicationParameterA and ApplicationParameterB to the application CloudFormation stack. For reference this is how the application stack could look like:

1
2
3
4
5
6
7
8
9
10
11
12
# application.yaml
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ApplicationParameterA:
Type: String
ApplicationParameterB:
Type: String
Resources:
...

Alternative way of parameter passing with Template Configurations

Injecting parameters from the pipeline stack to the application stack can become awkward with the ParametersOverrides method. Especially when there are many parameters and they are passed into the pipeline stack as parameters as well, the pipeline template could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# pipeline.yaml
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ApplicationParameterA:
Type: String
ApplicationParameterB:
Type: String
Resources:
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
Stages:
...
Actions:
- Name: DeployApplication
...
Configuration:
...
TemplatePath: Source::application.yaml
ParameterOverrides: !Sub '{"ApplicationParameterA": "${ApplicationParameterA}", "ApplicationParameterB": "${ApplicationParameterB}"}'

An alternative way is to place a so called template configuration into the same artifact which contains the application.yaml template, and reference it via the TemplateConfiguration:

1
2
3
4
5
6
7
8
9
# pipeline.yaml
...
- Name: DeployApplication
...
Configuration:
...
TemplatePath: Source::application.yaml
ParameterOverrides: '{"ApplicationParameterA": "foo", "ApplicationParameterB": "bar"}'
TemplateConfiguration: Source::template_configuration.json

In our case, the template_configuration.json file would look like this:

1
2
3
4
5
6
{
"Parameters" : {
"ApplicationParameterA" : "foo",
"ApplicationParameterB" : "bar"
}
}

This might be much nicer to handle and maintain depending on your setup.

Btw you can also use the TemplateConfiguration to protect your resources from being deleted or replaces with Stack policies.

"Service Discovery" with AWS Elastic Beanstalk and CloudFormation

How to dynamically pass environment variables to Elastic Beanstalk.

Elastic Beanstalk is a great AWS service for managed application hosting. For me personally, it’s the Heroku of AWS: Developers can concentrate on developing their application while AWS takes care of all the heavy lifting of scaling, deployment, runtime updates, monitoring, logging etcpp.

But running applications usually means not only using plain application servers the code runs on, but also databases, caches and so on. And AWS offers many services like ElastiCache or RDS for databases, which should usually preferred in order to have lower maintenance overhead.

So, how do you connect Elastic Beanstalk and other AWS services? For example, your application needs to know the database endpoint of an RDS database in order to use it.

“Well, create the RDS via the AWS console, copy the endpoint and pass it as an environment variable to Elastic Beanstalk”, some might say.

Others might say: Please don’t hardcode such data like endpoint host names, use a service discovery framework, or DNS and use that to look up the name.

Yes, manually clicking services in the AWS console and hardcoding configuration is usually a bad thing(tm), because it violates “Infrastructure as Code”: Manual processes are error-prone, and you’ll loose documentation through codification, traceability and reproducibility of the setup.

But using DNS or any other service discovery for a relatively simple setup? Looks like a oversized solution for me, especially if the main driver for Elastic Beanstalk was the reduction of maintenance burden and complexity.

The solution: CloudFormation

Luckily, there is a simple solution to that problem: CloudFormation. With CloudFormation, we can describe our Elastic Beanstalk application and the other AWS resources it consumes in one template. We can also inject e.g. endpoints of those AWS resources created to the Elastic Beanstalk environment.

Let’s look at a sample CloudFormation template - step by step (I assume you are familiar with CloudFormation and Elastic Beanstalk itself).

First, let’s describe an Elastic Beanstalk application with one environment:

1
2
3
4
5
6
7
8
9
10
11
...
Resources:
Application:
Type: AWS::ElasticBeanstalk::Application
Properties:
Description: !Ref ApplicationDescription
ApplicationEnv:
Type: AWS::ElasticBeanstalk::Environment
Properties:
ApplicationName: !Ref Application
SolutionStackName: 64bit Amazon Linux 2016.09 v2.5.2 running Docker 1.12.6

Ok, nothing special so far, let’s add a RDS database:

1
2
3
4
DB:
Type: AWS::RDS::DBInstance
Properties:
...

CloudFormation allows it to get the endpoint of the database with the GetAtt function. To get the endpoint of the DB database, the following code can be used:

1
!GetAtt DB.Endpoint.Address

And CloudFormation can also pass environment variables to Elastic Beanstalk environments, so let’s combine those two capabilities:

1
2
3
4
5
6
7
8
9
ApplicationEnv:
Type: AWS::ElasticBeanstalk::Environment
Properties:
ApplicationName: !Ref Application
...
OptionSettings:
- Namespace: aws:elasticbeanstalk:application:environment
OptionName: DATABASE_HOST
Value: !GetAtt DB.Endpoint.Address

Et voila, the database endpoint hostname is now passed as an environment variable (DATABASE_HOST) to the Elastic Beanstalk environment.
You can add as many environment variables as you like. They are even updated if you change their value (Cloudformation would trigger an Elastic Beanstalk enviroment update is this case).

CodePipeline and CloudFormation with a stack policy to prevent REPLACEMENTs of resources

Some operations in CloudFormation trigger a REPLACEMENT of resources which can have unintended and catastrophic consequences, e.g. an RDS instance being replaced (which means that the current database will be deleted by CloudFormation after a new one has been created).

While CloudFormation does support deletion policies natively which prevent the deletion of resources, there is no simple way to do this for REPLACEMENTs as of writing this.

When using CodePipeline in combination with CloudFormation to deploy infrastructure changes in an automated Continuous Delivery manner, one should have protection from accidental deletions even more mind. This blog post shows how to use CloudFormation Stack Policies to protect critical resources from being replaced.

Let’s start with the CodePipeline (expressed as CloudFormation) piece which deploys a database stack called db (I assume you are confident with CloudFormation and CodePipeline):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
...
Stages:
- Name: Source
...
- Name: DB
Actions:
- Name: DeployDB
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: db
TemplatePath: Source::db.yaml
TemplateConfiguration: Source::db_stack_update_policy.json
InputArtifacts:
- Name: Source
RunOrder: 1

The important part is the TemplateConfiguration parameter which tells CloudFormation to look for a configuration at this particular path in the Source artifact. In this case db_stack_update_policy.json.

db_stack_update_policy.json looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"StackPolicy" : {
"Statement" : [
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
},
{
"Effect" : "Deny",
"Action" : "Update:Replace",
"Principal": "*",
"Resource" : "LogicalResourceId/DB"
}
]
}
}

While the first statement allows all updates to all resources in the db stack, the second will deny operations which would result in a REPLACEMENT of the resource with the logical id DB in this stack.

A CloudFormation stack update of db would fail with an error message like Action denied by stack policy: Statement [#1] does not allow [Update:Replace] for resource [LogicalResourceId/DB].

Idempotent CloudFormation stack creation/update one-liner with Ansible

When developing CloudFormation templates, I regularly missed an idempotent one-liner command which does something like “create or update stack N with these parameters”, which provides a fast feedback loop.

So here it is with Ansible (and virtualenv for convenience):

1
2
3
4
virtualenv venv
source venv/bin/activate
pip install ansible boto
ansible localhost -m cloudformation -a "stack_name=stack_name template=path/to/template region=eu-west-1 template_parameters='template_param1=bar,template_param2=baz'"

It will create a new or update an existing CloudFormation stack and wait until the operation has finished. It won’t complain if there are no updates to be performed.

PS: Michael Wittig has released a CloudFormation CLI wrapper (NPM module) for this problem, too!

Continuous Infrastructure Delivery Pipeline with AWS CodePipeline, CodeBuild and Terraform

Overview

This article explores how to build low-maintenance Continuous Delivery pipelines for Terraform, by using AWS building blocks CloudFormation, CodePipeline and CodeBuild.

CloudFormation

CloudFormation is the built-in solution for Infrastructure-as-Code (Iac) in AWS. It’s usually a good choice because it offers a low-maintenance and easy-to-start solution. On the other hand, it can have some drawbacks based on the use case or the usage level. Here are some points which pop up regularly:

  • AWS-only: CloudFormation has no native support for third-party services. It actually supports custom resources, but those are usually awkward to write and maintain. I would only use them as a last resort.
  • Not all AWS services/features supported: The usual AWS feature release process is that a component team (e.g. EC2) releases a new feature, but the CloudFormation part is missing (the CloudFormation team at AWS is apparently a separate team with its own roadmap). And since CloudFormation isn’t open source, we cannot add the missing functionality by ourselves.
  • No imports of existing resources: AWS resources created outside of CloudFormation cannot be “imported” into a stack. This would be helpful for example when resources had been set up manually earlier before (maybe because CloudFormation did not support them yet).

Terraform to the rescue!

Terraform is an IaC tool from HashiCorp, similar to CloudFormation, but with a broader usage range and greater flexibility than CloudFormation.

Terraform has several advantages over CloudFormation, here are some of them:

  • Open source: Terraform is open source so you can patch it and send changes upstream to make it better. This is great because anyone can, for example, add new services or features, or fix bugs. It’s not uncommon that Terraform is even faster than CloudFormation with implementing new AWS features.
  • Supports a broad range of services, not only AWS: This enables automating bigger ecosystems spanning e.g. multiple clouds or providers. In CloudFormation one would have to fall back to awkward custom resources. A particular use-case is provisioning databases and users of a MySQL database,
  • Data sources: While CloudFormation has only “imports“ and some intrinsic functions to lookup values (e.g. from existing resources) Terraform provides a wide range of data sources (just have a look at this impressive list.
  • Imports: Terraform can import existing resources (if supported by the resources type)! As mentioned, this becomes handy when working with a brownfield infrastructure, e.g. manually created resources.

(Some) Downsides of Terraform

TerraForm is no managed service, so the maintenance burden is on the user side. That means we as users have to install, upgrade, maintain, debug it and so on (instead of focusing on building our own products).

Another important point is that Terraform uses “state files” to maintain the state of the infrastructure it created. The files are the holy grail of Terraform and messing around with them can bring you into serious trouble, e.g. bringing your infrastructure into an undefined state. The user has to come up with a solution how to keep those state files in a synchronized and central location (Luckily Terraform provides remote state handling, I will get back to this in a second). CloudFormation actually also maintains the state of the resources it created, but AWS takes care of state storage!

Last but not least, Terraform currently does not take care of locking, so two concurrent Terraform runs could lead to unintended consequences. (which will change soon).

Putting it all together

So how can we leverage the described advantages of Terraform while still minimizing its operational overhead and costs?

Serverless delivery pipelines

First of all, we should use a Continuous Delivery Pipeline: Every change in the source code triggers a run of the pipeline consisting of several steps, e.g. running tests and finally applying/deploying the changes. AWS offers a service called CodePipeline to create and run these pipelines. It’s a fully managed service, no servers or containers to manage (a.k.a “serverless”).

Executing Terraform

Remember, we want to create a safe environment to execute Terraform, which is consistent and which can be audited (so NOT your workstation!!).

To execute Terraform, we are going to use AWS CodeBuild, which can be called as an action within a CodePipeline. The CodePipeline will inherently take care of the Terraform state file locking as it does not allow a single action to run multiple times concurrently. Like CodePipeline, CodeBuild itself is fully managed. And it follows a pay-by-use model (you pay for each minute of build resources consumed).

CodeBuild is instructed by a YAML configuration, similar to e.g. TravisCI (I explored some more details in an earlier post). Here is how a Terraform execution could look like:

1
2
3
4
5
6
7
8
9
10
11
version: 0.1
phases:
install:
commands:
- yum -y install jq
- curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq 'to_entries | [ .[] | select(.key | (contains("Expiration") or contains("RoleArn")) | not) ] | map(if .key == "AccessKeyId" then . + {"key":"AWS_ACCESS_KEY_ID"} else . end) | map(if .key == "SecretAccessKey" then . + {"key":"AWS_SECRET_ACCESS_KEY"} else . end) | map(if .key == "Token" then . + {"key":"AWS_SESSION_TOKEN"} else . end) | map("export \(.key)=\(.value)") | .[]' -r > /tmp/aws_cred_export.txt # work around https://github.com/hashicorp/terraform/issues/8746
- cd /tmp && curl -o terraform.zip https://releases.hashicorp.com/terraform/${TerraformVersion}/terraform_${TerraformVersion}_linux_amd64.zip && echo "${TerraformSha256} terraform.zip" | sha256sum -c --quiet && unzip terraform.zip && mv terraform /usr/bin
build:
commands:
- source /tmp/aws_cred_export.txt && terraform remote config -backend=s3 -backend-config="bucket=${TerraformStateBucket}" -backend-config="key=terraform.tfstate"
- source /tmp/aws_cred_export.txt && terraform apply

First, in the install phase, the tool jq is installed to be used for a little workaround I had to write to get the AWS credentials from the metadata service, as Terraform does not yet support this yet. After retrieving the AWS credentials for later usage, Terraform is downloaded, checksum’d and installed (they have no Linux repositories).

In the build phase, first the Terraform state file location is set up. As mentioned earlier, it’s possible to use S3 buckets as a state file location, so we are going to tell Terraform to store it there.

You may have noticed the source /tmp/aws_cred_export.txt command. This simply takes care of setting the AWS credentials environment variables before executing Terraform. It’s necessary because CodeBuild does not retain environment variables set in previous commands.

Last, but not least, terraform apply is called which will take all .tf files and converge the infrastructure against this description.

Pipeline as Code

The delivery pipeline used as an example in this article is available as an AWS CloudFormation template, which means that it is codified and reproducible. Yes, that also means that CloudFormation is used to generate a delivery pipeline which will, in turn, call Terraform. And that we did not have to touch any servers, VMs or containers.

You can try out the CloudFormation one-button template here:

Launch Stack

You need a GitHub repository containing one or more .tf files, which will in turn get executed by the pipeline and Terraform.

Once the CloudFormation stack has been created, the CodePipeline will run initially:

CodePipeline screenshot

The InvokeTerraformAction will call CodeBuild, which looks like this:

CodeBuild log output screenshot

Stronger together

The real power of both TerraForm and CloudFormation comes to light when we combine them, as we can actually use best of both worlds. This will be a topic of a coming blog post.

Summary

This article showed how AWS CodePipeline and CodeBuild can be used to execute Terraform runs in a Continuous Delivery spirit, while still minimizing operational overhead and costs. A CloudFormation template is provided to ease the set up of such a pipeline. It can be used as a starting point for own TerraForm projects.

References

https://blog.gruntwork.io/how-to-manage-terraform-state-28f5697e68fa?gi=9769367dd11

AWS CodeBuild: The missing link for deployment pipelines in AWS

This is a follow-up of my AWSAdvent article Serverless everything: One-button serverless deployment pipeline for a serverless app , which extends the example deployment pipeline with AWS CodeBuild.

Deployment pipelines are very common today, as they are usually part of a continuous delivery/deployment workflow. While it’s possible to use e.g. projects like Jenkins or concourse for those pipelines, I prefer using managed services in order to minimize operations and maintenance so I can concentrate on generating business value. Luckily, AWS has a service called CodePipeline which makes it easy to create deployment pipelines with several stages and actions such as downloading the source code from GitHub, and executing build steps.

For the build steps, there are several options like invoking an external Jenkins Job, or SoranoCi etcpp. But when you want to stay in AWS land, your options were quite limited until recently. The only pure AWS option for CodePipeline build steps (without adding operational overhead, e.g. managing servers or containers) was invoking Lambda functions, which has several drawbacks that I all experienced:

Using Lambda as Build Steps

5 minutes maximum execution time

Lambda functions have a limit of 5 minutes which means that the process gets killed if it exceeds the timeout. Longer tests or builds might get aborted and thus result in a failing deployment pipeline. A possible workaround would be to split the steps into smaller units, but that is not always possible.

Build tool usage

The NodeJS 4.3 runtime in Lambda has the npm command pre-installed, but it needs several hacks to be working. For example, the Lambda runtime is a read-only file system except for tmp, so in order to use e.g. NPM, you need to fake the HOME to /tmp. Another example is that you need to find out where the preinstalled NPM version lives (checkout my older article on NPM in Lambda).

Artifact handling

CodePipeline works with so called artifacts: Build steps can have several input and output artifacts each. These are stored in S3 and thus have to be either downloaded (input artifact) or uploaded (output artifact) by a build step. In a Lambda build step, this has to be done manually, means you have to use the S3 SDK of the runtime for artifact handling.

NodeJS for synchronous code

When you want to use a preinstalled NPM in Lambda, you need to use the NodeJS 4.3 runtime. At least I did not manage to get the preinstalled NPM version running which is part of the Lambda Python runtime. So I was stuck with programming in NodeJS. And programming synchronous code in NodeJS did not feel like fun for me: I had to learn how promises work for code which would be a few lines of Python or Bash. When I look back, and there would be still no CodeBuild service, I would rather invoke a Bash or Python script from within the NodeJS runtime in order to avoid writing async code for synchronous program sequences.

Lambda function deployment

The code for Lambda functions is usually packed as ZIP file and stored in an S3 bucket. The location of the ZIP file is then referenced in the Lambda function. This is how it looks in CloudFormation, the Infrastructure-as-Code service from AWS:

1
2
3
4
5
6
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref DeploymentLambdaFunctionsBucket
S3Key: !Ref DeploymentLambdaFunctionsKey

That means there has to be another build and deployment procedure which packs and uploads the Lambda function code to S3 itself. Very much complexity for a build script which is usually a few lines of shell code, if you ask me.

By the way, actually there is a workaround: In CloudFormation, it’s possible to specify the code of the Lambda function inline in the template, like this:

1
2
3
4
5
6
7
8
LambdaFunctctionWithInlineCode:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
exports.handler = function(event, context) {
...
}

While this has the advantage that the pipeline and the build step code are now in one place (the CloudFormation template), this comes at the cost of losing e.g. IDE functions for the function code like syntax checking and highlighting. Another point: the inline code is limited to 4096 characters length, a limit which can be reached rather fast. Also the CloudFormation templates tend to become very long and confusing. In the end using inline code just felt awkward for me …

No AWS CLI installed in Lambda

Last but not least, there is no AWS CLI installed in the Lambda runtime, which makes things to be done in build steps, like uploading directories to S3, really hard, because this has to be done in the programming runtime. What would be a one-liner in AWS CLI, can be much more overhead and lines of code in e.g. NodeJS or Python.

At the recent re:invent conference, AWS announced CodeBuild which is a build service, very much like a managed version of Jenkins, but fully integrated into the AWS ecosystem. Here are a few highlights:

  • Fully integrated into AWS CodePipeline: CodePipeline is the “Deployment Pipeline” service from AWS and supports CodeBuild as an action in a deployment pipeline. It also means that CodePipeline can checkout code from e.g. a GitHub repository first, save it as output artifact and pass it to CodeBuild, so that the entire artifact handling is managed, no (un)zipping and S3 juggling necessary.
  • Managed build system based on Docker Containers: First you don’t need to take care of any Docker management. Second you can either use AWS provided images, which provide a range of operating systems / environments, e.g. Amazon Linux and Ubuntu for several pre-built environments, e.g. NodeJS or Python or Go (http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html). Or you can bring your own container (I did not try that out yet).
  • Fully supported by CloudFormation, the Infrastructure-as-Code service from AWS: You can codify CodeBuild projects so that they are fully automated, and reproducible without any manual and error-prone installation steps. Together with CodePipeline they form a powerful unit to express entire code pipelines as code which further reduces total cost of ownership.
  • YAML-DSL, which describes the build steps (as a list of shell commands), as well as the output artifacts of the build.

Another great point is that the provided images are very similar to the Lambda runtimes (based on Amazon Linux) so that they are predestinated for tasks like packing and testing Lambda function code (ZIP files).

CodeBuild in action

So, what are the particular advantages of using CodeBuild vs. Lambda in CodePipeline? Have a look at this Pull Request. It replaces the former Lambda-based approach with CodeBuild in the project I set up for my AWS Advent article: Several hundred lines of JavaScript got replaced by some lines of CodeBuild YAML. Here is how a sample build file looks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
version: 0.1
phases:
install:
commands:
- npm install -g serverless
- cd backend && npm install
build:
commands:
- "cd backend && serverless deploy"
- "cd backend && aws cloudformation describe-stacks --stack-name $(serverless info | grep service: | cut -d' ' -f2)-$(serverless info | grep stage: | cut -d' ' -f2) --query 'Stacks[0].Outputs[?OutputKey==`ServiceEndpoint`].OutputValue' --output text > ../service_endpoint.txt"
artifacts:
files:
- frontend/**/*
- service_endpoint.txt

This example shows a buildspec.yml with two main sections: phases and artifacts:

  • phases apparently lists the phases of the build. These predefined names actually have no special meaning and you can put as many and arbitrary commands into it. The example shows several shell commands executed, in particular first - in the install stage - the installation of the serverless NPM package, followed by the build stage which contains the execution of the Serverless framework (serverless deploy). Lastly, it runs a more complex command to save the output of a CloudFormation stack into a file called service_endpoint.txt: That file is later picked up as an output artifact.
  • artifacts lists the directories and files which CodePipeline will generate as an output artifact. Used in combination with CodePipeline, it provides a seamless integration into the pipeline and you can use the artifact as input for another pipeline stage or action. In this example the frontend folder and the mentioned service_endpoint.txt file are nominated as output artifacts.

The artifacts section can also be omitted, if there are no artifacts at all.

Now that we learned the basics of the buildspec.yml file, lets see how this integrates with CloudFormation:

CodeBuild and CloudFormation

CloudFormation provides a type AWS::CodeBuild::Project to describe CodeBuild projects - an example follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
DeployBackendBuild:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/eb-nodejs-4.4.6-amazonlinux-64:2.1.3
Type: LINUX_CONTAINER
Name: !Sub ${AWS::StackName}DeployBackendBuild
ServiceRole: !Ref DeployBackendBuildRole
Source:
Type: CODEPIPELINE
BuildSpec: |
version: 0.1
...

This example creates a CodeBuild project which integrates into a CodePipeline (Type: CODEPIPELINE), and which uses a AWS provided image for nodejs runtimes. The advantage is that e.g. NPM is preinstalled. The Source section describes again that the source code for the build is coming from a CodePipeline. The BuildSpec specifies in inline build specification (e.g. the one shown above).

You could also specify that CodeBuild should search for a buildspec.yml in the provided source artifacts rather than providing one via the project specification.

CodeBuild and CodePipeline

Last but not least, let’s have a look at how CodePipeline and CodeBuild integrate by using an excerpt from the CloudFormation template which describes the pipeline as code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
...
Stages:
- Name: Source
Actions:
- Name: Source
InputArtifacts: []
ActionTypeId:
Category: Source
Owner: ThirdParty
Version: 1
Provider: GitHub
OutputArtifacts:
- Name: SourceOutput
- Name: DeployApp
Actions:
- Name: DeployBackend
ActionTypeId:
Category: Build
Owner: AWS
Version: 1
Provider: CodeBuild
OutputArtifacts:
- Name: DeployBackendOutput
InputArtifacts:
- Name: SourceOutput
Configuration:
ProjectName: !Ref DeployBackendBuild
RunOrder: 1

This code describes a pipeline with two stages: While the first stage checks out the source code from a Git repository, the second stage is the interesting one here: It describes a stage with a CodeBuild action which takes the SourceOutput as input artifact, which ensures that the commands specified in the build spec of the referenced DeployBackendBuild CodeBuild project can operate on the source. DeployBackendBuild is the actual sample project we looked at in the previous section.

The Code

The full CloudFormation template describing the pipeline is on GitHub. You can actually test it out by yourself by following the instructions in the original article.

Summary

Deployment pipelines are as valuable as the software itself as they ensure reliable deployments, experimentation and fast time-to-market. So why shouldn’t we treat them like software, namely as code. With Codebuild, AWS completed the toolchain of building blocks which are necessary to codify and automate the setup of deployment pipelines for our software:

  • without the complexity of setting up / maintaining third party services
  • no error-prone manual steps
  • no management of own infrastructure like Docker clusters as “build farms”.
  • no bloated Lambda functions for build steps

This article showcases a CloudFormation template which should help the readers to get started with the own CloudFormation/CodePipeline/CodeBuild combo which provisions within minutes. There are no excuses anymore for manual and/or undocumented software deployments within AWS ;-)

Website now powered by Hexo, AWS CloudFront and S3

Over the past days I moved my blog over to AWS CloudFront and S3, powered by the static blog generator Hexo.

Here are a few highlights:

  • The source code of the website is now open source on GitHub.
  • The infrastructure for the website is automated and codified by a CloudFormation template.
  • The website is secured via HTTPS thanks to CloudFront and the Amazon Certificate Manager
  • The build of the website if entirely codified and automated with AWS CodePipeline and CodeBuild (see the CloudFormation template for details).
  • The website and building infrastructure are serverless. No servers, VMs or containers to manage.
  • Major performance enhancements since the website is now static and powered by a CDN.

New AWS CloudFormation YAML syntax and variable substitution in action

I’ve been using CloudFormation YAML syntax for a while now with Ansible and the serverless framework which would convert the YAML to JSON before uploading the template. That already gave me the YAML advantages of e.g. code comments, not having to care about commas etc.

A few days ago, AWS announced native YAML support for CloudFormation templates, in addition to the existing JSON format.

And along with that they added new shorthand syntax for several functions.

Let’s go through a template which I created not only in order to get used to the new syntax :)

Injecting “arguments” to inline Lambda functions

One of the real powers of Lambda and CloudFormation is that you can use Lambda to add almost any missing functionality to CloudFormation (e.g. custom resources), or to create small functions, without having to maintain another deployment workflow for the function (In this example I created an Lambda function which polls some web services and writes the result into a CloudWatch custom metric.)

The interesting part is how AccessPointName is injected into the Lambda function (in this example some Python code). We are making use of the new short substitution syntax here which allows us to replace CloudFormation references with their value:

CheckProgram:
  Type: AWS::Lambda::Function
  Properties:
    Code:
      ZipFile: !Sub |
        ...
        def handler(event, context):  
          ...
          found_access_points = [access_point for access_point in api_result["allTheRouters"] if access_point["name"] == "${AccessPointName}"]

In this example the variable “AccessPointName“ gets then substituted by the value (in this particular case a stack parameter). Please also mind the “|” which is no special CloudFormation syntax but multi line YAML syntax.

Throughout the
template
you can find other usage examples of the new substitution syntax, for example a cron job with CloudWatch events which gets:

CheckProgramTrigger:
  Type: AWS::Events::Rule
  Properties:
    ScheduleExpression: !Sub rate(${CheckRateMinutes} minutes)
    Targets:
      - Arn:
          !GetAtt [CheckProgram, Arn]
        Id: InvokeLambda

Referencing with the !Ref and !GetAttr shortcuts

Another feature addition is a short hand syntax for Ref and GetAttr calls.

AccessPointOfflineAlertTopic:
  Type: AWS::SNS::Topic
  Properties:
    Subscription:
      - Endpoint: !Ref NotificationEmail
        Protocol: email

This example creates an SNS topic with an email subscription which is once again a CloudFormation template parameter.

Recap

With the new syntax it’s now possible to create YAML syntax, and we have nice shortcuts for commonly used functions. My personal highlight is the shorthand substitution syntax, esp. when using inline Lambda functions.

How to install and use a newer version (3.x) of NPM on AWS Lambda.

My current experiment is to build a serverless deploy pipeline (With AWS CodePipeline) which uses AWS Lambda for the build steps. One step includes to invoke NPM to build a static website out of JavaScript components (which would be deployed to an S3 bucket in a later step).

Ok, so let’s go ahead and look what is actually installed in the preconfigured Node 4.3 env on AWS Lambda. First we want to find out if NPM is actually already installed. So we just create a new Lambda function which invokes a `find’ command, here is the used source code:

exports.handler = (event, context, callback) => {  
  var child_process = require('child_process'); 
  console.log(child_process.execSync('find /usr -name npm -type f', {encoding: 'utf-8'}));   
}; 

And, voila, we found something, here is the output:

/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm

So let’s try to execute it!

console.log(child_process.execSync('/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm version', {encoding: 'utf-8'}));

And here is the output:

module.js:327
    throw err;
    ^

Error: Cannot find module '/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/node_modules/npm/bin/npm-cli.js'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Function.Module.runMain (module.js:441:10)
    at startup (node.js:134:18)
    at node.js:962:3

    at checkExecSyncError (child_process.js:464:13)
    at Object.execSync (child_process.js:504:13)
    at exports.handler (/var/task/index.js:4:29)

Ok, that doesn’t look good, does it? Actually the ‘node_modules/npm/bin/node_modules/npm/bin/npm-cli.js’ part looks broken.

Ok, so my next step was to find the correct path to npm-cli.js, so I have a chance to call it without the apparently broken executable wrapper:

console.log(child_process.execSync('find /usr -type f -name npm-cli.js', {encoding: 'utf-8'}));


/usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm-cli.js

So let’s try to call it directly:

console.log(child_process.execSync('node /usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm-cli.js version', {encoding: 'utf-8'}));

gives us:

{ npm: '2.14.12',  ... }

Yay! We got NPM working!

But NAY, it’s an old version!

So let’s go ahead and try to install a newer version! Lambda gives us a writable /tmp, so we could use that as a target dir. NPM actually wants to do much stuff in the $HOME directory (e.g. trying to create cache dirs), but it is not writable within a Lambda env.

So my “hack” was to set the $HOME to /tmp, and then install a newer version of NPM into it (by using the --prefix option):

process.env.HOME = '/tmp';
console.log(child_process.execSync('node /usr/local/lib64/node-v4.3.x/lib/node_modules/npm/bin/npm-cli.js install npm --prefix=/tmp --progress=false', {encoding: 'utf-8'}));
console.log(child_process.execSync('node /tmp/node_modules/npm/bin/npm-cli.js version', {encoding: 'utf-8'}));

Ok, NPM got installed and is ready to use!

npm@3.10.5 ../../tmp/node_modules/npm

The last step is to symlink the npm wrapper so it can be used without hassle. And actually many build systems seem to expect a working npm executable:

fs.mkdirSync('/tmp/bin');
fs.symlinkSync('/tmp/node_modules/npm/bin/npm-cli.js', '/tmp/bin/npm');
process.env.PATH = '/tmp/bin:' + process.env.PATH;
console.log(child_process.execSync('npm version', {encoding: 'utf-8'}));

And here we go! Now it’s possible to use a up-to-date version of NPM within a Lambda function.

Some additional learnings:

  • NPM needs a lot of memory, so I configured the Lambda function with max memory of 1500MB RAM. Otherwise it seems to misbehave or garbage collect a lot.
  • You should start with a clean tmp before installing NPM in order to avoid side effects, as containers might get reused by Lambda. That step did it for me:
child_process.execSync('rm -fr /tmp/.npm');  
// ... npm install steps ...
  • Downloading and installing NPM every time the build step is executed makes it more flaky (remember the fallacies of networking!). It also reduces the available execution time by 10 seconds (the time it takes to download and install NPM). One could pack the installed npm version as an own Lambda function in order to decouple it. But that’s a topic for another blog post.