Jekyll2023-04-12T17:37:38+10:00https://charleskorn.com/feed.xmlCharles KornThe ramblings of Charles Korn. I'm a developer at <a href="https://www.thoughtworks.com/">Thoughtworks</a> in Melbourne, and the maintainer of <a href="https://batect.dev">Batect</a>. I like improving the developer experience, large-scale systems, cloud-native architecture, internal platforms, travel, photography, chocolate and Lego.
Charles Kornme@charleskorn.comThe Melbourne Kotlin meetup is back!2022-07-27T21:41:00+10:002022-07-27T21:41:00+10:00https://charleskorn.com/2022/07/27/the-melbourne-kotlin-meetup-is-back<p>I’m super excited that <a href="https://www.meetup.com/melbourne-kotlin-meetup">the Melbourne Kotlin meetup</a> is back after a
COVID-induced hiatus, and I’m just as excited to be presenting!</p>
<p>Come down to the Block-AfterPay offices at 6pm on August 11 to hear Zenab Bohra present “Introduction to FP in Kotlin”,
and I’ll be talking about “Embedding Golang in a Kotlin app: how to do it and why you shouldn’t”.</p>
<p>For more details, and to RSVP, check out <a href="https://www.meetup.com/melbourne-kotlin-meetup/events/287443299/">the August meetup page</a>.</p>
<p><strong>Update</strong>: the recording of both talks are now <a href="https://www.youtube.com/watch?v=lFX43T1t6l8">available on YouTube</a>.
Zenab’s talk begins at <a href="https://youtu.be/lFX43T1t6l8?t=214">3:34</a>, and my talk starts at <a href="https://youtu.be/lFX43T1t6l8?t=2700">45:00</a>.</p>Charles Kornme@charleskorn.comI’m super excited that the Melbourne Kotlin meetup is back after a COVID-induced hiatus, and I’m just as excited to be presenting!How I collect telemetry from Batect users2022-07-09T15:48:00+10:002022-07-09T15:48:00+10:00https://charleskorn.com/2022/07/09/how-i-collect-telemetry-from-batect-users<p>I’m constantly asking myself a bunch of questions to help improve Batect: are Batect’s users having a good experience? What’s not working well? How are they using Batect?
What could Batect do to make their lives easier?</p>
<p>I first released <a href="https://batect.dev">Batect</a> back in 2017. Since then, the userbase has grown from a handful of people that could fit around a single
desk to hundreds of people all over the globe.</p>
<p>Back when everyone could fit around a desk, answering these questions was easy: I could stand up, walk over to someone, and ask them. (Or, if I was feeling particularly
lazy, I could stay seated and ask over the top of my screen.)</p>
<p>But with this growth, now I have another question: how do I break out of my bubble and feedback and ideas from those I don’t know?</p>
<p>This is where telemetry data comes into the picture. Besides signals like GitHub issues, discussions, suggestions and the surveys I occasionally run,
telemetry data has been invaluable to help me get a broader understanding of how users use Batect, the problems they run into and the environments they use Batect in.
This in turn has helped me prioritise the features and improvements that will have the greatest impact, and design them in a way that will be the most useful for Batect’s users.</p>
<p>In this (admittedly very long) post, I’m going to talk through what I mean by telemetry data, how I went about designing the telemetry collection system, Abacus, and finish
off with two stories of how I’ve used this data.</p>
<ul>
<li><a href="#telemetry">Telemetry?</a></li>
<li><a href="#key-design-considerations">Key design considerations</a></li>
<li><a href="#the-design">The design</a></li>
<li><a href="#examples-of-where-ive-used-this-data">Examples of where I’ve used this data</a>
<ul>
<li><a href="#shell-tab-completion">Shell tab completion</a></li>
<li><a href="#renovate">Renovate</a></li>
</ul>
</li>
<li><a href="#what-works-well-and-what-doesnt">What works well, and what doesn’t</a></li>
<li><a href="#in-closing">In closing</a></li>
</ul>
<h2 id="telemetry">Telemetry?</h2>
<p>Similar to Honeycomb’s definition of <a href="https://www.honeycomb.io/blog/so-you-want-to-build-an-observability-tool/">observability</a> for production systems, I wanted to be able to answer
questions about Batect and it’s users I haven’t even thought of yet, and do so without having to make any code changes and push out a new version of Batect.</p>
<p>This is exactly the problem that Batect’s telemetry data solves. Every time Batect runs, it collects information like:</p>
<ul>
<li>what version of Batect you’re using</li>
<li>what operating system you’re using</li>
<li>what version of Docker you’re using</li>
<li>a bunch of other interesting environmental information (what shell you use, whether you’re running Batect on a CI runner or not, and more)</li>
<li>data on the size of your project</li>
<li>what features you are or aren’t using</li>
<li>notable events such as if an update nudge was shown, or if an error occurred</li>
<li>timing information for performance-critical operations such as loading your configuration file</li>
</ul>
<p>…and more! There’s a full list of all the data Batect collects in <a href="https://batect.dev/privacy/#in-app-telemetry">its privacy policy</a>.</p>
<p>With this data, I can answer all kinds of questions. Here are just a few real examples:</p>
<ul>
<li>How long does it take for new versions of Batect to be adopted? (answer: anywhere from hours to months, depending on the user)</li>
<li>How many times does a user see an update nudge before they update Batect? (answer: lots - some users even seem to never respond to these)</li>
<li>Does using BuildKit really make a difference to image build times? (answer: yes)</li>
<li>Could adding shell tab completion really be useful? (answer: yes, humans make lots of typos)</li>
<li>Was adding shell tab completion actually useful? (answer: yes, it prevents lots of typos)</li>
<li>What’s the common factor behind this odd error that only a handful of users are seeing? (answer: one particular version of Docker)</li>
</ul>
<h2 id="key-design-considerations">Key design considerations</h2>
<p>There were a couple of key aspects I was considering as I sketched out what the telemetry collection system could look like, in rough priority order:</p>
<ul>
<li>
<p>Privacy and security: given the situations and environments where Batect is used, preserving users’ and organisations’ security
and privacy are critical. This is not only because it’s simply the right thing to do, but because any issues here would likely lead to a loss of trust and to them
blocking telemetry data or not using Batect, which is obviously not what I want.</p>
</li>
<li>
<p>User experience impact: Batect is part of developers’ core feedback loop, and so any changes that made it noticeably less reliable or less performant
were non-starters.</p>
</li>
<li>
<p>Cost: any expense to build or run the system would be coming out of my own pocket, so minimising the cost was important to me.</p>
</li>
<li>
<p>Ease of maintenance: this is something I largely maintain in my own time, so minimising the amount of ongoing care and feeding was another high priority.
Another aspect of maintenance was also designing something simple and easy to understand, so that when I come back to do any future maintenance, I wouldn’t need to spend
a lot of time re-learning a tool, service or library.</p>
</li>
<li>
<p>Flexibility: I want to be able to investigate new ideas and answer questions I haven’t thought of yet, as well as expand the data I’m collecting as
Batect evolves, and so I needed a system that supports these goals.</p>
</li>
<li>
<p>Learning: given I was building this on my own time, I also wanted to use this as an opportunity to learn more about a few technologies and techniques I hadn’t used much before.</p>
</li>
</ul>
<h2 id="the-design">The design</h2>
<p>It’s probably easiest to understand the overall design of the system by following the lifecycle of a single session.</p>
<p><a href="/images/2022/abacus-overview.svg"><img src="/images/2022/abacus-overview.svg" alt="Diagram showing main components in the system and the steps in data flow" /></a>
<em>Diagram showing main components in the system and steps in data flow (click to enlarge)</em></p>
<p>A <em>session</em> represents a single invocation of Batect - so if you run <code class="language-plaintext highlighter-rouge">./batect build</code> and then <code class="language-plaintext highlighter-rouge">./batect unitTest</code>, this will create two telemetry sessions,
one for each invocation.</p>
<p>1️⃣ As Batect runs, it builds up the session, recording details about Batect, the environment it’s running in and your configuration, amongst other things.
It also records particular events that occur, like when an error occurs or when it shows you a “new version available” nudge, and captures timing spans for
performance-sensitive operations like loading configuration files.</p>
<p>Just before Batect terminates, it writes this session to disk in the upload queue, ready to be uploaded in a future invocation.</p>
<p>2️⃣ When Batect next runs, it will launch a background thread to check for any outstanding telemetry sessions waiting in the upload queue.</p>
<p>3️⃣ Any outstanding sessions are uploaded to Abacus. Once each upload is confirmed as successful, the session is removed from disk.</p>
<p>4️⃣ Once Abacus receives the session, it performs some basic validation and writes the session to a Cloud Storage bucket. Each session is assigned a unique UUID by Batect
when it is created, so Abacus uses this ID to deduplicate sessions if required.</p>
<p>5️⃣ Once an hour, a BigQuery Data Transfer Service job checks for new sessions in the Cloud Storage bucket and writes them to a BigQuery table.</p>
<p>6️⃣ With the data now in BigQuery, I can either perform ad-hoc queries over the data to answer specific questions, or use Google’s free Data Studio to visualise and report on the data.</p>
<p>For example, I can report on what version of Batect is being used:</p>
<p><img src="/images/2022/abacus-batect-version.png" alt="Chart showing the proportion of each day's sessions, broken down by Batect version" /></p>
<p>Or the distribution of the number of tasks and containers in each project:</p>
<p><img src="/images/2022/abacus-project-size.png" alt="Chart showing the proportion of each day's sessions, broken down by Batect version" /></p>
<p>There a bunch of other important details I’ve skipped over here in the interest of brevity - topics like not collecting personally identifiable information,
or even potentially-identifiable information and managing consent - but these are all important aspects to consider if you’re thinking of building a similar
system yourself.</p>
<h2 id="examples-of-where-ive-used-this-data">Examples of where I’ve used this data</h2>
<p>Some of my favourite stories of how I’ve used this data are for prioritising Batect’s shell tab completion functionality and validating that it had the
intended impact, and for prioritising contributing support for Batect to <a href="https://github.com/renovatebot/renovate">Renovate</a>.</p>
<h3 id="shell-tab-completion">Shell tab completion</h3>
<p>Batect’s main user interface is the command line: to run a task, you execute a command like <code class="language-plaintext highlighter-rouge">./batect build</code> or <code class="language-plaintext highlighter-rouge">./batect test</code> from your shell.
One of the most frequently-requested features was support for shell tab completion, so that instead of typing out <code class="language-plaintext highlighter-rouge">build</code> or <code class="language-plaintext highlighter-rouge">test</code>, you could type
just the first letter or two, press <kbd>Tab</kbd>, and the rest of the task name would be completed for you.</p>
<p>While I could definitely see the value in this, I was worried I was over-emphasising features I’d like to see based on how I usually work and interact
with applications, and wanted to confirm that this would actually be valuable for a large portion of Batect’s userbase. I formed a hypothesis that shell
tab completion would help reduce the number of sessions that fail because the task name cannot be found, likely because of a typo.</p>
<p>So, I turned to the data I had and found that roughly 2.6% of sessions fail because the task cannot be found. This seemed very high and was enough for me to
feel confident investing some time and effort implementing <a href="https://batect.dev/docs/getting-started/shell-tab-completion">shell tab completion</a>.</p>
<p>Of course, the next question was which shell to focus on first: Bash? Zsh? My personal favourite, Fish? My gut told me most people would likely be using
Zsh or Bash as their default shell, but here the data told a different story: the shells used most commonly by Batect’s users are actually Fish and Zsh.
So that’s where I started - I added tab completion functionality for both Fish and Zsh - and, thanks to the data I had, I was able to prioritise my limited
time to focus on the shells that would help the largest number of users.</p>
<p>And, thanks once again to telemetry data, once I’d released this feature, I was also able to quickly validate my hypothesis: users that use shell tab completion experience
“task not found” errors over 30% less than those that don’t use it.</p>
<h3 id="renovate">Renovate</h3>
<p>Batect is driven by the <code class="language-plaintext highlighter-rouge">batect</code> and <code class="language-plaintext highlighter-rouge">batect.cmd</code> wrapper scripts that you drop directly into your project and commit alongside your code.</p>
<p>Most of the time, this is great: you can just clone a repository and start using it without needing to install anything, and you can be confident
everyone on your team is using exactly the same version.</p>
<p>However, there’s a drawback hiding in the last part of that sentence: this design also means that you or one of your teammates must upgrade the wrapper script
to use a new version of Batect. Based on both anecdotal evidence from users asking for features that already existed, as well as telemetry data, I could
see that many teams were not upgrading regularly. And from telemetry data, I also knew this was happening even while users were seeing an upgrade nudge like this
many hundreds of times a day:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Version 0.79.1 of Batect is now available (you have 0.74.0).
To upgrade to the latest version, run './batect --upgrade'.
For more information, visit https://github.com/batect/batect/releases/tag/0.79.1.
</code></pre></div></div>
<p>Armed with this information, I decided to try to help users upgrade more often and more rapidly - there was little point spending time building new features and fixing bugs if only a
small proportion of users benefited from these improvements.</p>
<p>My solution was to add support for Batect’s wrapper script to <a href="https://github.com/renovatebot/renovate">Renovate</a>. This means that projects that use Renovate will automatically receive
pull requests to upgrade to the new version whenever I release a new version of Batect, and upgrading is as simple as merging the PR.</p>
<p>Again, the data shows this has been a success: whereas previously, a new version would only slowly be adopted, now, new versions of Batect make up over 40% of sessions within a month
of release.</p>
<h2 id="what-works-well-and-what-doesnt">What works well, and what doesn’t</h2>
<p>The Abacus API is a single Golang application deployed as a container to GCP’s Cloud Run. I’ve been really happy with Cloud Run: it just works. My only wish is that it had better
built-in support for gradual rollouts of new versions. This is something that can be done with Cloud Run, but as far as I can tell, it requires some form of external orchestration
to monitor the new version and adjust the proportion of traffic going to the new and old versions as the rollout progresses.</p>
<p>Cloud Run also integrates nicely with GCP’s monitoring tooling. I wasn’t sure about using GCP’s monitoring tools at first - I was expecting that I’d quickly run into a variety of limitations
and rough edges like with AWS’ CloudWatch suite - but I’ve been pleasantly surprised. For something of this scale, the tools more than meet my needs, and I haven’t felt the need to consider any alternatives.
I recently added Honeycomb integration, but this was more out of curiousity and wanting to learn more about Honeycomb than a need to use another tool.</p>
<p>The pattern of dropping session data into Cloud Storage to be synced by BigQuery Transfer Service into BigQuery has also been more than sufficient for my needs. While this does introduce
a delay of up to two hours between data being received and it being ready to query in BigQuery, I rarely want to query data so rapidly, and the alternative of streaming data
would be both more complex and more costly.</p>
<p>All of this also almost fits within GCP’s free tier: the only item that regularly exceeds the free allowance is upload operations to Cloud Storage, as each individual session is uploaded
individually as a single object, and the free tier includes only 5,000 uploads per month. I could reduce this cost by batching sessions together and uploading a batch as a single file,
but this would again introduce more complexity and cost for little benefit.</p>
<p>On the client side, the concept of the upload queue has been really successful: uploading sessions in the background of a future invocation minimises the user impact of uploading this data,
and provides a form of resilience against any transient network or service issues. (If a session can’t be uploaded for some reason, Batect will try to upload it again
next time, or give up once the session is 30 days old.)</p>
<p>The biggest area for improvement is how I manage reports and dashboards. I’m currently using Data Studio for this, and while it too largely just works, there are a couple of key
features missing that I’d really like to see. For example, it’s not possible to version control the definition of a report, making changes risky, and doesn’t have built-in support for
visualisations like histograms or heatmaps, limiting how I can visualise some kinds of data.</p>
<h2 id="in-closing">In closing</h2>
<p>Collecting telemetry data is something I wish I’d added to Batect far earlier. But now that I have this rich data, I can’t imagine life without it. It’s helped me prioritise
my limited time and enabled me to deliver a more polished product to users.</p>
<p>On top of this, I’ve achieved my design goals for Abacus, including preserving users’ privacy and minimising the performance impact of collecting this data.
And I’ve learnt about services like BigQuery and Data Studio along the way.</p>
<p>If you’re interested in checking out the code for Abacus, it’s <a href="https://github.com/batect/abacus">available on GitHub</a>, as is
<a href="https://github.com/batect/batect/tree/main/libs/telemetry">the client-side telemetry library</a> used by Batect.</p>
<p><em>Updated July 10 to include more details of the tools and services behind Abacus.</em></p>Charles Kornme@charleskorn.comI’m constantly asking myself a bunch of questions to help improve Batect: are Batect’s users having a good experience? What’s not working well? How are they using Batect? What could Batect do to make their lives easier?Released: okhttp-system-keystore2022-04-24T13:23:00+10:002022-04-24T13:23:00+10:00https://charleskorn.com/2022/04/24/released-okhttp-system-keystore<p>I’ve just published <a href="https://github.com/charleskorn/okhttp-system-keystore/"><code class="language-plaintext highlighter-rouge">okhttp-system-keystore</code></a>, a small library that makes it easy
to use trusted certificates from the operating system keystore (Keychain on macOS, Certificate Store on Windows) with <a href="https://square.github.io/okhttp/">OkHttp</a>.</p>
<p>Feedback, suggestions and ideas are welcome!</p>Charles Kornme@charleskorn.comI’ve just published okhttp-system-keystore, a small library that makes it easy to use trusted certificates from the operating system keystore (Keychain on macOS, Certificate Store on Windows) with OkHttp.Why is Batect written in Kotlin?2022-02-07T09:19:00+11:002022-02-07T09:19:00+11:00https://charleskorn.com/2022/02/07/why-is-batect-written-in-kotlin<p>After quite a hiatus, I’m going to try to post here more regularly… let’s start with two topics close to my heart: Kotlin and Batect.</p>
<p>A fairly frequent question I get is: why is <a href="https://batect.dev/">Batect</a> written in Kotlin? And often the question behind the question is:
why isn’t it written in Golang?</p>
<p>To answer those questions we need to go back in time to 2017, when I first started working on Batect.</p>
<p>I had the idea for what would become Batect after working with a couple of different teams who were all trying
to use Dockerised development environments. And after going through the pain of trying to once again reinvent the same less-than-ideal setup
we’d had on a previous team, I decided to build a proof of concept for my idea.</p>
<p>At the time, all I wanted to do was quickly build a proof of concept. I was more concerned about building it quickly and having fun and learning
while I built it than anything else: I wasn’t expecting the proof of concept to be anything more than some throwaway code. (Famous last words.)
I had been dabbling in Kotlin for a while and thought this would be a great opportunity to play with a language I really liked.</p>
<p>Fast forward a bit and we come to late 2017. I’d finished the proof of concept in Kotlin and demoed it to my team for feedback. They gave some really
positive feedback and that encouraged me to seriously consider turning Batect into something more than a proof of concept.</p>
<p>At this point, I had a codebase that could run a sample project, a working interface to Docker that invoked the <code class="language-plaintext highlighter-rouge">docker</code> command to pull images, create
containers etc., and some basic tests. I had two choices: continue with this largely working app, or throw it away and start afresh.</p>
<p>As part of considering whether or not to start afresh, I thought for a while about whether to continue in Kotlin or switch to Golang.
There were four things that were going through my mind:</p>
<p><strong>I had something that worked</strong> and a codebase that was in relatively good shape.</p>
<p>Switching to Golang would require starting from scratch. Given I was
doing this entirely on my own time, that didn’t seem like a great use of time, especially for something that still didn’t have any active users.</p>
<p>The main argument in favour of Golang was the fact that <strong>the vast majority of the Docker ecosystem is written in Golang</strong>.</p>
<p>If Batect was written in Golang, I could take advantage of things such as the Docker client library for Golang, rather than write my own for Kotlin.
I was using the <code class="language-plaintext highlighter-rouge">docker</code> CLI to communicate with the Docker daemon from Kotlin and changing to use <a href="https://docs.docker.com/engine/api/v1.41/">the HTTP API</a>
didn’t seem <em>that</em> difficult if needed later.</p>
<p>Another argument in favour of Golang was <strong>the ability to build a single self-contained binary</strong> for distribution, rather than requiring users to install
a JVM.</p>
<p>However, <a href="https://blog.jetbrains.com/kotlin/2017/04/kotlinnative-tech-preview-kotlin-without-a-vm/">earlier that year</a>, JetBrains had
announced the first preview of Kotlin/Native, which would allow the same thing for Kotlin code. Younger, naïve-r me assumed that would be good enough
and easy enough to adopt in the future should the use of a JVM really turn out to be a problem.</p>
<p>The last thing was simply <strong>personal preference</strong> and how much I enjoyed working with the language.</p>
<p>Again, this was something I was doing on my own time and had no active users yet, so personal enjoyment was one of the main priorities for me.
At the time, I was working on a production Golang system and was finding the language somewhat lacking in comparison to Kotlin. Dependency management in
Golang was a pain (this would be fixed with the introduction of <a href="https://go.dev/doc/go1.11#modules">modules in Go 1.11</a> in August the next year), and I found
Kotlin’s syntax and type system enabled me to write expressive, safe code.</p>
<p>So I chose to continue in Kotlin.</p>
<p>Looking back on that decision, there are definitely some things that remain true today, and others where the situation turned out to be
a bit different:</p>
<p><strong>Not rewriting Batect from scratch</strong> meant I was able to continue adding new features and incorporating feedback. This meant that Batect was ready to
introduce to a new team in late 2017. This team chose not only to take a risk and adopt a completely unproven tool, but continue to this day to be some
of its strongest advocates. If I’d stopped to rewrite Batect in Golang, I would have missed that opportunity.</p>
<p><strong>Using the <code class="language-plaintext highlighter-rouge">docker</code> CLI worked reasonably well for quite some time</strong>, but eventually the performance hit of spawning new processes to interact with the Docker
daemon was starting to have a noticeable impact.</p>
<p>Switching to use the API directly was, sadly, not as straightforward as I hoped. In particular, I had failed to consider the client-side complexities of some
of Docker’s features, such as connection configuration management, image registry credentials and managing the terminal while streaming I/O to and from the daemon.</p>
<p>This has played out over and over again, and has been a significant drain on my time, especially when it came to adding support for BuildKit, which is largely
undocumented, requires extensive client-side logic to implement correctly and relies on
<a href="https://github.com/batect/batect/commit/98262d74c3e26b36b9d89eebb2838c48365e68d5">a number of Golang idiosyncrasies</a>.</p>
<p><strong>Kotlin/Native sadly hasn’t matured as quickly as I expected.</strong> So Batect still requires a JVM, and this adds a small barrier to entry for some people. Having
said that, JetBrains is still actively developing Kotlin/Native and significant progress has been made in the last 12 months or so, so I remain hopeful that
removing the need for a JVM is still an achievable goal. This will not be painless – Batect has dependencies on some JVM-only libraries at the moment –
but it certainly seems within reach.</p>
<p>The last point will always be a matter of personal opinion, but <strong>Kotlin remains my favourite language to this day.</strong></p>
<p>Every now and then I question my choice and whether it was the right decision to continue building Batect in Kotlin. While some things may well have been easier had
I chosen to use Golang, Batect has hundreds of active users who love using it, and I still enjoy working on it after all this time, and those are the two things that
matter most to me.</p>
<p><em>Thank you to Andy Marks, Inny So and Jo Piechota for providing feedback on a draft version of this post.</em></p>Charles Kornme@charleskorn.comAfter quite a hiatus, I’m going to try to post here more regularly… let’s start with two topics close to my heart: Kotlin and Batect.Build and testing environments as code: because life’s too short not to2018-07-03T20:45:00+10:002018-07-03T20:45:00+10:00https://charleskorn.com/2018/07/03/build-and-testing-environments-as-code<p>Last month I spoke at ThoughtWorks Australia’s inaugural ‘Evolution by ThoughtWorks’ conference in Melbourne and Sydney,
presenting on <em>Build and testing environments as code: because life’s too short not too</em>.</p>
<p>The slides and video from the talk are now available
<a href="https://www.thoughtworks.com/evolution-by-thoughtworks/content#Presentations">on the ThoughtWorks website</a>.</p>
<p>The sample environment I used during the presentation is <a href="https://github.com/charleskorn/batect-sample-java">available on GitHub</a>,
and you can find more information about batect on its <a href="https://github.com/charleskorn/batect">GitHub page</a>.</p>Charles Kornme@charleskorn.comLast month I spoke at ThoughtWorks Australia’s inaugural ‘Evolution by ThoughtWorks’ conference in Melbourne and Sydney, presenting on Build and testing environments as code: because life’s too short not too.Build and testing environments as code: because life’s too short not to2018-01-31T21:10:00+11:002018-01-31T21:10:00+11:00https://charleskorn.com/2018/01/31/build-and-testing-environments-as-code<p>Last night I spoke at the <a href="https://www.meetup.com/devops-melbourne/events/246634083/">DevOps Melbourne meetup</a>,
presenting on <em>Build and testing environments as code: because life’s too short not too</em>.</p>
<p>The slides are available to <a href="/files/2018/Build and testing environments as code.pdf">download as a PDF</a>,
the sample environment I used during the presentation is <a href="https://github.com/charleskorn/batect-sample-java">available on GitHub</a>,
and you can find more information about batect on its <a href="https://github.com/charleskorn/batect">GitHub page</a>.</p>Charles Kornme@charleskorn.comLast night I spoke at the DevOps Melbourne meetup, presenting on Build and testing environments as code: because life’s too short not too.Docker-based development environment IDE support2017-05-25T17:54:00+10:002017-05-25T17:54:00+10:00https://charleskorn.com/2017/05/25/docker-based-development-environment-ide-support<p>I’ve been talking quite a bit lately about Docker-based build environments
(<a href="https://charleskorn.com/2017/01/17/dockers-not-just-for-production-using-containers-for-your-development-environment/">in Hamburg</a>,
<a href="/2017/03/30/dockers-not-just-for-production-using-containers-for-your-development-environment/">in Munich</a>, and at many of our clients).</p>
<p>One of the biggest drawbacks of the technique is the poor integration story for IDEs. Many IDEs require that your build environment
(eg. target JVM and associated build tools) is installed locally to enable all of their useful features like code completion and test runner
integration. But if this is isolated away in a container, the IDE can’t access it, so all these handy productivity features won’t work.</p>
<p>However, it looks like JetBrains in particular is starting to integrate these ideas into their products more:</p>
<ul>
<li>WebStorm will now allow you to configure a ‘remote’ Node.js interpreter in a local Docker image (<a href="https://blog.jetbrains.com/webstorm/2017/04/quick-tour-of-webstorm-and-docker/">details here</a>)</li>
<li>RubyMine takes this one step further: you can configure a Ruby interpreter based on a service definition in a Docker Compose
file (<a href="https://blog.jetbrains.com/ruby/2017/05/rubymine-2017-2-eap-1-docker-compose/">details here</a>). A similar feature is available
for Python in PyCharm.</li>
</ul>
<p>Both of these are great steps forward, and if you’re using Docker-based build environments, I’d encourage you to take a look at this.</p>
<p>Now, if only they’d do this for JVMs in IntelliJ…</p>Charles Kornme@charleskorn.comI’ve been talking quite a bit lately about Docker-based build environments (in Hamburg, in Munich, and at many of our clients).Docker’s not just for production: using containers for your development environment2017-03-31T05:00:00+11:002017-03-31T05:00:00+11:00https://charleskorn.com/2017/03/31/dockers-not-just-for-production-using-containers-for-your-development-environment<p>Tonight I spoke at the <a href="https://www.meetup.com/ThoughtWorks-Munich/events/238297588/">ThoughtWorks Munich meetup</a>,
presenting an updated version of my presentation <em>Docker’s not just for production: using containers for your development environment</em>.
There are two major changes over the <a href="/2017/01/17/dockers-not-just-for-production-using-containers-for-your-development-environment/">previous version</a>:</p>
<ul>
<li>I’ve added a section about using containers to manage test environments, in addition to the existing content about build environments</li>
<li>I’ve reworked the structure of the presentation based on feedback to focus more on the technique itself rather than a discussion of existing alternatives</li>
</ul>
<p>The slides are available to <a href="/files/2017/Docker for development environments v2.pdf">download as a PDF</a>,
and the sample environment I used during the presentation is <a href="https://github.com/charleskorn/docker-dev-env">available on GitHub</a>.</p>Charles Kornme@charleskorn.comTonight I spoke at the ThoughtWorks Munich meetup, presenting an updated version of my presentation Docker’s not just for production: using containers for your development environment. There are two major changes over the previous version:Docker’s not just for production: using containers for your development environment2017-01-18T06:00:00+11:002017-01-18T06:00:00+11:00https://charleskorn.com/2017/01/18/dockers-not-just-for-production-using-containers-for-your-development-environment<p>Tonight I spoke at the <a href="https://www.meetup.com/ThoughtWorks-Hamburg/events/235900641/">ThoughtWorks Hamburg meetup</a>,
presenting <em>Docker’s not just for production: using containers for your development environment</em>.</p>
<p>The slides are available to <a href="/files/2017/Docker for development environments.pdf">download as a PDF</a>,
and the sample Docker-based development environment I used during the presentation is
<a href="https://github.com/charleskorn/docker-dev-env-sample-app">available on GitHub</a>.</p>Charles Kornme@charleskorn.comTonight I spoke at the ThoughtWorks Hamburg meetup, presenting Docker’s not just for production: using containers for your development environment.A deeper look at the STM32F4 project template: the sample application2016-08-14T20:45:00+10:002016-08-14T20:45:00+10:00https://charleskorn.com/2016/08/14/a-deeper-look-at-the-stm32f4-project-template-the-sample-application<p><a href="/2016/05/08/a-deeper-look-at-the-stm32f4-project-template-linking-it-all-together/">Over three months ago</a> (yeah… sorry about that) we took a look at the linker script for the <a href="https://github.com/charleskorn/stm32f4-project-template">STM32F4 project template</a>. I promised that we’d examine <a href="https://github.com/charleskorn/stm32f4-project-template/blob/master/main/main.cpp">the sample application</a> next.</p>
<p>Note: the sample app assumes that you’re using the STM32F4 Discovery development board. If you’re not using that board, you should still be able to easily follow along, but some of the pin assignments might be slightly different.</p>
<p>The app is pretty straightforward: all it does is blink the four LEDs on the board in sequence, like this:</p>
<p><img src="https://github.com/charleskorn/stm32f4-project-template/raw/master/doc/flashing-leds.gif" alt="Blinking LEDs" /></p>
<p>If you’ve familiar with Arduinos, you’d probably expect to have something along these lines, perhaps without the repetition:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_1</span><span class="p">,</span> <span class="n">HIGH</span><span class="p">);</span>
<span class="n">delay</span><span class="p">(</span><span class="mi">1000</span><span class="p">);</span>
<span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_1</span><span class="p">,</span> <span class="n">LOW</span><span class="p">);</span>
<span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_2</span><span class="p">,</span> <span class="n">HIGH</span><span class="p">);</span>
<span class="n">delay</span><span class="p">(</span><span class="mi">1000</span><span class="p">);</span>
<span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_2</span><span class="p">,</span> <span class="n">LOW</span><span class="p">);</span>
<span class="n">digitalWrite</span><span class="p">(</span><span class="n">LED_3</span><span class="p">,</span> <span class="n">HIGH</span><span class="p">);</span>
<span class="n">delay</span><span class="p">(</span><span class="mi">1000</span><span class="p">);</span>
<span class="p">...</span>
<span class="p">}</span></code></pre></figure>
<p>However, I’ve taken a different approach. While it’s definitely possible to do something like that, I wanted to illustrate the use of the timer hardware and interrupts.</p>
<h1 id="setup">Setup</h1>
<p>Most of the work is just configuring all the peripherals we need, and this all happens in <code class="language-plaintext highlighter-rouge">main()</code>:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">enableGPIOD</span><span class="p">();</span>
<span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">pinCount</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">enableOutputPin</span><span class="p">(</span><span class="n">GPIOD</span><span class="p">,</span> <span class="n">pins</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="p">}</span>
<span class="n">enableTIM2</span><span class="p">();</span>
<span class="n">enableIRQ</span><span class="p">(</span><span class="n">TIM2_IRQn</span><span class="p">);</span>
<span class="n">enableTimerUpdateInterrupt</span><span class="p">(</span><span class="n">TIM2</span><span class="p">);</span>
<span class="n">setPrescaler</span><span class="p">(</span><span class="n">TIM2</span><span class="p">,</span> <span class="mi">16</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span> <span class="c1">// Set scale to microseconds, based on a 16 MHz clock</span>
<span class="n">setPeriod</span><span class="p">(</span><span class="n">TIM2</span><span class="p">,</span> <span class="n">millisecondsToMicroseconds</span><span class="p">(</span><span class="mi">300</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
<span class="n">enableAutoReload</span><span class="p">(</span><span class="n">TIM2</span><span class="p">);</span>
<span class="n">enableCounter</span><span class="p">(</span><span class="n">TIM2</span><span class="p">);</span>
<span class="n">resetTimer</span><span class="p">(</span><span class="n">TIM2</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>What does this all mean? Why is it necessary? (I’ve omitted the individual method definitions above and below for the sake of brevity, but you can find them in <a href="https://github.com/charleskorn/stm32f4-project-template/blob/master/main/main.cpp">the source</a>.)</p>
<ul>
<li>
<p><code class="language-plaintext highlighter-rouge">enableGPIOD()</code>: Like most peripherals on ARM CPUs, GPIO banks (groups of I/O pins) are turned off by default to save power. All four of the LEDs are in GPIO bank D, so we need to enable it.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">enableOutputPin(GPIOD, pins[i])</code>: just like <code class="language-plaintext highlighter-rouge">pinMode()</code> for Arduino, we need to set up each GPIO pin. Each pin can operate in a number of modes, so we need to specify which mode we want to use (digital input and output are the most common, but there are some other options as well).</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">enableTIM2()</code>: just like for GPIO bank D, we need to enable the timer we want to use (<code class="language-plaintext highlighter-rouge">TIM2</code>). We’ll use the timer to trigger changing which LED is turned on at the right moment.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">enableIRQ(TIM2_IRQn)</code> and <code class="language-plaintext highlighter-rouge">enableTimerUpdateInterrupt(TIM2)</code>: in addition to enabling the <code class="language-plaintext highlighter-rouge">TIM2</code> hardware, we also need to enable its corresponding IRQ, and select which events we want to receive interrupts for. In our case, we want timer update events, which occur when the timer reaches the end of the time period we specify.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">setPrescaler(TIM2, 16 - 1)</code>: timers are based on clock cycles, so one clock cycle equates to one unit of time. However, that’s usually not a convenient scale to use – we’d prefer to think in more natural units like microseconds or milliseconds. So the timers have what is called a prescaler: something that scales the clock cycle time units to our preferred time units.</p>
<p>In our case, the CPU is running at 16 MHz, so setting the prescaler value to 16 sets up a 16:1 scaling – 16 CPU cycles is one timer time unit. But there’s an additional complication: the value we set in the register is not exactly the divisor used. The divisor used is actually one more than the value we set, so we set the prescaler value to 15 to achieve a divisor of 16.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">setPeriod(TIM2, millisecondsToMicroseconds(300) - 1)</code>: this does exactly what it says on the tin. We want the timer to fire every 300 ms, so we configure the timer’s period, or auto-reload value, to be 300 ms.</p>
<p>The reason it’s called an ‘auto-reload value’ is due to how the timer works internally. The timer counts down ticks until its counter reaches zero, at which point the timer update interrupt fires. Once the interrupt has been handled, the auto-reload value is loaded into the counter, and the timer starts counting down again. So by setting the auto-reload value to our desired period, we’ll receive interrupts at regular intervals.</p>
<p>And, just like the prescaler value, the value used is one more than the value we set, so we subtract one to get the interval we’re after.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">enableAutoReload(TIM2)</code>: we need to enable resetting the counter with the auto-reload value, otherwise the timer will count down to zero and then stop.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">enableCounter(TIM2)</code>: the counter won’t actually start updating its counter in response to CPU cycles until we enable the counter</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">resetTimer(TIM2)</code>: any changes we make in the timer configuration registers don’t take effect until we reset the timer, at which point it pulls in the values we’ve just configured.</p>
</li>
</ul>
<p>So, after all that, we’ve setup the GPIO pins for the LEDs and configured the timer. Now all we have to do is wait for the timer interrupt to fire, and then we’ll change which LED is turned on.</p>
<p>You might be wondering how to find out what you need to do to use a piece of hardware. After all, there was a lot of stuff that needed to be done to set up that timer, and not all of it was particularly intuitive. The answer is usually a combination of trawling through the datasheet, looking at examples provided by the manufacturer (ST in this case) and Googling.</p>
<h1 id="timer-interrupt-handling">Timer interrupt handling</h1>
<p>In comparison to the configuration of everything, actually responding to the timer interrupts and blinking the LEDs is relatively straightforward.</p>
<p>First of all, we need an interrupt handler:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">extern</span> <span class="s">"C"</span> <span class="p">{</span>
<span class="kt">void</span> <span class="n">TIM2_IRQHandler</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">TIM2</span><span class="o">-></span><span class="n">SR</span> <span class="o">&</span> <span class="n">TIM_SR_UIF</span><span class="p">)</span> <span class="p">{</span>
<span class="n">onTIM2Tick</span><span class="p">();</span>
<span class="p">}</span>
<span class="n">resetTimerInterrupt</span><span class="p">(</span><span class="n">TIM2</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Because this method is called directly by the <a href="/2016/04/17/a-deeper-look-at-the-stm32f4-project-template-getting-things-started/">startup assembly code</a>, we have to mark it as <code class="language-plaintext highlighter-rouge">extern "C"</code>. This means that the method uses C linkage, which prevents C++’s name-mangling from changing the name. We don’t want the name to be changed because we want to be able to refer to it by name in the assembly code. <a href="http://stackoverflow.com/questions/1041866/in-c-source-what-is-the-effect-of-extern-c">This Stack Overflow question</a> has a more detailed explanation if you’re interested.</p>
<p>The handler itself is relatively straightforward:</p>
<ul>
<li>we check if the reason for the interrupt is the update event we’re interested in</li>
<li>if it is, we call out to our handler function <code class="language-plaintext highlighter-rouge">onTIM2Tick()</code></li>
<li>we reset the timer interrupt – otherwise our interrupt handler will be called again straight away</li>
</ul>
<p><code class="language-plaintext highlighter-rouge">onTIM2Tick()</code> is also straightforward:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">onTIM2Tick</span><span class="p">()</span> <span class="p">{</span>
<span class="n">lastPinOn</span> <span class="o">=</span> <span class="p">(</span><span class="n">lastPinOn</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="n">pinCount</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="k">auto</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">pinCount</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">BitAction</span> <span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="n">i</span> <span class="o">==</span> <span class="n">lastPinOn</span> <span class="o">?</span> <span class="n">Bit_SET</span> <span class="o">:</span> <span class="n">Bit_RESET</span><span class="p">);</span>
<span class="n">GPIO_WriteBit</span><span class="p">(</span><span class="n">GPIOD</span><span class="p">,</span> <span class="mi">1</span> <span class="o"><<</span> <span class="n">pins</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">value</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>All we do is loop over each of the four LEDs, turning on the next one and turning off all of the others. (<code class="language-plaintext highlighter-rouge">GPIO_WriteBit()</code> is a function from the standard peripherals library that does exactly what it sounds like.)</p>
<h1 id="the-end">The end</h1>
<p>That’s all there is to it – a lot of configuration wrangling and then it’s smooth sailing. Next time (which hopefully won’t be in another three months), we’ll take a quick look at the build system in the project template.</p>Charles Kornme@charleskorn.comOver three months ago (yeah… sorry about that) we took a look at the linker script for the STM32F4 project template. I promised that we’d examine the sample application next.