Hands-On: Sidecar | Topz (Designing Distributed Systems)

November 2, 2024 · 2 min read

Sidecar Hands-on Exercise: TOPZ

The purpose of this exercise is to explore the "sidecar" pattern of deploying software using containers and container groups.

This exercise has the reader creating an arbitrary containerised application and a sidecar in the same namespace. The sidecar provides resource monitoring for the virtual host.

The provided instructions use a purpose-built utility called topz, and leaves the main application ambiguous (to be determined by the implementer). The implementation is started using Docker CLI commands.

Below is the exercise as defined in the book's text:

Hands On: Deploying the topz Container To see the topz sidecar in action, you first need to deploy some other container to act as the application container. Choose an existing application that you are running and deploy it using Docker:

$ docker run -d <my-app-image>
<container-hash-value>
After you run that image, you will receive the identifier for that specific container. It will look something like: cccf82b85000…​ If you don’t have it, you can always look it up using the docker ps command, which will show all currently running containers. Assuming you have stashed that value in an environment variable named APP_ID, you can then run the topz container in the same PID namespace using:

$ docker run --pid=container:${APP_ID} \
    -p 8080:8080 \
    brendanburns/topz:db0fa58 \
    /server --addr=0.0.0.0:8080

This will launch the topz sidecar in the same PID namespace as the application container. Note that you may need to change the port that the sidecar uses for serving if your application container is also running on port 8080. Once the sidecar is running, you can visit http://localhost:8080/topz to get a complete readout of the processes that are running in the application container and their resource usage.

You can mix this sidecar with any other existing container to easily get a view into how the container is using its resources via a web interface.

Local Setup

I'm using the default installation of Rancher Desktop (v1.16.0) on Apple M1 Silicon, along with the provided dockerd container engine

Challenges

The author doesn't provide the main application for the exercise

In the absence of having a specific application to run, on the advice of my code-assistant AI, I've opted to use a standard NGINX container as my base application. This gives me some default behavior for the "main application" that can be observed to verify the setup
The provided image expects a Linux/AMD64 architecture. My code assistant AI has suggested two alternatives:
1. use an alternate sidecar application (cAdvisor)
2. Build a Multi-Archtecture Image of Topz

Since this is all practice, I will try both approaches

Solutions

Solution 01
Solution 02

Overview

This solution applies Nova's suggestion of replacing the topz sidecar (not compatible with either Mac OS or with the Mac Silicon hardware) with an alternate sidecar. It uses NGINX as the "main application", assuming some static content is being served, and cAdvisor as the sidecar.

Solution Artifacts

The solution is implemented as

an nginx.conf file that contains the service configuration
a docker-compose.yaml file that contains the docker network and container information

Solution Design

Running the solution

In the directory that contains the solution files: Run the following command to start the solution implementation (will run on port :8080)

docker-compose up -d

Screen captures

Screen capture of terminal output of docker-compose up command

Screen capture of Rancher Desktop UI with created containers running

Observe the solution

With the Docker stack running:

point your browser to localhost:8080 to see the default NGINX welcome page... this is a stand-in for the "main application"
point your browser to localhost:8080/cadvisor or localhost:8080/cadvisor/containers to see the UI of the cadvisor sidecar
Navigate the cadvisor page to be sure the links work, and external resources are loading (you should see graphs, gauges, etc. with data updating in real time)

caution

If running in Mac OS, you will see regularly recurring 500 errors, if you check the Network tabs. This is because cAdvisor is trying to invoke a command that runs the ps command-line utility to retrieve some additional system data. This utility is not available on Mac OS. But the absence of this data doesn't otherwise affect cAdvisor, and it remains a useful and valuable tool withoutt these calls.

Screen captures

Screen capture of NGINX default page standing-in as the "main application" in the sidecar pattern

_Note: navigating to /cadvisor (which has no actual content) redirects to /cadvisor/containers, which is the root of the cAdvisor content. Screen recording showing navigation to 'sidecar' app, with redirect from sidecar root to main page content

Screen capture with cAdvisor styles and fetched data displayed, indicating successful redirect of URLS in meta tags

Screen capture showing navigation with links, including style and data fetching success

Stopping the solution

Navigate to this folder in your clone of this repository Run the following command:

docker-compose down

Screen capture

Screen capture showing output of docker-compose down

Solution 01 Challenges

Path and Port Rewriting:

Managed cAdvisor’s links and assets that initially lacked correct paths and port information, leading to broken links and missing resources.

Handling Trailing Slashes and Redirects:

Configured Nginx to append trailing slashes on specific routes (top-level routes) that otherwise report a 404. Note that for routes below the top level (e.g., /containers/docker), including the trailing slash is an error
Addressed issues where URLs were incorrectly rewritten, causing multiple redirects or incorrect paths.

Caching and Profile-Specific Issues:

Resolved persistent caching issues where browser profiles retained outdated URL mappings, leading to debugging obstacles.

Optimizing Resource Buffering:

Increased Nginx buffer sizes and disabled disk buffering to handle the larger upstream data from cAdvisor, which required handling more data-intensive API responses. This was largely a developer-experience issue, as I like to view log output in [lnav](https://lnav.org/ "The LogFile Navigator | command-line log viewer) and the warnings announcing exceeding buffer sizes were disrupting the lnav display output.

Comments

This turned out to be as much or more an exercise in learning NGINX as a lesson in the sidecar pattern, but I seem to have gotten there in the end.

I've gone through several iterations, with only the final iteration captured here. The first iteration exposed all of the cAdvisor paths as top-level paths, just below the root, (e.g., /containers, /docker, etc.). It occurred to me to apply the sidecar as a general pattern, it was probably wiser to use a less-ambiguous path (some hypthetical 'main application' might conceivably require a containers path at the root), and to only occupy one namespace for the cadvisor paths, so the second solution takes that approach, using sub_filter directives to rewrite URLs in the HTML returned from the downstream cAdvisor service, so that all urls meant to redirect to the sidecar have paths prefixed with /cadvisor.

A good deal of this work was also teasing out which suggestions of the AI were valuable and which were not. For example, when I had some links that were not working correctly, the AI suggested rewriting all relative paths, which ended up having disastrous consequences for loading non-html resources like stylesheets and scripts, and proved unnecessary anyway. Another dubious AI suggestion was using $host:$server_port to force re-written URLs to have the correct port address. It turns out the correct solution was to use the $http_host variable, but the AI continued to suggest the other usage.

Taking a step back to more general observations, I find that the AI (in this case ChatGPT 4o) does pretty well with general purpose languages like JavaScript, but falters a bit with purpose-built languages like the configuration language in NGINX or other DSLs. Not sure if that observation will hold up to scrutiny over time. But I will keep it in mind.

Creating Your Own Multi-Architecture Topz Image

Prequisites

Docker Hub account and command-line access
A current docker login session

I'm using Rancher Desktop (v1.16.0), configured to use dockerd as the container engine. I'm working on a Macbook Pro with Apple Silicon (M1) running on Mac OS Sonoma 14.6. I'm using Docker Hub as my image repository, under the name iaindavisdotdev.

Activate QEMU for Cross-Platform Builds
- Run the following command to set up QEMU for cross-platform emulation.
  
  info
  This command installs QEMU emulation support in Docker, enabling cross-platform builds by allowing the host system to build and run images for architectures different from its own.
  docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
Create and Use a Multi-Architecture Builder:
- Initialize a new Buildx builder for multi-architecture without requiring any experimental flags.
```
docker buildx create --name multiarchbuilder --use
docker buildx inspect --bootstrap
```
Write a New Dockerfile Create a new Dockerfile to use brendanburns/topz:db0fa58 as the base:
```
# Start from the existing topz image
FROM brendanburns/topz:db0fa58
```
Build the Multi-Architecture Docker Image:
- Using brendanburns/topz:db0fa58 as a reference, build your local image. This example skips pushing to a registry:
```
docker buildx build --platform linux/amd64,linux/arm64 -t <docker-user>/topz-multiarch:latest --load .
```
Run the topz Container Locally:
- Once the build completes, run the container locally:
```
docker run -d --name topz-local <docker-user>/topz-multiarch:latest
```

Reverting Docker/Rancher Desktop to Default Settings

If you wish to undo the settings for multi-architecture builds, use the steps below:
1. Reset Rancher Desktop’s container engine to its original state, if it was modified.
2. Delete the mybuilder buildx instance (optional):
```
docker buildx rm mybuilder
```

By following these steps, you’ll have a multi-architecture topz image ready to use locally without making global configuration changes.

Sidecar Hands-on Exercise: TOPZ​

Local Setup​

Challenges​

Solutions​

Overview​

Solution Artifacts​

Solution Design​

Running the solution​

Observe the solution​

Stopping the solution​

Solution 01 Challenges​

Path and Port Rewriting:​

Handling Trailing Slashes and Redirects:​

Caching and Profile-Specific Issues:​

Optimizing Resource Buffering:​

Comments​

Creating Your Own Multi-Architecture Topz Image​

Prequisites​

Reverting Docker/Rancher Desktop to Default Settings​