Optimizing Skaffold's Local Builds For Dotnet

Motivation

#

This article stands on its own, but if you’ve been following this series then you will have noticed something annoying after the previous article.

In that article, we meshed our services with linkerd, and to do that we made a script that would create four new local tls certificates every time we run skaffold dev. But, if you watch closely you’ll notice that skaffold rebuilds everything a ton of times when you run skaffold dev; it’s so bad that it makes it hard to shut down the process sometimes!

This happens because of skaffolds dev loop and specifically the file watcher. You see, because of how our common/skaffold.yaml is set up, Skaffold thinks that every file in the solution is a dependency for each project. So, when any file changes anywhere, skaffold will rebuild every project you have running, rededploy, etc… This is especially bad when we create 4 new certifiates and it kicks off the dev loop four times.

So, let’s look a little at how Skaffold does this and make some optimizations (and concessions… sigh) to our common/skaffold.yaml to optimize this. It’s not a great solution for dotnet projects, but it’s what we have.

Goals

#

• Fix our build section in our skaffold.yaml so Skaffold only watches actual dependencies for each project
  • reduce the number of build / deploy loops that happen locally
• Fix our Dockerfile(s) in our dotnet projects within our solution to go along with this solution
  • This is great until it’s not…
  • Right now there’s no problem…
  • If your projects are highly independent and they don’t reference one another within the solution, this approach is excellent
  • If you need a shared library from within this solution, then some sort of concession must be made… but we won’t get there today

Fix common/skaffold.yaml

#

The Problem

#

The problem is if we make a change in a file in one project, we expect skaffold to only rebuild and deploy that project, but instead it will rebuild and deploy all projects. Go run skaffold dev and make a small change in your Users.Service and notice how all projects are rebuilt and deployed. We want only the Users.Service to rebuild and deploy.

The cause is our build context in the build section in skaffold.yaml:

build:
  artifacts:
    - image: hcgaron/identity-service-starter
      context: ../../../
      docker:
        dockerfile: Identity.Service/Dockerfile
    - image: hcgaron/users-service-starter
      context: ../../../
      docker:
        dockerfile: Users.Service/Dockerfile
# ...

Notice that we have set the build context to the root of our solution (which we did in a previous article).

The build context tells skaffold what the dependencies of this project will be, so any file in the context directory (or any sub-directory) is considered a dependency for each project. The skaffold file watcher watches for changes, and rebuilds any project that is dependent on whichever file changed. We have it set so every project depends on every file in the solution… so every change rebuilds ALL OUR PROJECTS.

Now before you get angry and say “why did you tell me to do it that way!?”, remember that we were just updating the build context to be consistent with what skaffold assumes, that the build context will be the root of the project. We also were just following the bootstrapped dotnet convention; the dotnet scaffolding tool (or whatever makes that original dockerfile) creates the dockerfile with the assumption that you will have the root of the solution as your build context.

Here’s the dockerfile you get out of the box:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /src
COPY ["Identity.Service/Identity.Service.csproj", "Identity.Service/"]
RUN dotnet restore "Identity.Service/Identity.Service.csproj"
COPY . .
WORKDIR "/src/Identity.Service"
RUN dotnet build "Identity.Service.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "Identity.Service.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Identity.Service.dll"]

Line 8 above copies the .csproj into the docker container, but notice that it needs to navigate into the Identity.Service project first! Then, line 10 is COPY . ., which means it copies EVERYTHING from the solution into the /src directory (set as our WORKDIR on line 7) of our container.

Think about all that extra garbage we don’t need bloating our (build) image!

Luckily our final image doesn’t contain all that, but this is majorly wasteful and will slow down our build if our solution gets really large. Plus, having this build context is what causes all our extra skaffold builds / deploys in the dev loop.

The Solution

#

Fixing skaffold.yaml

#

We will fix our common/skaffold.yaml so that the build.artifacts.context (ie, the docker build context) points to the root of each project folder, not the root of the solution, and make sure the docker.dockerfile property points just to Dockerfile (which is relative to the context we set):

#...
build:
  artifacts:
    - image: hcgaron/identity-service-starter
      context: ../../../Identity.Service
      docker:
        dockerfile: Dockerfile
    - image: hcgaron/users-service-starter
      context: ../../../Users.Service
      docker:
        dockerfile: Dockerfile
    - image: hcgaron/web-bff-starter
      context: ../../../BFF.Web
      docker:
        dockerfile: Dockerfile
#...

Fixing Dockerfile

#

We need to make sure the Dockerfile for each project does two things:

• References files in the correct place, relative to the build context at the project root
• Copies files from the correct directory into the container

Here’s the new Dockerfile for one project… do the same for all the projects:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /src
COPY ["Identity.Service.csproj", "Identity.Service/"]
RUN dotnet restore "Identity.Service/Identity.Service.csproj"

WORKDIR "/src/Identity.Service"
COPY . .

RUN dotnet build "Identity.Service.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "Identity.Service.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Identity.Service.dll"]

Notice how on line 8 we’ve removed the prefix from the first entry in the COPY array, because that file is now at the root of our build context.

For the same reason, we’ve swapped lines 11 and 12. This is because we only want to COPY the files from the Identity.Service into the container. This not only speeds up our build, but if we didn’t do this, docker build would throw an error, because you can’t access files outside the build context (but, later in this series we might do just that…).

Go run skaffold dev -f ./K8S/skaffold/skaffold.yaml, then after everything is deployed make a small change in one project… SUCCESS, notice that only one project was rebuilt and deployed! No more garbage!

Next Steps

#

One last bit of architecture stuff to take care of before getting into some real coding…

Up next we will enable TSL / HTTPS for our Kubernetes ingress!

After that, I think it’s time we started on scalable and secure authentication. Up next we will start creating logic in our Identity.Service and our BFF.Web project to allow authentication at the ingress. Let’s GOOOOOO!