Hosting a CTF - UIUCTF'21 Overview + Infra

Ankur Sundara -

UIUCTF'21 has reached an end! With over 1,000 teams registering, and 5,500+ unique IP addresses, this has been our largest event yet.

I set up a majority of the infrastructure for this event, along with @kuilin.

In this post I'll give an overview of the event, infrastructure we used, challenges we faced, and overall thoughts.

Scoring

This year, for UIUCTF we chose to use dynamic scoring for challenges

A few reasons:

  • no longer have to accurately determine challenge difficulty
    • unintended solutions do not break the scoring
  • fewer ties
  • more granular scoring

The main concern we had with dynamic scoring is that it would be hard for beginners to know what challenges were approachable at first (before the point values for the challenges started to settle)

We addressed this by adding beginner and extreme tags to select challenges, allowing us to indicate relative difficulty, without setting a static score value. This worked well, and our tags ended up being pretty accurate.

Tagged Challenges

Stats / Scoring

Final Scoreboard

The 1st-place team, Super Guesser, finished with a grand total of 10995 points!

Team Score Distribution

Our score distribution was definitely very left-leaning, with the vast majority of teams scoring <250pts, and extremely few teams scoring > 2000 points.

We had a decent solve distribution, in my opinion, with challenges receiving different amounts of solves. This is good, because it hopefully meant that there were challenges of appropriate difficulty for everyone that participated.

Hardest challenges:

  • pwnyIDE (web, 0 solves)
  • wasmcloud (web, 0 solves)
  • ropfuscated (rev, 1 solve)
  • bpf_badjmp (kernel, 1 solve)
  • PwnyOS: Hidden Hard Drive (kernel, 2 solves)

Infra

For infrastructure, the main things we utilized were:

  • Google Cloud Platform - where we deployed everything. Google offers CTF sponsorship, which we applied for and received, allowing us to pay nothing for hosting costs. Thanks Google :)
  • kCTF - a platform for deploying and running challenge containers on kubernetes (k8s). Developed by people @ Google for googleCTF. Note that kCTF only works with GCP
  • CTFd - a popular CTF platform (the website, where participants viewed challenge descriptions and downloaded challenge files)
  • ctfcli - a CLI tool made by CTFd to store challenge information (description, flag, hints, etc.) in a .yml file, and push the information to CTFd
  • github actions - we used github actions for Continuous Integration, allowing us to deploy updated challenge prompts to CTFd

GCP Gotcha's

  • quotas, quotas, quotas - GCP has a lot of granular quotas, and sometimes they can be very annoying to get increased. Most of our quotas were automatically approved, but some were not. Particularly, they are picky about granting static IP addresses. Start running stuff on GCP at least a month before the event, so they are more likely to approve it. A tip – include the keywords "according to load testing" when asking for quota increases and it seems more likely to get approved automatically ;)

CTFd Tuning / Gotcha's

Because CTFd is a Flask application, it comes with its fair share of overhead. It's not the most performant thing in the world, especially if you leave it on default settings. Here's a list of things we did to get CTFd to run more smoothly, which I'd recommend for anyone using it to run a larger CTF.

  • Separate the MySQL Database - We spun up a Cloud SQL instance with Private Networking, and pointed CTFd to use that by changing the docker-compose environment variables. E.g. DATABASE_URL=mysql+pymysql://root:zzzzzzzzzzzz@10.x.x.x/ctfd
  • Don't use CTFd's default static file serving - CTFd supports S3 for hosting file uploads/assets. Use this! It is very beneficial to keep load off of CTFd for serving file downloads. We thought we would have to make an entirely separate plugin for file assets, but luckily Google cloud platform has an S3 compatability mode for its Cloud Storage. We set that up, and only had to make a 1-line change to CTFd to get it to work.
  • Increase SQLAlchemy connection pool - SQLALCHEMY_MAX_OVERFLOW=100
  • Increase gUnicorn workers - WORKERS=65
  • Use a compute-heavy instance - We used an e2-medium the 2 weeks before the competition while registration was open, and resized our CTFd to a e2-highcpu-32 instance (gunicorn is CPU-heavy) for the duration of the competition, and had no issues after we did this.
  • We did not use gcloud memorystore, just stuck with the redis container in the docker-compose. No idea if it would be beneficial or not.

Continuous Integration

For keeping our challenge descriptions/assets in version control, and pushing them to CTFd, we used github actions + ctfcli. Here are our actions that may come in handy.

Sync/install challenge.yml to CTFd whenever someone pushes to its folder

name: ONPUSH - ctfd sync
# Controls when the action will run. 
on:
  # Triggers the workflow on pushes to the main branch that include changes to any files in the /challenges dir
  push:
    branches: [ main ]
    paths: [ 'challenges/**' ]
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      - uses: actions/checkout@v2 # Checks-out repository under $GITHUB_WORKSPACE, so job can access it
        with:
          fetch-depth: 0 # fetch all history so we can check what files changed
      - uses: actions/setup-python@v2 # install python
      - name: Install ctfcli
        run: |
          python -m pip install --upgrade pip
          pip install ctfcli
      - name: init ctfcli
        run: |
          printf "$CTFD_URL\n$CTFD_TOKEN\ny" | ctf init
        env:
          CTFD_URL: ${{ secrets.CTFD_URL }}
          CTFD_TOKEN: ${{ secrets.CTFD_TOKEN }}
      - name: sync updated chals to ctfd
        run: |
          shopt -s globstar
          cd $GITHUB_WORKSPACE/challenges
          echo "changed files: $(git diff --name-only ${{ github.event.before }} ${{ github.sha }})"
          for i in ./**/challenge.yml; do 
            CHAL_DIR=$(dirname $i)
            git diff --name-only ${{ github.event.before }} ${{ github.sha }} | grep -q "$CHAL_DIR"
            if [ $? == 0 ]; then
              echo "SYNCING CHAL TO CTFd: $CHAL_DIR"
              { ctf challenge sync $CHAL_DIR; ctf challenge install $CHAL_DIR; } &
            fi
          done
          wait
        shell: bash --noprofile --norc {0}

Manual action to sync/install all challenge.yml files

name: MANUAL - ctfd sync
# Controls when the action will run. 
on:
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      - uses: actions/checkout@v2 # Checks-out repository under $GITHUB_WORKSPACE, so job can access it
      - uses: actions/setup-python@v2 # install python
      - name: Install ctfcli
        run: |
          python -m pip install --upgrade pip
          pip install ctfcli
      - name: init ctfcli
        run: |
          printf "$CTFD_URL\n$CTFD_TOKEN\ny" | ctf init
        env:
          CTFD_URL: ${{ secrets.CTFD_URL }}
          CTFD_TOKEN: ${{ secrets.CTFD_TOKEN }}
      - name: sync chals to ctfd
        run: |
          shopt -s globstar
          cd $GITHUB_WORKSPACE/challenges
          for i in ./**/challenge.yml; do 
            CHAL_DIR=$(dirname $i)
            echo "SYNCING CHAL TO CTFd: $CHAL_DIR"
            ctf challenge sync $CHAL_DIR
            if [ $? == 1 ]; then
              echo "::warning::Error syncing chal $CHAL_DIR"
            fi
            ctf challenge install $CHAL_DIR
          done
        shell: bash --noprofile --norc {0}

kCTF

kCTF is very cool. It allowed us to deploy our challenges to a k8s cluster, easily horizontally load balance them, and it definitely came in handy during the event.

One challenge - ponyDB, initially had a bug that allowed people to crash the container. However, because we had written healthcheck scripts for every challenge possible, kCTF would automatically restart the container whenever it crashed for us because the healthcheck failed, allowing people to continue solving the challenge without too much downtime while we went and fixed the issue.

Two other challenges, yana and pow_erful, ran into issues with people DOSing them by sending way too many requests. This ended up being super easy to resolve with kCTF, as we just had to turn on the horizontal scaling for them.

I won't say kCTF is flawless, it definitely had a decent amount of learning overhead, but I do think it was worthwhile. We gave a list of feedback to the kCTF developers, and I hope they take it into account and improve it. The biggest struggles had to do with things being unintuitive or not documented well.

Potential Overhead:

  • you may have to learn nsjail
  • you may have to learn k8s
  • you may have to refactor your challenge containers to be compatible with kCTF (e.g. kCTF currently deploys containers with a readonly filesystem by default)
  • you may have to spend time debugging some unusual behavior of k8s, hopefully this reduces in the future.

I have to say that the developers were extremely responsive and helpful with resolving issues and getting things to work. We asked for an IP allowlist feature, and it was ready within a week or two. We reported a bug with being unable to set capabilities on the containers, and it was resolved in a day. I am not sure I've ever interacted with more helpful developers. Even though I felt like I was being bothersome at times, they were always happy to assist :)

ModMail

We utilized a discord bot: https://github.com/kyb3r/modmail/, as a method for participants to contact admins on the discord server.

This was extremely useful and I would highly recommend it. It allowed us to all read through messages and reported issues. This way, even if a challenge creator was offline, another staff member could respond and provide support.

Sample ModMail thread

Our CTFd Plugins

We have a few useful inhouse CTFd plugins that we developed for this/previous events. Feel free to use or fork them if you find them helpful!

  • ctfd-discord-auth - we used this plugin to verify that people were who they claimed to be on discord. it adds an endpoint /discordauth to CTFd which initiates the discord OAuth flow and sends the results to a channel
  • ctfd-dynamic-challenges-mod - we did not like the default dynamic scoring function of CTFd (thought it kept challenges at high point values for too long), so we wrote a plugin that modifies the function.
  • ctfd-discord-webhook-plugin - post in a discord channel, #solves, whenever the first 3 solves for a challenge happen. participants seem to like this a lot :)

Things to do differently

  • utilize CTFd Notifications for announcing challenge updates
    • we assumed that all competitors would be on our discord, but this was not the case, and some people missed out on challenge fixes/updates during the event
    • A CTFd plugin to do this would be very useful. Either to send CTFd notifications -> discord channel, or the other way around. Maybe it could be added to our CTFd solve discord plugin.
  • test, test, test
    • lack of load testing made us launch the event with an undersized CTFd instance
    • maybe we should have set up CTFd for use w/ appengine? we never looked into this
  • more mid-tier challenges - we had a decent amount of beginner-friendly challenges and hard challenges, but some categories were lacking things in between. Namely, pwn and web.
  • challenge balancing - some feedback we got: web challenges too hard (not enough beginner ones), misc beginner-friendly challenges mildly guessy/uninteresting, kernel challenges too hard, lack of hard crypto.

Closing thoughts

UIUCTF'21 was a blast to run, and I learned a lot setting up infra (finally got around to learning some k8s). If you plan to run a CTF event and have questions, feel free to contact me on Discord (arxenix#1337) or  Twitter (@ankursundara).