The filesystem - a quick & dirty way to store temp data in your API

I have an API service for a Google Sheets add-on that I built, and I wanted to temporarily store some request metadata that I started collecting in the client.

How the add-on works. The API generates the forecast data.

The service was built with Django Ninja, and it’s not connected to a database. I thought about setting one up for this feature. But I couldn’t be bothered dealing with the configuration and the migrations. I wasn’t sure how long I’d be collecting this data for.

I then considered dumping the metadata in the logs. The problem with that approach is that it’s a hassle to aggregate all the relevant messages. I was planning to move the information to another place to do some analysis.

I briefly reflected on a global variable as well, but I was worried about the memory management.

And then, at last, I got to the most robust solution in the galaxy - the filesystem. I would store each piece of request metadata in a separate file, so as to avoid issues with concurrent writes. And, I would expose an endpoint in the API that combines all those files to show me all of the data.

Voilà the code:

def request():
  # at the end, before I return the response
  entry = {
          "timestamp": datetime.now().isoformat(),
          "request_id": request_id,
          "metadata": metadata,
  }
  with open(f"{REQUESTS_DIR}/{request_id}.json", "w") as f:
      json.dump(entry, f)

  return response

# decorator to authenticate with API key header
def get_logs():
  requests = []
    for filename in glob.glob(f"{REQUESTS_DIR}/*.json"):
        with open(filename, "r") as f:
            try:
                data = json.load(f)
                # Add filename as id for reference
                data["id"] = os.path.basename(filename).replace(".json", "")
                requests.append(data)
            except json.JSONDecodeError:
                pass

    # Sort by timestamp
    requests.sort(key=lambda x: x["timestamp"])
    return requests

With that deployed, I can now fetch all of the data in my API’s OpenAPI docs. Django Ninja generates this page automatically for me. Lovely jubbly!

The swagger endpoints

The response for the endpoint I just created