Read at BYK's

Adaptation: new tools in town!

Tue, 14 Apr 2026 00:00:00 GMT

It was about a year ago. I was getting bombarded with LLMs in my editor. It started with VSCode offering code snippets as random auto complete suggestions. I hated them. They were mostly dumb, confused me and now I actually had to think deeper before accepting a suggestion because they actually could be subtly broken code. Then, very occasionally but at an increasing frequency, the suggestions started to understand my intent and predict my actual next step. That was oddly satisfying¹. Even worked on my blog posts since I write these in markdown in my editor. Then came the inline ask mode where I asked it to do a very boring and repetitive task. It did it quite well and quite inefficiently. I could have used ast-grep for that but, that also costs a lot of brain tokens y’know? Cramer gave us all Cursor access and encouraged us to try. Syntax podcast was full of episodes of comparing Cursor to Copilot to Windsurf, my new colleague Miguel seemed to be cranking out big patches with AI (they did have some silly stuff in it but overall, they were good) and I still couldn’t get myself to use it. Don’t get me wrong, I did try many times. The issue was, it felt like trying to work with a thick but self-assured person getting all the core concepts wrong. I kept hearing people talking about generating a plan or a todo list for the agents to follow and I was thinking “The reason I became a software engineer is because I’m lazy. If I’m gonna write down how to do it exactly, I might as well do it myself, and better²”.

Then, during one school break trip came that fateful moment. I was too tired to actually code and the code that needed to be written was quite tedious. I needed to parse stdout of random processes, fuzzy match them with logs I might have been receiving and do more stuff accordingly.³ But again, I was tired and even if I wasn’t, stdout plumbing with random processes is as fun as a root canal⁴. At that point I said “screw it, I won’t be able to get this done and I already planned it out in my head. I can dump it to Cursor and see where it goes with it.” Turns out Cursor added the “plan mode” around the same time. See, you don’t have to write the plan, the AI can write the plan for you and execute it too! Believe it or not, it one-shotted the entire patch and probably better than what my first pass would have been. I was excited, dumbstruck, curious, furious, and everything in between. I threw more problems to it and just kept getting good results. I also got a lot of literal heat though! Don’t know why⁵ but despite the model running in the cloud, just the agent loop itself was making my laptop sweat⁶.

This was only the beginning though. A few months later, on another school break trip⁷, a wild ~~Pokémon~~ build in cloud feature appeared! This was a game changer and became my other breaking point. Building in cloud means my computer becomes free after the planning session. Which means the only bottleneck is now the planning phase. And that, my friends, is the actual force multiplier. The real productivity gain that people have been talking about. See, even if the AI does a slightly worse job than myself, I can now practically work on many things in parallel through delegation. I started to realize there was a hidden PM in me! 😱

I rapidly started throwing GitHub issues and ideas at it, generating plans and then offloading to cloud build agents. For some silly reason, the planning phase is restricted to the local computer but it’s not a deal breaker. To review the code generated by AI, I decided to use draft PRs on GitHub as it is a much nicer experience than my editor. It also only involves switching tabs rather than Git branches. The only issue is, now I look like a mad man arguing with himself on the PRs, publicly 😅⁸. I created this “skill”, telling my agent to fix CI builds and address all review comments until there are none. This was bliss. I even bought a foldable phone⁹ so I can review these PRs easily on the go and keep the agents busy! I was hooked. I was shipping like crazy and I was checking my phone frequently to see ~~my likes on Instagram~~ my agents’ progress. And then, just like it came out of nowhere, “build on cloud” went to nowhere. “The Lord giveth, The Lord taketh away” I guess. Back were the dark days of being stuck with a single task and a loud and hot laptop.

That’s when I decided to build my ultimate setup. But that, friends, is the topic of my next post.

And creepy. ↩
With blackjack and hookers. ↩
This was the first prototype of spotlight run ↩
Ironically, I found it very exciting for the very first time! ↩
Hi Norah Jones. ↩
Well, more like boil, probably the biggest contributor to my thermal paste sputtering around the die. ↩
Yes, we have so many school breaks. No, I don’t know exactly why. ↩
Because the bot uses my own credentials for the PR through gh ↩
Pixel 10 Pro Fold. Yup, that’s a mouthful and is quite expensive but was okay with a trade-in with my old phone. ↩

Releasing Packages with a Valet Key: npm, PyPI, and beyond

Tue, 25 Nov 2025 00:00:00 GMT

Disclaimer: This post should have been written about 5 years ago but I never got around to it; with the most recent Shai-Hulud attack, I thought it would be a good time to finally check this off the list and hopefully help others avoid supply-chain attacks.

About 5 years ago, I sat in on a meeting at Sentry in the midst of their SOC 2 compliance efforts. There was Armin, telling us that we needed a secret storage service for our package repository tokens. The tokens we used to deploy Sentry SDKs to package repositories such as npm, PyPI etc. This was to ensure there were no unauthorized releases of our SDKs which were embedded into all Sentry customers’ products. There were a limited set of people who had access to these tokens back in the time. Now, they became the bottleneck for more and more frequent releases. There was also the auditability issue at hand: releases were performed from individuals’ workstations and there was no easy way to trace a release back to where it originated from or whether it was authorized or not.

For some reason I intuitively was against such a secret storage service and felt like the answer was somewhere in GitHub, GitHub Actions, and their secret storage service we already used. We already had the repo permissions, personnel structure, and all the visibility for auditing there. Heck, even the approval mechanics were there with pull requests. So I said “give me a week and I’ll get you a proof of concept” which Armin did and I delivered - though I think it took a bit more than a week 😅

Secrets in Plain Sight

Before we dive into the solution, let me paint a picture of the problem. Publishing packages to registries like npm, PyPI, or crates.io requires access tokens. These tokens are essentially the keys to the kingdom - whoever has them can publish anything under your organization’s name. At the time, these tokens were either distributed to select individuals, or lived in GitHub repository secrets, accessible to anyone with write access to the repository.¹

Now, here’s the scary part: at Sentry, we had 90-100+ engineers with commit rights to our SDK repositories. Any one of them could:

Create a new workflow or modify an existing one
Access these secrets within that workflow
Exfiltrate them to any web service they controlled
Do all of the above without triggering any alarms

And the truly terrifying bit? Even if someone did steal these tokens, there would be no indication whatsoever. No alerts, no logs, nothing. They could sit on these credentials and use them months later, long after they’ve left the company. We’ve seen this exact scenario play out recently with supply-chain attacks like the Shai-Hulud npm takeover where attackers compromised maintainer accounts to publish malicious versions of popular packages.

The Valet Key

Some fancy cars come with a “valet key” - a special key you give to parking attendants or car wash folks. Unlike your regular key, this one has limited capabilities: maybe it can only go up to 20mph, can’t open the trunk, or won’t let you disable the alarm. It’s the same car, but with reduced privileges for reduced risk of theft.

This concept maps beautifully to our problem. Instead of giving everyone the full keys (the publishing tokens), why not give them a way to request the car to be moved (a release be made)? The actual keys stay with a very small, trusted (and monitored) group who are the builders and maintainers of the infrastructure. Even the approvers don’t actually have access to the keys!

Here’s what we wanted:

Secrets in a secure, limited-access location - only 3-4 release engineers should have access
Clear approval process - every release needs explicit sign-off from authorized personnel
Low friction for developers - anyone should be able to request a release easily
Full audit trail - everything logged being compliance-friendly
No new infrastructure - we didn’t want to build or maintain a separate secrets service

As a side note, trusted publishing through OIDC and OAuth with limited and very short-lived tokens is the actual digital equivalent of valet keys. npm is slowly rolling this out², but at the time we built this system, it wasn’t an option. And even today, it’s not available at the organization/scope level which is what we’d need. Also, we publish way more places than npm so we need a more generic solution.

Another approach worth mentioning is Google’s Wombat Dressing Room - an npm registry proxy that funnels all publishes through a single bot account with 2FA enabled. It’s a clever solution if you’re npm-only and want something off-the-shelf. That said it still requires running a separate service.³

Enter getsentry/publish

The solution we landed on is beautifully simple in hindsight: a separate repository dedicated entirely to publishing. Here’s the trick:

Write access is extremely limited - only 3-4 release engineers can actually modify the repo
Release managers get “triage” access - GitHub’s triage role lets you manage issues and labels, but not code - perfect for approving releases
Everyone else can create issues - that’s all you need to request a release
Approval happens via labels - a release manager adds the “accepted” label to trigger the actual publish

The beauty of this setup is that the publishing tokens live only in this repo’s secrets. The repo itself is mostly static - we rarely need to modify the actual code - so the attack surface is minimal.

The Implementation (with Craft)

Under the hood, we use Craft, our CLI tool for managing releases. Craft was designed with a crucial architectural decision that predates the publish repo: it separates releases into two distinct phases - prepare and publish.

The prepare phase is where all the “dangerous” work happens: npm install, build scripts, test runs, changelog generation. This phase runs in the SDK repository without any access to publishing tokens. The resulting artifacts are uploaded to GitHub as, well, build artifacts.

The publish phase simply downloads these pre-built artifacts and pushes them to the registries. No npm install, no build scripts, no arbitrary code execution - just download and upload. This dramatically reduces the attack surface during the privileged publishing step. Even if an attacker managed to inject malicious code into a dependency, it would only execute during the prepare phase which has no access to publishing credentials.

This two-phase architecture is what makes supply-chain attacks like Shai-Hulud much harder to pull off against Sentry’s SDKs. The malicious code would need to somehow persist through the artifact upload/download cycle and execute during a phase that deliberately runs no code.

The magic happens with our GitHub Actions setup:

Developer triggers release workflow in their SDK repo (e.g., sentry-javascript)
action-prepare-release runs craft prepare: creates the release branch, updates changelogs, builds artifacts, uploads them to GitHub
An issue is automatically created in getsentry/publish with all the details: what changed, what’s being released, which targets
Release manager reviews and approves by adding the “accepted” label
Publishing workflow triggers craft publish: downloads artifacts from GitHub and pushes to npm, PyPI, crates.io, etc. - no build step, just upload

Fighting Overprotective Parents

GitHub, bless their security-conscious hearts, put up quite a few guardrails that we had to work around. Here’s where things got… creative:

The Token Trigger Problem: For the automation, we had to use the Sentry Release Bot, a GitHub App that generates short-lived tokens. This is crucial because GITHUB_TOKEN (default token GitHub Actions creates) has a security restriction: actions triggered by it don’t trigger other actions⁴. We needed workflows in getsentry/publish to trigger based on issues created from SDK repos, so we had to work around this.

The Admin Bot Account: We needed a bot that could commit directly to protected branches. GitHub’s branch protection rules ~~are~~ were all-or-nothing - you can’t say “this bot can commit, but only to update CHANGELOG.md”. So our bot ended up with admin access on all repos. Not ideal, but necessary⁵.

Composite Actions and Working Directories: If you’ve ever tried to use GitHub’s composite actions with custom working directories, you know the pain. There’s no clean way to say “run this composite action from this subdirectory”. We ended up with various hacks involving explicit cd commands and careful path management.

Some More Creative Workarounds: We maintain a small collection of ugly-but-necessary workarounds in our action definitions. They’re not pretty, but they work. Sometimes pragmatism beats elegance⁶.

Happily Ever After

After all this work, what did we actually achieve?

Compliance-friendly ✓ - every release is logged, approved, and traceable
Centralized secrets - tokens live in one place, accessible to very few
Developer convenience - anyone can request a release with a few clicks
Enterprise security - no individual has publishing credentials on their machine
Full transparency - the entire publish repo is open, notifications enabled for stakeholders

We’ve made more than 6,000 releases through this system and happily counting upwards. Every single one is traceable: who requested it, who approved it, what changed, when it shipped.

Why This Matters Today

Recent supply-chain attacks like Shai-Hulud show exactly why this architecture matters. When attackers compromise a maintainer’s npm account, they can publish malicious versions of packages that millions of developers will automatically install. With our system:

No individual at Sentry has npm/PyPI/crates.io credentials on their machine
Every release requires explicit approval from a release manager
The approval happens in a public repo with full audit trail
Any suspicious activity would be immediately visible

Is it perfect? No. Could a determined attacker with inside access still cause damage? Probably. But we’ve dramatically reduced the attack surface and made any compromise immediately visible and auditable.

Closing Thoughts

Looking back, this is one of my proudest achievements at Sentry. It’s not flashy - no one’s going to write a blog post titled “Revolutionary New Way to Click a Label” - but it’s the kind of infrastructure that quietly makes everything more secure and more convenient at the same time.⁷

If you’re dealing with similar challenges, I encourage you to check out getsentry/publish and the Craft. The concepts are transferable even if you don’t use our exact implementation.

And hey, it only took me 5 years to write about it. Better late than never, right? 😅

Thanks

I’d like to thank the following people:

Armin and Daniel for their trust and support in building this system.
Kamil for Craft as I knew it.
Jeffery for reviewing this post thoroughly and being my partner in crime for many things security at Sentry.
Michael for giving me the push I needed to write this post, coming up with the awesome post image idea, and for his support and guidance on the post itself.

This was before GitHub introduced “environment secrets” which allow more granular access control. Even with those, the problem isn’t fully solved for our use case. ↩
npm has OIDC support for individual packages, but not yet at the organization or scope level. See npm’s trusted publishers documentation. ↩
If only someone could make this run directly in GitHub Actions… ↩
This is actually a smart security feature - imagine a workflow that creates a commit that triggers itself. Infinite loop, infinite bills, infinite sadness. ↩
This is now fixed with special by-pass rules via rule sets recently and we also no longer have admin access for the bots, phew. ↩
If you peek at the repo, you’ll see what I mean. I’m not proud of all of it, but I’m proud it works. ↩
Especially while “security means more friction” is still a thing. ↩

Marking it Up (and Down)

Wed, 02 Jul 2025 00:00:00 GMT

First, there was plain text

When I first learned about Markdown, I was a bit skeptical. Why use weird punctuation when you can use HTML instead? But as I started using it more, especially on forums etc, I realized the power of it. Unlike HTML, it was way more accessible and easier to type. Even more importantly, it was still readable and expressed meaning without obstructing the text before rendering. And slowly but surely, all major platforms, including WhatsApp adopted it.

The Age of AI

And then ChatGPT happened. Due to the properties I listed above, Markdown was the perfect format for LLMs too. Once the agents hit the scene, they started generating Markdown formatting and they were also more than happy to ingest Markdown formatted text for their context. There was only one problem though: the web revolved around HTML, and some of that being dynamically generated. Even if you could teach or extract HTML, the dynamic JS part of it is still a challenge and usually requires a full browser environment. Sure, there’s Playwright MCP but it’s slow and resource-intensive. These challenges gave birth to services like Firecrawl which I think is awesome, especially when you cannot control the source of the information.

Recently, with a lot of ~~push~~ help from David, I started learning about agentic flows and how to use LLMs more than generating 0-shot responses¹. I wanted to write a bit about these too but Colin Eberhardt already did a great job with Re-implementing LangChain in 100 lines of code. This is the article that made it click for me. Once I read it, I even felt a bit silly for expecting something more complex. It’s deceptively simple: you use the LLM in a loop, parse the responses to trigger “tools”, and feed the results back (part of the loop) until you reach a final result (or the limit of your wallet). Anyway, great article, definitely go read it. Let’s go back to talking about Markdown and its cousins as this is a blog post about that, not LLMs.

Walking back from the X-factor

For my new internal project, I needed to use our docs. Although they were already authored in MDX, it was not pure Markdown. We can strip the MDX parts but Sentry Docs are architected to share certain parts of the content between different pages. This means we actually have to render the MDX to get the full content. As a person who spent some time around parsing out dependencies, building a dependency graph and working over it I had no interest in going down that path unless I really had to. So I decided to look for an existing “HTML to Markdown” solution. This led me to the awesome rehype-remark package which is a part of the unified project. I was already quite familiar with unified and remark which we also used in our docs rendering pipeline. So I simply jumped on this. My initial solution was simply to fetch the page of interest, convert it into markdown, find the relevant header and extract the contents until the next header. The code was also simple:

import type { Root, Heading } from "mdast";
import rehypeParse from "rehype-parse";
import rehypeRemark from "rehype-remark";
import remarkStringify from "remark-stringify";
import { unified } from "unified";

function extractMDSection({ section }: { section?: RegExp }) {
  return (tree: Root) => {
    const headingIdx = tree.children.findIndex((node) => {
      return (
        node.type === "heading" &&
        node.children[0] &&
        node.children[0].type === "link" &&
        section?.test(node.children[0].url)
      );
    });
    const heading = tree.children[headingIdx] as Heading;
    const nextHeadingIdx = tree.children.findIndex(
      (node, idx) =>
        idx > headingIdx &&
        node.type === "heading" &&
        node.depth === heading.depth
    );
    tree.children = tree.children.slice(
      headingIdx,
      nextHeadingIdx === -1 ? undefined : nextHeadingIdx
    );

    return tree;
  };
}

export const getWebpageAsMarkdown = async (url: string, section?: RegExp) => {
  const response = await fetch(url);
  const text = await response.text();
  return String(
    await unified()
      .use(rehypeParse)
      .use(rehypeRemark)
      .use(extractMDSection, { section })
      .use(remarkStringify)
      .process(text)
  );
};

Putting it everywhere, and fast

Then /llms.txt happened. All players in the field who wanted to be more useful in the “age of AI”² started publishing their content accessible to LLMs, in plain text or more commonly, Markdown format. Then a convention emerged: if you add .md at the end of the URL you may get lucky and get the Markdown version of that page. I’m not sure when this kind of convention started but it reminded me of the .patch trick that GitHub offers for their PRs. We wanted this for Sentry Docs! The first approach was to do this on the fly on a specific route. Not only did this prove tricky to implement in NextJS, which our docs are built in, it also had an efficiency problem. Since we cannot go directly from MDX to Markdown, we had to render the HTML from MDX first and then convert it to Markdown, essentially doubling the work. A nice trick Cody came up with was building the Markdown versions from the static HTML files that NextJS generates during pre-rendering, putting them under the public directory, and adding a rewrite rule to NextJS to serve them when the .md extension is requested. This worked beautifully but created another issue: we had to generate the Markdown files for all 8754 pages in Sentry Docs and doing this takes a lot of time, up to 6-7 minutes.

For a one-off job, spending several minutes is more than OK. But for a CI job that runs on every single commit, it is completely unacceptable. So I reached for 2 very old tricks used in every build pipeline: caching and parallelization. The script for Markdown generation was refactored to spawn multiple NodeJS Worker Threads for parallelization. Then I also added a very naive cache which got the MD5 hash of the source HTML file and created a cache file with that name containing the converted Markdown. This allowed me to just do a cp operation if the source file did not change. These worked great on my local environment. The parallelization cut down the processing time by about 6x on my 16-core machine and the caching reduced that time by another 10x. However, when I pushed this to Vercel, our hosting platform for Sentry Docs, it was still very slow. Looking carefully at the build logs I noticed 2 issues:

Vercel build machines usually had 2 or 4 cores, significantly lower than 16.
The cache was not being used at all!

Solving the first one was not possible. During my tuning (local and on CI), I discovered we needed about half of the available cores due to the CPU & I/O intensive nature of the task:

// On a 16-core machine, 8 workers were optimal (and slightly faster than 16)
const numWorkers = Math.max(Math.floor(cpus().length / 2), 2);

Then I started investigating the cache issue and after several hours of digging, I finally realized what was going on. NextJS creates a new “signing secret” for every build which also affects the file names of the JS files it generates as the names are created from file contents. This then causes the HTML files’ MD5 hashes to change although their actual contents were the same. To overcome this in a cheap manner I had to strip the <script> tags (along with their contents) from the HTML files before hash calculation and processing:

const text = (await readFile(source, { encoding: "utf8" }))
  // Remove all script tags, as they are not needed in markdown
  // and they are not stable across builds, causing cache misses
  .replace(/<script[^>]*>[\s\S]*?<\/script>/gi, "");

Surprisingly, this also reduced the processing time by about 2x as the HTML files were significantly smaller without the <script> tags.

We also started uploading these to a special Cloudflare R2 bucket for RAG processing that David started using for a much better search experience.

Can’t Stop the Feeling!

Once I get into optimization mode, I cannot stop until I hit a very hard wall or actually get every ounce of optimization implemented. So I started looking at other places where I can use the same old tricks of caching and parallelization. Turns out our MDX pipeline was not only uncached, it was also mostly using the sync version of the file system APIs in NodeJS. So I made every single file system operation async, used Promise.all to parallelize them and got a huge speed increase. That is until this was shipped to Vercel. This time, it was the “dynamic pages” which used Vercel Functions that caused crashes. These were crashing with an EMFILE file error, indicating that the file descriptor limit was reached. In hindsight, this is very obvious but back at the time I had to dig around as these were not happening locally. It first looked like a silly limitation in AWS Lambda, which is what Vercel Functions are based on, but it turned out to be a legitimate issue. With the top level Promise.all, I was creating all 8600+ promises at once, which themselves triggered more open file handles. Again, obviously this is insane so I reached out to another old friend: p-limit³. With a limit of 200 concurrent promises, we sailed on smoothly.

Then I moved on to the caching bit which turned out to be a bit more tricky. We are using this other awesome package called mdx-bundler. It takes in an MDX file, discovers all its dependencies, and bundles them together into a single JS file. Easy peasy, right? Just cache the output based on the input MD5 and we’re good! Well, almost. The catch is we ask the bundler to emit the assets (mostly images) into a separate folder. This means we also need to cache these assets too. The solution became a file and a directory, using the cache key as their names⁴ where we copy everything in place when we find them. This chopped another 3-4 minutes off the build time when only a few files changed, which is the common case.

Tying it all together

It took about 2 weeks and 12 PRs to tie all the loose ends but now not only do we have .md versions of every single page in Sentry Docs, we also have better RAG-based search (still in-progress), and faster builds (from ~16 minutes down to ~11 minutes). I love these kinds of intense periods where I can focus on a few high-impact things and just punch them out. Hopefully, there will be some more in the coming weeks and months. Here’s a list of the important PRs we made to get here:

These are nothing short of amazing but without the agent loop and with 0-shot approaches, they are not very useful for the tasks I have at hand. ↩
Yup, let’s cringe together. ↩
Btw, I still refuse to believe Sindre Sorhus is a real person. That is alien-level productivity and reach 🙇🏻‍♂️ ↩
Well, they cannot be the same name as you cannot have a file and a directory with the same name in the same place. So I just added a suffix to the file name. ↩

The magic word: TaxYearEnd

Tue, 01 Jul 2025 00:00:00 GMT

This blog wasn’t supposed to be all horror stories but looks like life sends me quite a few recently. I’ve spent more than a week at the end of April trying to fix an incorrect tax code with my employer on record (as Sentry does not have a UK entity yet). I was able to get it fixed… but not through the company nor through their support team.

The Payslip

It was an unusually sunny April day in south east England when I got my first payslip of the 2026 tax year. I was looking forward to this as it would be marking the end of a tax code that deducted extra tax from my salary and put me back onto the tax bracket and code where I was supposed to be. The reason for this was me switching jobs in the middle of the tax year and HMRC¹ thinking I got a massive pay rise I wish I had but didn’t. Thankfully, I got all that extra tax claimed back with the year-end self-assessment tax return but I digress. It was a sunny April day where I was shocked to find out that not only my tax code was incorrect, it was actually worse than before! I was expecting to get more money but my paycheck was actually lower than the month before 😭. I was confused at first but quickly recovered and reached out to my EOR company’s support team. It was not May 1st yet so I may have had a chance to get it corrected before the actual bank transfer. I could have wished for world peace instead.

The Denial

The initial reactions from the support team can be summarized as:

Oh you poor entitled prick, so sad your tax code makes you pay more tax but just deal with it.
Oh poor boy, you don’t know taxes and you are definitely not smart enough to understand this. Just accept what it is as we are doing what we are told by HMRC.
Oh, you are still here? Go figure this out with HMRC, they told us to use this tax code.

I am a person who doesn’t easily give up, especially when I think I’m right.² I have already logged into my HMRC account when I got a tax code change notification a few weeks ago and I was sure my tax code there was not the one I was seeing on my payslip. Nonetheless, I logged in again to see if anything has changed. Surprise surprise: nothing has changed. Took a screenshot, also found the tax code change letter’s PDF after a long series of clicking around and shared these in the support thread. Response? Your proof does not prove anything, you seem to be lying and even if it did and you weren’t lying, we just cannot change your tax code manually. Go to HMRC.

A game of letters

At this point, I was fuming. Also worth mentioning that after the initial few messages, the issue was escalated to the payroll team which seems to be only able to respond to these messages once a day. I started digging into HMRC’s self-service portal, reading up on tax codes, how they change, how they are communicated to your employer, and what your/my employer is responsible for. I found all the nice history and modern tech behind all this and essentially debugged my employer’s payroll system for them. For free.

Every time your tax code changes, HMRC sends a letter to you and to your employer about this change. This letter is called a P6 or a P9. The P6 is sent when you are a new employee or when your tax code changes. The P9 is sent when you are an existing employee and your tax code changes. The letter contains your new tax code, the effective date of the change, and a few other details³. So, for my case, I got two letters: one on February 2025 and one on April 2025. The first one, from February, was for the coming tax year, 2026, instructing my employer to set my tax code to what I was seeing on my HMRC account. The second one, from April, was for the tax year that just ended (2025) and setting my tax code retroactively to the one I was seeing on my payslip. Can you see what’s going on here?⁴

The “fineprint”

So the “fineprint” here is, to which year the letter applies to. Turns out the letter explicitly states the tax year it applies to but HMRC usually don’t send retroactive letters. So payroll automation systems are coded to only look at the “effective date” and “tax code” fields and ignore anything else⁵. So from the company’s perspective, they were correct. The HMRC indeed told them to set my tax code to the one on my payslip, effective April 2025 BUT FOR THE BLOODY⁶ 2025 TAX YEAR. The letter from February was for the 2026 tax year but it got overridden by the letter that came in April.

After discovering this entire kerfuffle, I documented and explained all of it to the support team. Demanded my tax code to be fixed ASAP, their payroll system checked and fixed, and my money returned. And of course they complied and we all lived happily ever after under the rainbows from the unicorns living in the beautiful British countryside.⁷

Being practical

What I got in response was still the same: the system was automated, their system showing that the tax code must be changed to the incorrect one effective April 2025, and if I have a problem with that I should contact HMRC to change my tax code. I was furious at this point, creating a local sun which caused the heatwave we had here in the UK in May. Before digging deeper though, I decided to stop the bleeding. I called HMRC, explained the situation briefly, and asked whether I can get another P6 issued. And unlike the other side of the story, what I got was a few cheerful clicks and “done, we issued a new P6”.⁸

The depth of the rabbit hole

Armed with the calmness of my new&improved™ tax code and the knowledge of the electrons and photons carrying that new P6 letter’s contents being on their way to my employer’s <insert expletive here> payroll system, I started digging for this “electronic PAYE system”. After a short period of Googling, I found out about the “Data Provisioning Service” (DPS) and HMRC API. The DPS is the “electronic PAYE system” that HMRC uses to send tax code changes to employers. That mythical thing I’ve been looking for! As usual with things related to big organizations, it was architected around SOAP and XML with a very long specification document. That said, as expected from an accountable and transparent organization, there is plenty of public documentation and tooling. If you have a special interest in overly long and complex specifications, I highly recommend reading all the documents and checking out the tooling.

Although I didn’t have this special interest, I did have the interest of proving someone wrong.⁹ Upon reading these, I discovered the awesome Outgoing XML Generator or OXG 4.5 as it is known among friends. This is an ugly tool written in Java that can generate the XML payloads that HMRC sends to employers. After some more digging I was able to generate a sample P6 message to test. And wouldn’t you know, it had this TaxYearEnd field in it! Upong seeing this, I asked for the “raw” data of this message from the payroll team. It took them long enough to provide it but they finally did. And guess what? The TaxYearEnd field was indeed set to 2025 vindicating my theory that the payroll system was the faulty part here. I nobly relayed this information back to the support agent to be passed to the payroll team, with full confidence that they now will apologize, fix the issue, and investigate their systems.

Instead, I just validated all those people who called me naive for all these years. The support agent came back with the stale old “we are sorry but we cannot change your tax code” response without admitting to any wrongdoing on their part. Lucky for me, I already got that new P6 issued so instead of creating another heatwave, I just sat down, looked at my screen in disbelief. Disbelief that not only such organizations can exist but they make a lot of money along the way.

Conclusion

The month turned to June, I got my new payslip with the correct tax code along with all the extra tax I paid returned to me. Staring into the distance, I then sat down to write this blog post which turned out to be a guide in implementing a payroll system that interfaces with HMPC’s DPS. In hindsight, I think I should have just called the HMRC for a new P6 as that was the solution in the end. That said I don’t regret all the research I did and things I’ve learned. I only regret the anxiety and stress during the period.

Anyway, if you ever implement that payroll system, please make sure to honor the TaxYearEnd field in the P6 messages.

Thanks,

A friend.

The UK tax authority: Her/His Majesty’s Revenue & Customs — for some reason I thought this was Her Majesty’s Revenue Chest but turns out I was wrong. Still way cooler than Internal Revenue Service. ↩
People who know me IRL are nodding in agony right now. Hey, I’m sorry. We don’t get to choose our personality traits. ↩
Turns out the devil was hidden in a specific detail here. Oh I’m not spoiling it, keep reading please ☝🏻 ↩
Don’t just read on, just think a bit. I can wait. Since this is a text that’s already written it doesn’t really cost any extra. Seriously, I can wait till the heat death of the universe. Okay okay, let’s get back? ↩
Well, they don’t ignore to whom the letter applies to obviously 😅 ↩
Oh yes, I lived in the UK enough to start using bloody instead of the f-word. Just in writing for now. ↩
Just in case you haven’t noticed, this is sarcasm. And if you haven’t for reals, maybe you should read something else? ↩
And a P6 that has an even “better” tax code for me! ↩
That someone being my employer’s payroll system, and the stakes being me overtaxed for a year definitely played a role here. ↩

Nightmare on Apple Street

Thu, 08 May 2025 00:00:00 GMT

*Clicks fingers, clears throat*

Okay, I have procrastinated on this post for, checks notes, 3 months now. It’s time. Time to let go of all the bad memories and the pain.

See, all I wanted to do was to create a terminal application that you can just download and run on Linux, macOS, and Windows. I also got a bit ambitious and wanted to create this app on, *gasp*, Linux! And you know, it worked on Windows. Yeah, that Windows that every developer loves to hate but secretly uses one way or another. But macOS? No no no no no, tsk tsk tsk, not so fast little boy. You need to sign & notarize your stuff, and you need to do it The Apple Way™.

The Apple Way™

The very first thing Apple wants is ~~your money~~ an Apple Developer account which will set you back $99¹. Every year that is. Oh, and you cannot just create a developer account. You see, you need One Apple Account™. If this is going to be a personal developer account, you just need your name, email, phone number, and your address². If you are trying to enroll your organization, god help you: you need your D-U-N-S number.³

Now that we have warmed up, it is time for you to create an identifier for your application. They recommend using a reverse-domain like name: com.my-company.my-app. I know, you just want to self-distribute a simple binary. Yes, you still need the unique identifier. No, you cannot use asdf or foobar. Okay head over to identifier creation page and get it over with please. I’ll await. I think you leave the capabilities empty.

Switching to the Highway

After this step we need to add a certificate to our account. Now, if you have XCode, there’s a built in UI for this. But remember, we don’t have access to macOS where XCode can only survive in. Hence we will go rogue and will create a certificate signing request (CSR) using the command line. We need a “private key” to create a CSR so we’ll be creating that via the CLI too. Might seem complicated but it is just answering a bunch of questions and shuffling some files around.

For this, we’ll be needing 2 tools:

openssl⁴
rcodesign

Let’s start with the private key:

openssl genrsa -out private.pem 2048

Now that we have generated our private key in private.pem, we can create the CSR using rcodesign:

rcodesign generate-certificate-signing-request --pem-file private.pem --csr-pem-file csr.pem

Okay, we have the signing request in csr.pem. Now we head to the page where you can add a certificate and follow the steps below:

From the gazillion options, select Developer ID Application.
Select G2 Sub-CA (Xcode 11.4.1 or later) for Profile Type.
Upload the csr.pem file we just created.
Now we should arrive at a page saying “Download Your Certificate”
Save this file as pass.cer next to the other ones and keep them safe.

Download Apple’s root certificate and convert to PEM format (Apple Worldwide Developer Relations Certification Authority)

wget http://developer.apple.com/certificationauthority/AppleWWDRCA.cer
# Convert that to PEM format
openssl x509 -inform der -in AppleWWDRCA.cer -out AppleWWDRCA.pem
# Convert pass.cer to PEM format
openssl x509 -inform der -in pass.cer -out pass.pem

Note down your cert expiration date. You’ll need to do this entire dance again some days before this date:
```
openssl x509 -in pass.pem -noout -enddate
```
Now we are going to combine everything into a p12 file. Make sure to replace Company Name in the command line arguments below with your company name or your name.
```
openssl pkcs12 -export -clcerts -inkey pass.pem -in pass.pem -certfile AppleWWDRCA.pem -name "Company Name" -out codesign.p12 -passout pass:
```
We made our codesign.p12 file not password protected so you can use it in your CI/CD pipeline without having to enter a password. If you’d rather have it password protected, run the command above without -passout pass: at the end.

Before finishing, we need to note down your Team ID:

openssl pkcs12 -in codesign.p12 -nodes | grep OU

At this point, you only need the final codesign.p12 file.

Apple’s Sacred Stamp of Approval

To be admitted to Apple’s sacred notarization service, you need to get an App Store Connect API key. If you enjoy a good read from Apple go read their documentation. For the twitchy ones, like myself:

Head to API Key Creation page
Click on the + next to Active at the top of the table.
Enter a name like Code Signing
Put Developer for the Access field
Hit Generate
Notice the Download button in the last column for the key you just created (bottom row)
Download and save the key with the name apikey.p8 next to codesign.p12
Note the Key ID somewhere
Note the Issuer ID somewhere. This is a separate section above the key table.
Now let’s combine all these 3 into a single JSON file so we don’t have to manage them separately:
```
rcodesign encode-app-store-connect-api-key -o codesign_key.json <issuer-id> <key-id> apikey.p8
```

At this point you only need the codesign_key.json file. This will be used for notarization.

The Entitled Apps

To be able to get your app notarized, it needs to have “entitlements”. This is essentially letting Apple know ahead of time, which sensitive APIs your application will be using. Then Apple’s servers will issue a “ticket” for this specific version of your app and when someone tries to run it, it will be checked and restrained to these limitations.

Since I don’t have cybernetic powers, I cannot (yet) deduce which entitlements your app needs over a blog post. That said I can at least make a recommendation. Since I did this for fossilized Node.js applications, I just copied what Node.js used for itself.

You can use this or create your own by picking and choosing from the vast array of entitlements that Apple offers. There’s also more excellent prose for those to understand deeper and follow the Apple cult even closer.

At the end of this section, I’ll just assume you have an entitlements.plist file that is properly formatted⁵ next to the binary you want to sign and notarize.

Sign here please⁶

Now that we got everything we need for signing and notarization, we can get to actual business. Signing is quite straightforward but getting the notarization right took a few tries. Let’s start with signing:

rcodesign sign --team-name <your_team_id> --p12-file codesign.p12 --for-notarization -e entitlements.plist <your_binary_path>

If you opted for a password-protected p12 file above, you can add --p12-password <password> or --p12-password-file <password_file_path> at the end of the command above.

Knock Knock Knocking on Notary’s Door

Now that we have a signed binary, we will get it notarized. We already got the prerequisites by using the --for-notarization and -e entitlements.plist parts above so we are in good hands. We still need to zip the file before though⁷.

zip app.zip <path_to_your_app>
rcodesign notary-submit --api-key-file codesign_key.json --wait app.zip
rm app.zip

We’re Done Here

Yup, we really are done. At this point you can start distributing the signed binary. People using a macOS should be able to use it without errors or warnings. If they double click on it (instead of running from a terminal), they may still see a security warning as we cannot “staple” the notarization tickets to plain binaries. To be able to do this you need to package your app as a .pkg or .dmg file but I wasn’t (and still am not) interested in learning more Apple stuff so you’ll need to figure that part out yourself.

If you want to have this process on a CI/CD pipeline you need to remember a few things:

Make sure you don’t do signing and notarization on PR branches as that means anyone who can create a PR can generate and distribute a binary with their potentially malicious changes and with your signature on it.
I don’t think you need to password-protect your p12 file but if you are using a service like GitHub you probably cannot store files as secrets. A quick hack for this is to store the base64 encoded string versions of these 2 files (codesign.p12 and codesign_key.json) as secrets. Then you base64 decode these into their respective files and continue business as usual.
Also, don’t forget to store the signed binary as the artifact of your build.

Resources

I’ve used the excellent docs Gregory Szorc created for his amazing apple-codesign project. I essentially summarized these two pages:

I also found this amazing gist for creating pkpass.p12 files from the GitHub user karnauskas and used parts of it.

Finally, I’ve used this little hack from StackOverflow for providing you with a command for creating password-less p12 files from the get go.

Thanks

I’d like to thank my colleague Daniel Szoke for his help for establishing this entire flow and proof-reading this post. I should have written this before he also got the pain to get sentry-cli signed but hey, better late than never, right? 😅

You need to scroll all the way to the bottom to see this very unimportant detail. ↩
Yep, I’m being snarky. ↩
First time I heard about it. I wish patience to people dealing with Apple. And no, I have no intention of learning more about this but you have that link there. ↩
If you don’t have openssl around, just search for how you can install it. Should be as easy as <package_manager> install openssl where <package_manager> is apt or yum or something akin to those. ↩
Being a bit picky, are we dear Apple? ↩
and here, and here, and here, and here… ↩
Don’t ask me why they cannot be bothered with on-the-fly zipping or HTTP content encoding etc., I don’t know. ↩

Making your Node.js application last centuries

Wed, 12 Feb 2025 00:00:00 GMT

I’ve been working on Sentry Spotlight for the past several months. One of the things I wanted to do was to reduce the friction on trying out and adopting Spotlight. You don’t need to know what Spotlight is (yet!) to enjoy this thriller but if you really must know, it is a local and offline debugging tool leveraging Sentry SDKs. It supports errors, traces, and very soon profiling data 🤞🏻.

One binary to rule them all

Now, where were we? Right, it was a bright San Francisco morning when I decided to create a self-contained binary for Spotlight that you could “just download” and run. Nothing else needed. Without such a binary, you either need to have node & npx or docker on your system. I think we have enough haters for both (rightfully so). Besides, I wanted to make Spotlight accessible to everyone. For instance, if you are an Android developer you probably neither have node nor docker on your system and have no reason to install any of them.

We do have the Electron app, that said we only have it for macOS, and, I don’t really like the idea of shipping an entire browser for an application that has a simple web interface and works over HTTP.¹

Enter Node.js Single Executable Applications

So, I started looking into ways to create a self-contained binary for a Node.js application. I know tools exist for Python so I was hoping that there would be something for Node.js too. I came by nexe and I was about to give it a shot when I noticed this “Node.js Single Executable Applications (SEA)” entry on Google. Sure enough, Node.js folks were adding exactly what I was looking for into Node.js itself! I quickly tried out the steps listed and started jumping up and down with some hideous dance moves in between when I got a working binary for Spotlight.

It was a bit laborious but OK for a local test. To be able to actually use this in a fully-automated CI system, there were a few things that needed sorting out:

Spotlight server needs to become a single, dependency-free CommonJS file
I need to ship the Spotlight frontend assets with the binary
I need a maintainable script to do all the above and build the binary

Single-file Node.js Application (not a binary)

Creating dependency-free CommonJS files is not something I’m unfamiliar with. I’ve first encountered this technique when I was working on Yarn quite a while ago. Back then, some smart folks at Facebook (nee Meta) realized they can pack a Node.js app into a single file just like a bundled web application². This was using a bundler such as Webpack (remember, this is 2015). I then used this technique on Craft during my first stint at Sentry. This method already makes it easier to distribute and run a Node.js application without needing to install any dependencies. But it still requires node to be installed on the system (and it needs the correct version of it).

Due to my past good memories from Craft, I chose esbuild as my trusty (and swift) bundler for the job. Just as I was thinking this was too easy, I found myself on the sidelines of the great ESM vs CJS war. As an application built in the modern times, Spotlight is using ESM modules all around. This also meant no more pesky __filename and __dirname globals and using the new import.meta instead. When you compile this into a CommonJS bundle naively, import.meta becomes an empty object, making import.meta.url undefined, making it impossible to determine where your script is running. Thankfully³, I was not the first person to bump into this and there was a simple yet crude solution that I’d happily take.

Packing the frontend assets in

The assets needed for Spotlight’s UI are not much: just an HTML page and an accompanying JS bundle. The first thing I tried was to bake these in with hard-coded names which worked just fine. But I was acutely aware that it was not future-proof at all. It is easy to add more resources to a frontend application: be it split JS chunks, some images, or separate CSS files. I could just pack everything in the dist folder where the assets were generated into, but currently, the Node SEA resources API does not have a discovery mechanism. If you know the name of the resource(s), you can read them but if you don’t GLHF.

Luckily again, all the bundlers produce a manifest.json file that lists all the resources they’ve generated and their relationship with each other. I could just read this file and pack all the resources listed in it along with the manifest file with the well-known name manifest.json. This way, I could read the manifest file and discover all the resources I need to serve the UI. And that is exactly what I did.

Now all that is left was codifying all this logic in a neat little script that I could run on my CI system and get a shiny new binary at the end. Or was it?

A wild boss appears: signing and notarizing on macOS

Of course, if it wasn’t for my arch nemesis, macOS, how could we have fun⁴? Starting from macOS Catalina (circa 2019), Apple requires all applications to be signed and notarized to be able to run without any warnings. The signature is a hard-requirement to be able to run the file at all whereas notarization is to remove the warning and prompt.

Any security-conscious developer would not eschew code signing and maybe even some sort of permission grants. That said since this is Apple, the grand builder and guardian of walled gardens, the Apple-specific way of doing these are quite tyrannical. You need to have an Apple Developer account (only $99/annum!), you need to have a Mac, you need to use XCode and its toolchain, and you need to have a lot of patience. I had none of these. I’m a creature of speed and efficiency and rebellion. I could run the signing portion on a macOS runner on GitHub Actions but I can create all the binaries (including Windows ones) on a Linux machine, with a neat list of target architectures. I just don’t want to split just that part of the process.

After a lot of reading, exploration, and trial & error, I discovered the minimal steps and required files and certificates and secrets you need to get this done⁵. I also remembered the ambitious project from indygreg, opening Apple’s code signing black box to the masses and to other platforms: apple-platform-rs. Now, with the power of rcodesign, I could sign and notarize my bespoke binaries for macOS on the standard Linux CI machines.

Take that, final boss!

A maintainable script tool for all this

With all the stuff built in, my “simple” build script became a ~200-line monster with a few support files around. It was somewhat generalized but not enough for me to share it easily with others to prevent further suffering. This is why I decided to create a tool that would encapsulate all this logic and make it easy for anyone to create a self-contained binary for their Node.js application: presenting fossilize!

Fossilize does all the things above, including macOS signing and auto-discovery of assets through a Vite-compatible manifest.json file. It also caches the Node.js binaries it downloads to speed things up on repeated builds. It supports using different Node.js versions and understands a few simple aliases such as local, latest, and lts.

One irony is fossilize itself cannot be fossilized at the moment due to some of its dependencies requiring dynamically determined native binaries per platform and some obscure issue with postject not being able to postject code containing itself. I’m planning to tackle these with the help of WASM but for now, I think fossilize is in a good place to serve the need.

Onwards 🚀

Yet I happily use VS Code and Slack. Oh the hypocrisy! ↩
They also did even smarter things like code caching to speed up start up times. Node SEA also supports this. ↩
Or unfortunately, depending on how you look at it. ↩
Hoping your definition of fun includes several days of trial & error, reading docs written as if you have to use Apple devices competently with an ambition of reaching Lord of the Rings levels of prose, and some late nights. ↩
A blog post dedicated to this journey is ~~being written as of this writing~~ available now. ↩

Docker Volume Caching on GitHub Actions

Tue, 14 Jan 2025 00:00:00 GMT

I joined Sentry to exclusively work on their self-hosted product in 2019. Back then, Sentry was just using a few services: Postgres, Memcached, Redis, and Sentry itself. But it was on the cusp of becoming a multi-service application with the introduction of Snuba and along with that Kafka, Relay, Symbolicator and others. Because it was supposed to be simple, self-hosted (or onpremise as it was called back then) did not have any tests or even any automation: just a bunch of instructions and commands to run in the README. With the rapid increase in the number of engineers working on Sentry and the changes being made, it was clear that we needed to automate the testing and setup of the self-hosted repository.

To summarize about a year’s worth of work: we created an install script based in bash (as that was the most common denominator across all platforms), and a very cursory test suite which ran the install script, tried to ingest an event, and read it back. The entire test suite took about 5-6 minutes to run and about half of that time was spent on running Django migrations, from scratch, on a fresh database, over, and over, and over. The thing is we didn’t even add migrations frequently but we still had to run them all to get the service up and running.

The solution was obviously caching but caching Docker volumes was not really a thing that seemed feasible back then. Remember, this is 2019-2020, GitHub Actions was still in its infancy. I was also barely getting comfortable with all that Bash and Docker stuff. Then I got distracted by other things, changed jobs, and eventually came back to Sentry to see that this was still a problem. So I decided to tackle it head-on. I was going to cache the hell out of those Docker volumes for our databases. We already had actions/cache now so how hard could it be? Famous last words.

I have spent about 2 weeks to completely figure this out. About 50% of this was my ignorance about basic Linux tools such as tar, file/directory permissions, and Docker’s way of storing volumes. About 30% was me not trying things locally properly and just pushing to CI and waiting for the results. The remaining 20% was the actual hard parts to figure out, mostly thanks to StackOverflow (yeah, still not on that “ChatGPT for everything” bandwagon¹). I’ll summarize some of the findings here so you don’t have to go through the same pain as I did:

Docker volumes are stored under /var/lib/docker/volumes (by default, and please don’t change it)
You cannot stat a directory or anything under it if you don’t have x permission on the directory itself (╯°□°)╯︵ ┻━┻
tar does preserve permissions and ownership by default but only if you are running it as root (or with sudo) (╯°□°)╯︵ ┻━┻ x 2
tar preserves ownership information as names and not as IDs so if your Docker container uses a user id like 1000, GLHF ² (╯°□°)╯︵ ┻━┻ x 3
Linux (Unix?) fs permissions are not just rwx but there’s also an s you can set on executables to allow them to set ownership of other things³ ＼（〇_ｏ）／
Not only GitHub Actions doesn’t run tar with sudo, and not only it refuses to do this, it also doesn’t allow you to run tar with --same-owner or --numeric-owner (╯°□°)╯︵ ┻━┻ x 4
Bonus: there are these awesome tools called getfacl and setfacl that lets you backup and restore ACLs BUT NOT OWNERSHIP INFORMATION ~~(╯°□°)╯︵ ┻━┻ x 5~~
Bonus 2: mv would happily overwrite your target without even mentioning, especially if you use sudo.

So, with all this information, what is needed to cache Docker volumes on GitHub Actions and restore them properly? Let’s see:

Set +x permission on /var/lib/docker
Set +rx permission on /var/lib/docker/volumes
Set u+s permission on tar
Use tar --numeric-owner to create the archive — oh wait, you can’t because actions/cache doesn’t let you (╯°□°)╯︵ ┻━┻^{(╯°□°)╯︵ ┻━┻^{(╯°□°)╯︵ ┻━┻^{(╯°□°)╯︵ ┻━┻}}}

Side quest: Hacking `tar` on GitHub Actions

Once I realized that I had to change the options passed to tar, I very reluctantly decided to “wrap” the actual tar executable:

sudo cp /usr/bin/tar /usr/bin/tar.orig
sudo echo 'exec tar.orig --numeric-owner -p --same-owner "$@"' > /usr/bin/tar

Oh, but wait, you cannot sudo redirect output to a file as sudo just runs the command and redirection is done by the shell which you are not running as root. Let’s try that again:

sudo cp /usr/bin/tar /usr/bin/tar.orig
echo 'exec /usr/bin/tar.orig --numeric-owner -p --same-owner "$@"' | sudo tee /usr/bin/tar > /dev/null

Once I added this monstrosity, my GitHub Actions runs… started to hang indefinitely. Can you see the issue? ಠಿ_ಠ Well, I couldn’t. I spent about 2 hours trying to figure out why this was happening. I suspected exec might be the culprit and when I removed it, the runs at least started crashing with an error: cannot fork. What? Well, see I was doing this both in my restore and save actions. So, when the restore action ran, it wrapped/replaced tar but then did not restore the original back. After some time, save action ran trying to do the same. Now remember our “Bonus 2” learning from above: when save also backed up tar (which was actually my wrapper script) to /usr/bin/tar.orig, mv didn’t even flinch when tar.orig already existed. Now I had 2 copies of my wrapper script where the second one just execed itself. Nice fork bomb there, me⁴.

Once the fork bomb was defused, I was able to run actions/cache and viola! My volumes were cached and restored properly. Space time is saved Marty!

Final boss

After all this, I was still not very happy as it made all action/cache calls in my workflow doubled, and with the same hack repeated in both parts. So I decided to create a GitHub Action that would contain the chaos, the madness, the fork bomb minefield, and all the other ugliness. Both from my sight and others’. Please enjoy BYK/docker-volume-cache-action and cache responsibly.

That said all images for this article was generated by DeepAI Image Generator ↩
Looking at you confluentinc/cp-kafka ↩
Yes, yes, there are even more. Can you believe it? I couldn’t either. But I digress. ↩
Me when I realized this: mother forking shirt balls! ↩

Having a Good Ol' RSS Feed in Astro

Mon, 30 Dec 2024 00:00:00 GMT

After reviving this blog with Astro, I realized that I didn’t have an RSS feed even though the theme I’m using already has support for that. So I got to work to enable it. For some reason, I just did not have the rss.xml file generated. After much trial and error, I finally figured out that the method name in the endpoint definition should be ALL_CAPS as in GET instead of get. I’m guessing this was because of a major Astro version upgrade since the demo page for Pacamara has a working RSS feed. Fixed that and problem solved, right? RIGHT?

Sort of.

Yes, I got a feed but I noticed 3 major problems:

The feed did not have the full content of the posts
The feed did not limit the number of entries
The feed did not sort the posts in any way

Although unsorted and uncapped posts was not a big deal as I only had 3 posts at the time, it looked like an easy fix so I started with that:

const MAX_ITEMS = 10;

const posts = (await getCollection("posts"))
  .sort(descDateSort)
  .slice(0, MAX_ITEMS);

Quite straightforward: get all the posts, sort by date in descending order, and slice the first 10. Don’t know why this is not the default or a least is offered through a built-in helper, but let’s move onto the bigger issue.

Getting the full content of the posts in RSS was much trickier as the RSS endpoint was defined as an “endpoint” and was not able to render Astro components. Even the recipe for RSS on Astro docs says this is only possible for Markdown only and it uses a custom Markdown renderer 🤯. But I was determined, and was a devotee of “the search church” so off I went to find a solution.

Although there was this very creative solution, I bumped into a more straightforward one first: https://blog.damato.design/posts/astro-rss-mdx/. This solution uses the new and experimental Astro Containers to be able to render an Astro component in isolation inside the endpoint. I followed the instructions and voilà! I had a working RSS feed with full content indeed. Committed, pushed, and got yelled at by GitHub Actions with the following cryptic error:

cannot test case insensitive FS, CLIENT_ENTRY does not point to an existing file: /home/runner/work/byk.github.io/byk.github.io/dist/client/client.mjs

I ran npm run build on my local terminal immediately, and was able to reproduce the error locally. At least, I was not going to play the “try blind commits to see what the CI says” game.

After searching for this error for about an hour, I realized that something was triggering a client-side render mode in Vite (Astro’s underlying bundler) and I started to remove every single new line of code I added. Indeed, once I disabled the import for both @astrojs/container and @astrojs/mdx the build error disappeared. As to why this was happening, I still had no idea. I kept digging and finally found this random (and very helpful) message on the Astro Containers Stage 3 proposal thread: https://github.com/withastro/roadmap/pull/916#issuecomment-2256059117

// astro.config.mjs -- add the following
{
  vite: {
    ssr: {
      external: ['astro/container', '@astrojs/mdx'],
    },
  },
}

Of course this makes sense! Without the above configuration, Vite tries to put these Astro packages into a client bundle whereas I am strictly operating in a server-side rendering world. Once this is in, the build error went away with MDX rendering still intact. I quickly pushed the code, got my deploy, and had my RSS feed! 🎉

I wanted to test my feed before declaring a complete victory so I loaded it up in Readwise Reader, my RSS reader of choice, and saw that the images were not loading. I quickly realized that I (well, Astro) was using relative paths for the images and that’s simply not how RSS works! I had to make all these image URLs absolute which was supposed be quite straightforward.

For some reason, I couldn’t find that simple answer after much searching. Then I tried to hook into the MDX pipeline to modify the URLs only to be disappointed as the image URLs are generated much later in the process and all I got was a JS identifier for the image source 🤦🏻‍♂️. After more research, I learned all about Astro’s image processing pipeline, found out about its getURL() method, dug into its source code and finally saw that it uses import.meta.env.BASE_URL as the base!

Easy, I thought: I’ll just set that in the config under vite: {base: '...'}. That didn’t work. Then I tried setting it on the top-level Astro config only to be disappointed again. I also tried some other, sillier things that I don’t want to admit doing here. Finally, like really finally, I found the answer in build.assetsPrefix! Set this to my blog’s main URL, tested in dev mode to make sure it still works, got a full build, checked the rss.xml output and saw that the image URLs were now absolute! 🎉🎉🎉

So, if you ever want the same (Astro blog with full content RSS feed and working images), I hope I can save you some hours with this post.

Oh, by the way, if you want to subscribe to my blog, now you can 😏

Roots

Wed, 25 Dec 2024 00:00:00 GMT

It started when I was about to turn 6. The day when we were moving from my first home. I was losing everything, including our kitchen sink¹. The sink I was using since I was born.

Then I started school, started biking, and it was all good. For a while. I even had a girlfriend². Then I changed schools, lost some of my friends, and definitely lost my girlfriend of 1 day. Resettling took a few years. We moved 2 times during this time but the school didn’t change, hence the friends stayed.

And then I changed schools again. Lost most of my friends again. I was not terrible at making new friends but the churn takes its toll. Some years passed, then we moved again and came university. The great thing about university is that it lasts at least 4 years³ and they tend to stay in the same place. This definitely helped me to settle down a bit. I got my first job, made a few lifelong friends, met and started dating my future wife, and started to feel like I belong somewhere. On my 5th year of university though, the chaos started again: most of my friends graduated, I switched jobs, and I was very close to graduating. Then I graduated, got engaged, moved to yet another city for compulsory military service, then got married, moved to San Francisco for a new job, and then moved back to Izmir just after 11 months.

I have never been able to stay in one place (city or home) for more than about 3 years since I was 6 and I am 36. That’s a lot of years and a lot of movement. There’s no single reason for this. We moved a lot when I was a kid as my father was a military officer, then I moved to become a real adult, then I had to move across countries for work. With every move, you lose friends along with some part of you. Settling into a new place can be extra hard when it is an entirely different country, let alone a city. New people, new culture, new everything.

I guess this is the curse of trying to escape your fate in a weird, almost-developed-but-not-quite country which you cannot completely leave behind, but also cannot come back to for a safe and secure life.

I don’t know why I’m publishing this. May be I’m tired⁴ or may be it was just a long writing exercise to share one of my favorite songs, “Roots” by Alice Merton:

I like standin’ still, but that’s just a wishful plan Ask me where I come from, I’ll say a different land But I’ve got memories and travel like gypsies in the night

I’ve got no roots, but my home was never on the ground

I asked my mum whether we were taking the sink with us too. ↩
Yes, I took a girl to the local movie theater with a romantic walk when I was 8. It was the talk of the town — at least between 8-year-olds. ↩
Yes some of you have the smarts to do it quicker, and some people take longer. Topic of another story. ↩
Just checked, yes I am tired. ↩

Life Lessons from a Rotary Encoder

Thu, 24 Mar 2022 00:00:00 GMT

Recently I got back into an archaic pastime activity of mine: working on hobby electronics. I already had a breadboard lying around but since I am lazy I wanted an Arduino board, complete with all the things I may possibly need: push buttons, LEDs, a 7-segment display, a dot-matrix LCD etc. I bought a TinyLab experiment board based on the recommendation from my poorer self from 6 years ago on Facebook. I know Facebook is an evil machine forced upon us by our alien overlords but it has a tender side surfacing ancient wisdom via its memories feature. Anyway, one of the crucial and interesting components on the board was the rotary encoder. With its infinite, tactile rotation and the new-found popularity among the mechanical keyboard community, this little knob quickly became my new obsession. Trying to read from what’s passing through its contacts would lead to profound realizations about life, universe, and everything — moving our understanding of 42, an inch forward.

My rotary encoder has 5 contacts: VCC, ground, Phase A, Phase B, and push button. It is of the mechanical type. A mechanical encoder means there is mechanical contact between these A and B pins and the rotating disk inside. It is a lot like a pair of buttons being smashed in perfect order tens of times per second. The most interesting part is the rotation direction detection which is roughly done by figuring out which of these buttons get smashed first. The reason these pins are called “phase” contacts is because when the encoder is rotated, you get a square wave out of them which are out of phase by 90 degrees. This is so that you can determine the direction of rotation based on whichever is ahead of the other.

As a self-thought programmer who’s been writing code for 24 years on “ideal” computers, I didn’t even bother to learn much of the above at first. For me, this was simple:

there are two buttons
in each loop you read the values of these
if both are 0 = no rotation
if one is 1 and the other is 0 = encoder is turning in the direction of the pin that is on

Oh the arrogance even at this ripe age of 33! I was so wrong that it took me a whopping 3 days to reliably read this tiny little marvel of electromechanics. Let’s start with the “obvious” issues:

Reading 0 from both pins does not mean we are stopping
It is possible to read 1 from both pins and the one being 1 does not dictate the direction by its own
If you read the values in each loop, you may actually miss values as they don’t sit there and wait for you to read. Life goes on, the encoder keeps rotating, and if your main loop is slow you miss your window of opportunity

So that’s 3 out of 4 from my initial assumptions. At least, I was right about there being two buttons. Sort of.

The solutions were simple but profound. For timing, you prioritize reading rotary encoder inputs by using pin interrupts. An interrupt is a special instruction in micro-controllers that tell them to drop whatever they are doing and attend a special task. It is a lot like when your kid starts screaming: you drop whatever you’re doing and immediately ~~shush~~ soothe her. Luckily, rotary encoders are less demanding than toddlers: they just want to be heard (well, maybe kids want just that too?). So we read and store the state of these A and B pins when there’s a change. Ideally we’d set up the interrupts on both pins but due to the wiring of my board, and some limitations of the Atmega 32u4, I could only listen to one pin. This is very much like hearing only from one of your ears as the other one is gone due to the earlier screaming. Not terrible but you just have to accept the fact that you may not register the initial click in one direction. Again, definitely much less worse than losing your sense of sound directionality along with your will to live after getting yelled at your right ear just because you shut the water faucet off that has been running for the past 187 seconds for the entertainment of a little human’s growing mind.

The solution to detecting the direction of a “click” is also simple: just do some book keeping. Record which pin got triggered first, then on the next cycle compare its value with the new state. For instance if you saw A go high (meaning it switched from 0 to 1), while B is 0 you are rotating in one direction (phase of A is ahead of B). If B was 1 while A was going high, that means it is rotating in the other direction as phase of B is ahead of A (or they are playing a weird version of beer pong). Since this is a bunch of if statements and some variables, I wrote it up and tested quickly. I surely was able to read every single click on the encoder, that said the direction was completely unstable. Just like a toddler learning to ride a scooter, the direction was flipping like crazy. This made no sense at all (except for toddlers and scooters). Computer chips and solid metal disks listen to reason unlike 2-year-old human beings!

I tried blaming the compiler gods but they were too busy torturing my boxed copy who is learning Rust from a borrowed future memory segment¹. Thus, I seized this rare moment where I got to put my computer science knowledge to work. I was going to build a state machine as one wise blog post suggested. The idea is deceptively simple: not all states you can read from the pins are valid states. For instance, if you look at the wave picture above, you’d see that when you are turning in one direction, you should never see pin A from going low to high while B is 0. So you construct a state table, listing all states and valid state transitions you may accept, and ignore everything else. Doing this fixed all the weird direction jumps! The little cost I paid was some skipped clicks very occasionally but that’s a little price to pay for stability.

Although this was a success, I was simply wondering why I had to use actual math and science and how these invalid state transitions could happen in the first place. After some googling, it finally hit me: nothing is ideal, especially mechanical contacts! We can model them and show diagrams like the ones above as their idealized approximations but real world is just messy. It was simply the stuttering of these imperfect copper contacts, sending hysterical signals to my code which expected a perfect square wave. No wonder why it was confused.

In the end, I was quite surprised by getting smacked in the face by a rotary encoder with bitter truths about life:

timing and catching your window of opportunity is very important
life is just messy, no matter how much you try to smooth it out

Rust is a newish programming language that has a notoriously high learning-curve but provides great memory safety. ↩

Read at BYK's

Adaptation: new tools in town!

Footnotes

Releasing Packages with a Valet Key: npm, PyPI, and beyond

Secrets in Plain Sight

The Valet Key

Enter getsentry/publish

The Implementation (with Craft)

Fighting Overprotective Parents

Happily Ever After

Why This Matters Today

Closing Thoughts

Thanks

Footnotes

Marking it Up (and Down)

First, there was plain text

The Age of AI

Walking back from the X-factor

Putting it everywhere, and fast

Can’t Stop the Feeling!

Tying it all together

Footnotes

The magic word: TaxYearEnd

The Payslip

The Denial

A game of letters

The “fineprint”

Being practical

The depth of the rabbit hole

Conclusion

Footnotes

Nightmare on Apple Street

The Apple Way™

Switching to the Highway

Apple’s Sacred Stamp of Approval

The Entitled Apps

Sign here please6

Knock Knock Knocking on Notary’s Door

We’re Done Here

Resources

Thanks

Footnotes

Making your Node.js application last centuries

One binary to rule them all

Enter Node.js Single Executable Applications

Single-file Node.js Application (not a binary)

Packing the frontend assets in

A wild boss appears: signing and notarizing on macOS

A maintainable script tool for all this

Footnotes

Docker Volume Caching on GitHub Actions

Side quest: Hacking tar on GitHub Actions

Final boss

Footnotes

Having a Good Ol' RSS Feed in Astro

Roots

Footnotes

Life Lessons from a Rotary Encoder

Footnotes

Sign here please⁶

Side quest: Hacking `tar` on GitHub Actions