Jason Maa

Setting up a Self-Hosted Single User Mastodon Instance

Sun, 08 Jun 2025 00:00:00 +0000

I’ve been self-hosting a single-user Mastodon instance for some time now. I figured it’s probably time to write up my experience setting it up so I can share my experiences with the rest of the world (and so I can also have it documented somewhere once I inevitably forget how I did do it).

Dude, Where’s My Pi?

Most stories start from the beginning so that’s where we shall start too.

It was early 2023. News of Silicon Valley Bank’s collapse was all the talk, COVID still existed (depending on who you asked), and I still could not get my hands on a Raspberry Pi. You could thank the global chip shortage for that. What I did manage to find, however, was an alternative product from Libre Computer called Le Potato.

Being the trigger happy guy I am, I purchased one instantly without thinking and soon found myself with a SBC and nothing more. Turns out that I needed a little bit more than just the board. Namely, I was still down a microSD card for storage, a microUSB cable for power, and an ethernet cable for internet. One more virtual grocery run later and I had a bunch more stuff with still absolutely no idea what to do with it. The Potato is pretty bare bones; it does not come with an instruction manual, so it was off to the Internet again, this time to figure out how in the world to get Linux on the thing.

Luckily, I stand on the shoulders of giants. Many others out there have already experimented with Le Potato, and the Libre Computer forums even offer a getting started guide which links to more resources for setting up various flavors of Linux. In terms of setup, I found both this video tutorial by Shotoku Tech and this one by Vincent Stevenson to be extremely helpful. After much trial and error, I ended up just following Shotoku Tech’s guide nearly step for step, flashing Armbian Jammy using balenaEtcher. I plugged everything in and turned on the monitor again but no luck. Just as I was about to go through another Sisyphean cycle of re-flash and reboot, suddenly, the monitor lit up, showing Linux booting up briefly, before turning off again.

What the fu-

A few more tries later with varying levels of success, and I discovered that the Potato’s LED blinking pattern was different the times when I was able to get Linux to boot. Before, only the Potato’s red and green LEDs were on. However, when Linux booted, the blue LED would also begin flashing. Some part of my setup was causing the Potato to be unable to boot. Searching the Libre Computer forums, I eventually found out that most deviations from the LED blinking pattern were usually due to a bad power supply. The Potato expects 5V of power. I looked at my Potato’s setup and there, connecting wall power to the micro USB cord charging the Potato lay the culprit, the power adapter of a shitty USB charger I had gotten for free at a math competition many moons ago (see appendix).

After swapping out the power adapter, the Potato seemed to boot reliably. I hooked up ethernet and my USB keyboard to the Potato, set up root and a personal user, and after a few more steps, a beautiful login screen greeted me. The login screen also displayed the Potato’s local IP which meant I was now able to SSH into it from my personal computer (yay!).

Throughout 2023, I ended up running a personal Discord music bot off of my Potato. During this time, I would also purchase two siblings for it, scaling up my Potato cluster to a grand total of 3.

Aside from attempting to self-host a Komga and Jenkins server via Tailscale Funnel though, not much else happened with these Potatoes. However, flash-forward to 2025 and things began to change. This is because I would start getting the urge to self-host a single-user Mastodon instance.

A Mastodon of My Own

Self-hosting meant that I needed to turn one of my Potatoes into a Mastodon server. Currently, I have three Potatoes named serval, gepard, and lynx (see appendix B). serval was already running a bunch of cron jobs and gepard already had too much stuff on it which meant the only candidate left for my instance was lynx. I cleaned up lynx a bit and got started.

My entire setup pretty much followed the Mastodon setup guide line-for-line. I did run into a bit of trouble with installing yarn, but it turned out this was due to lynx having an older version of Node. APT kept installing the older Node version even after I uninstalled Node and set the package source to Node 20. Eventually, I was able to get it to work by fiddling around and uninstalling enough dependencies so that APT wouldn’t keep installing the older Node.

The rest of the guide went mostly without a hitch, except that I did have to settle on a domain name. Once I got one, the next step was to get an SSL certificate for my server. This meant it was finally time to figure out how I was going to expose lynx to the Internet. I was originally planning to use Tailscale Funnel but soon found out that Funnel only allows DNS names that are a subdomain of the tailnet’s domain. I had already bought my own domain, and I also didn’t really want to call my Mastodon server something like foo.ts.net so that was a no-go.

After some more quick research, I ended up going with an alternative and setting up with Cloudflare Tunnel. Following the tunnel setup guide, I was able to register a domain name with Cloudflare and set up the tunnel. The entire process was very similar to using Tailscale Funnel and simply involved running the cloudflared daemon on lynx. However, trouble started brewing once I started configuring the routing.

I had initially set up my domain to route to http://localhost. When I tried to visit the domain, however, the browser ended up looping repeatedly. Digging a bit more, I found out that the cause was due to the nginx rules for Mastodon always attempting to upgrade HTTP to HTTPS. This meant that I had to point the tunnel routing to HTTPS instead. However, when I tried this, I ended up with a 502 Bad Gateway error. Searching around the Internet again, it turns out a few folks had fixed this by enabling the No TLS Verify option (see appendix C). I turned it on and, lo and behold, it worked!

I quickly created a new account on my server and the rest is history.

To See the Future

As of this writing, my Mastodon server has been online for 18 consecutive days. This is longer than what I had initially expected, especially since I know full well I am cheaping out on many parts of this setup. However, the instance seems to be running fine so far, barring some occasional lag. My only complaint is that it is a bit lonely with just myself, but I’m still managing for now.

Looking forward, probably the most notable risk I’ve taken going the self-hosting route is that it’s now also my job to manage the server’s infrastructure and content. This means it’s up to me that it doesn’t go down and doesn’t end up ingesting any undesirable content. As they say, “With great power comes great responsibility.” However, I honestly see this as more of an opportunity than a burden. I’ve had few chances to actually run and maintain a long-running service on an actual Linux box (Kuiperbowl being the only other), so I feel that this will be something I can use as a learning experience. Even if something catastrophic does happen, I won’t let anyone else down. This is a single user instance with just me after all.

There’s still a lot more I have to do as part of this. Most of this involves figuring out how to monitor both Mastodon and my Linux host, how to perform and manage Postgres and media backups, how to keep Mastodon up-to-date, and so on. I also need to get more familiar with Mastodon’s admin tools and do some work finding more ways to bring content I like to my instance.

Anyways, assuming all goes well, and the instance doesn’t disappear because I get lazy, bankrupted, ill, or arrested, then perhaps there will also be more posts about my Potatoes and the Mastodon in the future. But only time will tell. Until the next one!

Appendix A: My Love is a Phone Charger

Let’s step backwards in time. The year was 2017, and the world was quite different. Donald Trump had recently become president of the United States, COVID-19 had not yet rocked the world, and I was a bright-eyed kid in high school taking a trip up to CMU with a few friends to participate in the annual CMIMC math competition. Unlike my friends, I was (and still am) not very good at math. I was mainly there for the bagels and free swag.

A few companies were also there, I assumed not to do math but to scout for talent instead. To lure youngsters by and convince them of a promising future of corporate servitude, these companies had booths chock full of freebies. Like everyone else, we completely fell for these. Our group ended up making off with a bunch of pens, hoodies, pajamas.

For me, the most important thing I walked off with was one of these 12-foot long chargers that Two Sigma was handing out. It was love at first sight, because the charging cord was long which meant I could power my phone and still roll around all day in bed while facetiming my then-girlfriend. However, my friends and I soon found out that these chargers were, to put it bluntly, not very thermally efficient (and also probably very flammable). Most of us ditched them soon after, but I kept using mine.

Winter blossomed into Spring and my beloved 12-foot cord would stop working soon after that, leaving only the power adapter as the sole survivor. Although the adapter always generated a ridiculous amount of heat, I would keep using it as part of my primary phone charger setup, taking it with me into college and the years beyond.

Fast-forward to around Fall of 2023, during a trip to San Francisco with those very same friends, I would pull out my charger, and one of my friends would casually point it out and pose to me the following question:

“Jason. How the fuck are you still using that piece of shit?”

Appendix B: Machine Names

My Potatoes were originally nameless which was not great since just like children, without names, it is very hard to tell them apart. Since I was (and still am) into Honkai: Star Rail, I ended up naming them serval, gepard, and lynx after the Landau siblings.

The process to change the names was quite straightforward. I just SSH’d into each one and ran hostnamectl to change the name:

hostnamectl set-hostname lynx

Of course, this naming scheme will become problematic if I ever get a 4th Potato. I hope before this happens, Hoyo will introduce an estranged 4th Landau sibling who was lost in a snowstorm as a child and raised by a pack of wolves.

Appendix C: Fun with TLS

I’m still not entirely sure what the no TLS verify does and why it worked.

My best guess at what’s happening is that there is some sort of SSL certificate mismatch. Cloudflare Tunnels presents its own certificate for the website. However, since the Mastodon setup guide assumes nginx will be directly exposed to the Internet and not through a proxy, it guides you to generate a certificate from Let’s Encrypt to use with nginx. When the tunnel talks to nginx, nginx uses the Let’s Encrypt certificate which is not the same as the certificate on the Cloudflare side, causing the error seen in the original setup. However, with the TLS verification disabled, this mismatch doesn’t matter anymore because the tunnel won’t check that the certificate from nginx is mismatched with the Cloudflare certificate.

Most likely what I should’ve done is used Cloudflare’s origin server certificate with nginx instead of the Let’s Encrypt one like this person did to resolve the mismatch. However, whatever I have is working, so I’m just going to leave it alone for now. I believe the only risk I’m running is that I could get MITM’d somewhere in-between Cloudflare and lynx. Perhaps properly fixing this (or attempting to do so) will be its own mini project for some future weekend.

The 1000 AI Domains that Really Really Really Really Really Want Your Attention (and money probably)

Mon, 07 Apr 2025 00:00:00 +0000

Last week, I saw an interesting entry on the front page of Lobsters. It was a blog post about a math puzzle that the author had found on a telephone pole in the streets of San Francisco. The author had tried to solve the puzzle but eventually gave up and ended up writing a blog post about his experiences. Thinking I would have better luck, I also tried my hand at solving the puzzle, gave up, and am now writing a blog post about MY experiences.

However, unlike the original post, this isn’t going to be a story about math or puzzles but rather something else entirely. That’s because after a sound defeat at the hands of a telephone pole math puzzle, my lizard brain did a backflip and immediately thought it would be a fun idea to try to brute force the solution and see what would happen if I just tried visiting a bunch of numerical .ai pages. So thus began my journey to see every website from 1.ai to 1000.ai.

Background

Before we begin, let’s understand a bit more about the .ai domain. Officially, .ai is the TLD of the country of Anguilla. In practice however, the TLD is more commonly used to create vanity domains showing a site’s relationship to artificial intelligence instead, usually to market a product or a startup.

One thing to note is that .ai domains are generally quite pricey. On anecdote alone, the average .ai domain goes for about $100 to acquire and $100 / year to renew.

Special domains, like common words or numbers, however, can go as high as $10,000 to acquire with the same $100 annual renewal fee.

All this is to say that the sites I was visiting were less like the digital storefronts or houses that people commonly parallel websites with and more like Internet mansions, highly valued pieces of property that some person bought with a lot of money at some point. This made it all the more interesting since I wanted to know what kind of people had bought these domains and what they were setting up behind them.

Planning a trip

My next step was figuring out how to visit all 1000 sites. Doing it manually was going to take forever, so I wrote a simple Python script to iterate through a numerical range. For each number, I had Selenium visit the site and screenshot the page.

from selenium import webdriver
from selenium.common.exceptions import WebDriverException
import urllib3
import time

driver = webdriver.Chrome()
driver.set_page_load_timeout(10)

for i in range(1, 1001):
  try:
    url = f"http://{i}.ai/givemeprize"
    print(f"Trying {url}...")
    driver.get(url)
    time.sleep(1)
    driver.save_screenshot(f'./screenshots/{i}.png')
  except WebDriverException as e:
    print(e)
  except urllib3.exceptions.ReadTimeoutError as e:
    print(e)

driver.close()

One issue with this process was that due to the unpredictable nature of each page I tried to visit, earlier versions of the script would often crash, either due to domains not resolving or servers stalling and eventually timing out the client. I did eventually fix these by tweaking my Selenium configurations and adding error handling, but it was still annoying to deal with these edge cases.

Diamonds in the rough

Eventually, the script did finally visit all 1000 pages. Exactly what did it find? Well, unsurprisingly, most of the hits were either parking pages or generic 404s. However, a few of the pages did catch my attention:

0-9

Nothing very interesting to see here, but it looks like these domains are all owned by a single person. ICANN lookup indicates that the owner appears to be an associate professor who’s also been on the Internet forever. Both the owner’s academic background and the fact that these domains are directly registered by him and not through a third-party registrar makes me think these domains might be planned for some sort of Internet infrastructure. However, it’s anyone’s guess as to exactly what these domains will be used for (if anything at all).

14

This appears to be a Tumblr blog stylized like a retro Windows 98 desktop. The blog itself appears somewhat young and devoid of content. Most of the posts seem to be community-submitted and are either AI art showcases or ads. There’s not much else to say about this one other than the fact that it’s 1000x more interesting than either of its neighboring domains.

255

This site returns a JSON response, claiming to be a site for robots. However, further entry to the site requires solving an “anti-human CAPTCHA” first. The challenge in question is “stopwar” which I found to be both hilarious and depressing at the same time. I do wonder what would happen to the site if humanity collectively stopped all conflicts (or the more likely scenario, when humanity eventually does die out).

418

Claims to be HTCPCP compliant server. HTCPCP is a facetious protocol defined in a 1998 April Fool’s RFC. The protocol is infamous for having introduced Status 418: I’m a teapot which is of course also the very number in the domain name. Whether or not this server actually brews real coffee remains a mystery.

455

This appears to be an AI app of some kind (the marketing homepage is somewhat vague about exactly what, and I’m not signing up for an account to find out). What really caught my attention for this one was the EVA themed 404 page though. Pretty. Cool.

727

This is a redirect to a Vtuber’s Twitch. The Vtuber in question is called Neuro-sama. Unlike most Vtubers, however, Neuro is not a human; it’s an AI. The Vtuber was created by a guy named Vedal and functionally appears to be an LLM strapped to text-to-speech with a few more add-ons and AI systems attached that allow her to do things like respond to her creator and Twitch chat as well as play games like osu!.

What surprised me the most about Neuro is how popular she is. Apparently, she is the 7th most subscribed channel on Twitch as of January 2025. For all the criticism that AI generated entertainment gets, I found it odd to see an AI Vtuber on the other end of this spectrum, getting lots of love and attention from the public. At first I chalked it up to just the passing interest of others towards the novelty of having an LLM as a Vtuber. However, after watching a few of Neuro’s stream compilations as research for this blog post, I actually found myself enjoying her content quite a lot. Even though Neuro’s voice and personality give off a very robotic and artificial vibe, there’s some sort of inexplicable charm to her absurdity. I found myself laughing along as I listened to the banter between her, Evil Neuro, and their creator, Vedal. Personally, for all the hype in AI products today, I feel that it is perhaps AI companions like Neuro which will end up surviving in the long run.

After all this, one major question still remained for me though, and that was what in the world does Neuro have to do with the number 727? Perhaps it’s just some inside joke I don’t get? I guess I’d have to become her fan to find out.

Closing words

While this journey through cyberspace from 1 to 1000 was mostly just a bunch of barren pages and ads, there were enough small gems to make it worth my while.

It was particularly interesting to see the wide array of applications people were making use of their .ai domains for, whether that be selling real products, making jokes, promoting ideas and idols, or the tried-and-true practice of shilling crypto, each of them felt like a window into someone else’s world. I think it just goes to show that even in 2025, a time where people lament the collapse of the old Internet and the centralization of media on corporate-owned platforms like Instagram or Twitter/X, there are still fun little worlds floating out there in the Ether waiting to be discovered.

Anyways, until next time!

Appendix A: On Visiting Random Websites

Trawling down a list of unknown websites is kinda like trying a series of doors on a row of buildings. Sometimes you open the door, and it’s a quirky little coffee shop while other times, it’s a dank and dusty crack den. All this is to say that visiting a slew of random websites in Selenium probably wasn’t the smartest thing I’ve done so far this year.

Many of the websites I encountered were luckily just parking pages, but there were definitely a number of extremely shady sites. Even though I knew from a technical standpoint that the Selenium browser session is separate from my personal one, so any attempt to steal cookies or run CSRF probably didn’t work, the whole thing still just made me want to scrub my laptop down.

Appendix B: The Actual Answer

As I mentioned before, I found this puzzle off of Lobsters. I imagine so did a lot of other people. Typically, throwing a bunch of brains at a common problem often has the uncanny result of solving it. I thought I might as well document how the rest of the puzzle solving played out (since I didn’t really contribute to solving it at all).

Most of this puzzle was actually already figured out by the author of the original blog post. However, a few details were missing or off:

BusyBeaver(4). The answer they wanted was the one that gives 13.
log*(16). Apparently this means the iterated log2 of 16 which turns out to be 3.
The prefix function. This seemed to have been the major stumper in the original blog post. Another Lobster had thought it meant the Knuth-Morris-Pratt prefix function, but that didn’t work either. Eventually someone(s) figured out that it actually meant the SI prefix of the input.

Putting it all together we get:

prefix(
    [
        6
        + argmax(
            softmax(
                [0.693147181 , 5.85987448]
            )
        )
        + 3
    ] ^ [
        13
        + 4
        + 1
    ]
)

.ai/givemeprize

Which reduces to:

prefix(10^18)

.ai/givemeprize

The SI prefix for this is exa.

Visiting exa.ai/givemeprize results in a job ad. In all fairness, this is not an unexpected result considering that the problem originated from a flyer on a telephone pole.

In other news, it looks like ChatGPT managed to solve this puzzle as well. Maybe someone should hire it :P.

An Analysis of Cosmic Lucky Prize

Sat, 01 Feb 2025 00:00:00 +0000

What’s the best way to simultaneously get millions of weebs excited to play a gacha game AND to also motivate a tired guy to revive his blog for the nth time in his life?

A lottery it seems.

One of the games I play, Honkai Star Rail, is currently running an event titled Cosmic Lucky Prize. While most HSR events feature minigames, puzzles, or even full-blown side stories, Cosmic Lucky Prize is weird in that it offers nearly no gameplay at all! Instead, every day for 7 days, players will be given two choices: gamble or don’t gamble. Don’t gamble and you get a flat 100 jades, but gamble, and you could win a whopping 500,000 jades! Enough to roll gacha 3125 times!

While Cosmic Lucky Prize appears deceptively simple on the surface, a closer look at the event and its rules reveals a rather interesting problem of risk and reward underneath its veil of riches and fortune.

The Game and its Rules

In order to analyze this game, one must first know how it works. Cosmic Lucky Prize’s rules are quite straightforward. A drawing happens every day for 7 days. In each drawing, players can choose to either gamble or bail.

If a player gambles, they enter the pool of all other gamblers. From the pool, a limited number of participants are first selected to be Superstars based on a daily quota. The quota allocates 2 players on days 1, 2, and 3; 3 players on days 4, 5, and 6; and 5 players on day 7, adding up to a grand total of 20 Superstars for the entire event. If chosen, Superstars will receive 500,000 jades. They will also become exempt from being chosen as Superstar again.

After Superstars are selected for a given day, the rest of the gambler pool is divided up for first and second prizes. 10% of the pool is chosen for first prize and receives 600 jades while the remaining 90% get second prize and receive 50 jades.

Everyone else who bailed or didn’t participate automatically gets a flat 100 jades.

I’ve pasted the full rules below:

● In phases 1, 2, and 3, there will be 2 Trailblazers per phase who have a chance to become the Lucky Superstar. In phases 4, 5, and 6, there will be 3 Trailblazers per phase who have a chance to become the Lucky Superstar. In phase 7, 5 Trailblazers will have a chance to become the Lucky Superstar.

● In each phase’s lottery, if a Trailblazer participates but does not become the Lucky Superstar, there will be a 10% chance to win the First Prize and a 90% chance to win the Second Prize.

● In each phase, Trailblazers can only win one of the following: Lucky Superstar, First Prize, or Second Prize. The rewards cannot overlap.

● Across all 7 phases, a Trailblazer can only become the Lucky Superstar once. The same Trailblazer cannot be selected as the Lucky Superstar multiple times.

Math 7th

When I first heard the rules, some things became immediately obvious. For one, the Superstar prize was virtually unobtainable. One source reports HSR at over 5 million players globally (and I suspect this number is even higher in practice). This means that even if only 1% of the player base gambles in the lottery, the chance of being a Superstar is still orders of magnitude below 1%. The Superstar prize acts more as a marketing scheme than anything to drive player excitement towards the event since your chance of winning 500,000 jades is slim to none.

What ended up being more interesting for me, however, were the mechanics behind first and second prizes. Ignoring the Superstar prize and applying some probability, we can compare the expected number of jades awarded in a single drawing for a player who bails versus a player who gambles.

For a bailing player, there is a 100% chance that they receive 100 jades:

\[\begin{align} & E[X] = 1 * 100 \\ & = 100 \end{align}\]

So a bailer gets on average 100 jades per draw.

For a gambling player, there is a 10% chance that they receive 600 jades or a 90% chance that they receive 50:

\[\begin{align} & E[X] = 0.1 * 600 + 0.9 * 50 \\ & = 105 \end{align}\]

So a gambler gets on average 105 jades per draw.

Comparing these values alone, it seems like a better bet to always gamble since this will net you slightly more jades in the long run. However, this assumption really only holds when the number of draws is high. Unfortunately, this is not the case for this event; only 7 draws will ever happen.

So how does gambling compare with bailing when we account for the fact that there are only a small number of draws? Let’s instead calculate the winnings and chances for each outcome of a player who always bails versus one who always gambles. Let us call these personas the ColdFeet player and the ReachForTheStars player respectively.

The ColdFeet case is straightforward. There is a 100% chance that the player earns 100 jades every draw. This will net them 700 jades by the end of the event.

\[\begin{align} & p = 1 \\ & v = 7 * 100 = 700 \end{align}\]

The mean and standard deviation for this case are pretty straightforward since there is no variation:

Mean	SD
700	0

On the other hand, the calculation for the ReachForTheStars player is more complicated since they have a chance of losing the gamble and getting second prize. We can calculate the chances and winnings of all possible outcomes for ReachForTheStars by augmenting on the number of times that the player loses the gamble.

Let’s start by finding the chance and payout if ReachForTheStars gets second prize all 7 days.

\[\begin{align} & p = 0.9^7 = 0.48 \\ & v = 7 * 50 = 350 \end{align}\]

What if the ReachForTheStars wins first prize only once? We have to choose one of the days for the gambler to win first prize. This can be done with a combination. After that we have to figure out the chance that they win 1 first prize and 6 second prizes. The calculation is then as follows:

\[\begin{align} & p = 0.9^6 * 0.1^1 * {7 \choose 1} = 0.37 \\ & v = 6 * 50 + 1 * 600 = 900 \end{align}\]

We can generalize this for any number of times ReachForTheStars wins first prize. Suppose the number of times they win first is $t$. The probability and payouts are then:

\[\begin{align} & p(t) = 0.9^{7-t} * 0.1^t * {7 \choose t} \\ & v(t) = (7-t) * 50 + t * 600 \end{align}\]

Applying the formula for all possible values of $t$, we can calculate the distribution of jades won:

t	Probability	Value
0	0.48	350
1	0.37	900
2	0.12	1450
3	0.023	2000
4	0.0026	2550
5	0.00017	3100
6	0	3650
7	0	4200

Using some probability formulas that I definitely didn’t need to look up to jog my memory, we can also get the mean and standard deviation of this distribution:

\[\begin{align} & E[X] = \sum_{i} p(x_i) * x_i \\ & Var(X) = E[X^2] - E[X]^2 \\ & SD(X) = \sqrt{Var(X)} \end{align}\]

Mean	SD
735.00	436.55

As you can see, even with a small number of draws, always gambling still seems to be the better strategy, winning more jades on average than always bailing. Even winning first prize just once will put a ReachForTheStars player ahead of a ColdFeet one by 200 jades.

However, this all comes at the cost of some risk. The chance of winning first at least once is a measly 52%, just over half. Moreover, the effort that ReachForTheStars has to go through is also much higher. They have to have the discipline to log in and choose to gamble every single day of the event.

This led me to my next question: What happens to the other types of players who don’t fall into either of these personas, that is, players who don’t always bail or gamble? Is there a better strategy than always gambling? I didn’t really want to do more math, so it was time to run a little experiment instead!

A Simulated Universe

I ended up writing a simulation to model the Cosmic Lucky Prize lottery. The simulation runs the 7 draws with a population of 5 million. In addition, each of the simulated players acts based on different strategies I think real players will use. The strategies are as follows:

ColdFeet: Never gambles, always bails. Guaranteed to get 700 jades.
ReachForTheStars: Always gambles no matter what unless they get Superstar.
FirstPrizeEnjoyer: Gambles until they win first prize once and then stops gambling.
Latecomer: Only gambles during the last 4 days. This models players who are either late to the event or get pressured into gambling later because of FOMO.
Quitter: Always gambles on the 1st day and becomes less likely to gamble with each subsequent day. This models players who lose interest or suddenly get cold feet.
RandoNoob: Gambles randomly. This models an infrequent and casual player.

After making a few optimizations to my code so that it could scale to 5 million players, I divided the population randomly among the different strategies and ran the simulation.

This is the mean and standard deviation of the jades earned by the entire population after filtering out outliers:

Mean	SD
696.38	287.96

The overall distribution doesn’t tell us much about the strategies though, so I also grouped the data by agent type and ran the same analyses:

Agent	Mean	SD
ColdFeetAgent	700.00	0.00
FirstPrizeEnjoyerAgent	726.10	367.11
LatecomerAgent	669.54	253.98
QuitterAgent	690.06	282.22
RandoNoobAgent	691.40	265.69
ReachForTheStarsAgent	699.57	383.16

Observations

Taking a look at the distributions, we can see that most of the data is concentrated in small humps throughout the graph, with the humps decreasing in size as the number of jades increases. We see this pattern for both the overall distribution and for the distributions grouped by agent type. These humps seem to represent groups of players who won first prize a certain number of times. Winning first prize would essentially bump a player into the hump for the next bracket up.

Of course, the best performing agents were the ones that could land more reliably in higher bracket humps. Sorting these agents by performance from best to worst, the agents ranked as follows: FirstPrizeEnjoyer, ColdFeet, ReachForTheStars, RandoNoob, Quitter, and finally Latecomer.

Only FirstPrizeEnjoyer, ColdFeet, and ReachForTheStars outperformed the overall average. That being said, both FirstPrizeEnjoyer and ReachForTheStars also had greater standard deviations than the overall standard deviation, indicating higher risk when taking these strategies. ColdFeet was the only strategy beating the overall average that also offered lower risk (since these players obviously never gambled).

One additional observation is that the distributions of Quitter and RandoNoob are very similar in shape. This seems to indicate that quitting early was no different than playing randomly.

Despite the differences in agent performance, the actual outcomes per agent type did not actually seem to be that much different in the grand scheme of things. Even without doing any rigorous comparison of the distributions, it’s pretty clear that all of the agent averages hovered around 700 jades. The standard deviation for even the riskiest strategies only went up to about 400 which means that the vast majority of players were still getting at worst 350 jades and at best around 1100 jades. This kind of makes sense since most of the luckier gambling players likely won first prize once but couldn’t win it more times, landing them into the ~1000 jades earned bracket. Converting to rolls, most players will then earn about 2 to 6 rolls from this event no matter what they do. There is not much of a difference between the min and max of this range to be honest.

Still, if you are someone who wants to make the most out of Cosmic Lucky Prize, you’ll benefit the most if you commit to either being all or nothing. If you missed even a few days and started gambling late, or if you started out gambling but quit later, you end up with a higher chance of receiving fewer jades in the end than if you had just sat out all 7 days instead. Perhaps this is best illustrated in the famous words of a certain senior manager:

“Investing in victory means playing the long game!”

Dining Philosophers

Sat, 07 Dec 2024 00:00:00 +0000

I started watching Pantheon a few days ago. In the show, near the end of the first episode, we see a scene with one of the characters, a teenage loner named Caspian, at the dinner table with his parents. Caspian’s father begins to chide his son for learning differential calculus, an impractical subject he claims. He then poses the following problem to Caspian:

Suppose n geniuses are sitting at a table with n plates and n chopsticks. Each genius has a plate in front of them and a chopstick to their left and right. There is an infinite supply of stir-fry that is constantly being shoveled onto everyone’s plates.

To eat the stir-fry, a genius needs to be able to pick up the two chopsticks to their left and right. However, because there are not enough chopsticks, not every genius can eat at the same time. In the worst case scenario, you end up with deadlock where every genius attempts to pick up the chopstick to their left first and waits forever for the genius on their right to put down their chopstick.

Find a way for these geniuses to eat their stir-fry while avoiding deadlock.

This problem is a spin on a popular problem in concurrent programming known as the Dining Philosophers problem. Like any good problem, it has multiple solutions.

Caspian’s father proposes the following one.

Solution 1: Resource hierarchy

Get the geniuses to agree on some ground rules:

All chopsticks will be ordered and given a number.
When picking up a chopstick, a genius must always pick up the lower numbered chopstick first, then the higher ordered one.

Assuming the geniuses follow these rules, in the worst case scenario, all geniuses first pick up their lowest number chopstick at the same time.

This leaves only the highest-numbered chopstick left on the table. One of the geniuses adjacent to that chopstick will then pick it up, eat, and put down both of their chopsticks.

After this, the other geniuses who are waiting on the first genius’s chopsticks can pick them up and eat their stir-fry. Deadlock won’t happen because if the worst-case scenario ever comes up, there will always be a way to break out of it thanks to the ground rules.

As Caspian’s father explains this, Caspian suddenly cuts him off, saying that the resource hierarchy solution is impractical (a retort echoing to the way their conversation first started).

In particular, it doesn’t scale well as more resources (i.e. chopsticks) get added. Suppose instead of 2 chopsticks, a genius needs to pick up m chopsticks in order to eat, and instead of only sharing one chopstick on either side, the geniuses share a pool of chopsticks. A given genius would spend a small eternity slowly picking up chopsticks until they get the m they need.

Caspian then proposes a second, more “practical” solution.

Solution 2: Third-party arbitrator

Introduce a third party, like a waiter, who controls the geniuses.

Any given genius must have the waiter’s permission to pick up their chopsticks. When a genius wants to eat, they signal to the waiter. If no one is currently eating, the waiter gives the genius permission to pick up their chopsticks and becomes closed off to any new requests from other geniuses.

Once the genius is done eating and puts down their chopsticks, they tell the waiter they are done, and the waiter becomes open to granting permission for new requests from any genius again.

The introduction of the waiter effectively serializes the geniuses who were acting in parallel before. What was previously a chaotic mess of geniuses fighting over chopsticks now becomes an orderly sequence of turn-taking moderated by the waiter.

In concurrent programming, this third-party is known as a lock or mutex. A mutex acts similarly to the waiter from the Dining Philosophers: it represents some sort of permission that a concurrent program must obtain before executing parts of its logic. If a program hits a chunk of its logic that requires the mutex, but the program hasn’t acquired it, it won’t be able to continue until it can acquire that mutex. Naturally, mutexes are useful when you have logic that shouldn’t run in parallel, like file writes or mutating actions in a database.

Conclusion

After Caspian finishes, Caspian’s father, clearly pissed, reprimands his wife, calling stir-fry a sorry excuse for a home-cooked meal and storms out of the house. For Caspian, this is just another regular occurrence in his dysfunctional home. However, the viewers get to follow Caspian’s father out to the garage where we find out that that whole abusive husband persona was in fact just an act and that his father and mother are colluding on something under their son’s nose! It turns out that there’s more than meets the eye with Caspian’s family.

Just like how his family’s outward appearance is a facade to Caspian, the Dining Philosophers problem is also, in a sense, a facade to us. It is a theoretical problem, a toy example hiding the oft uglier and grungy programs written to solve real-world problems. These are programs that must juggle different processes all trying to write to memory at once; programs that must handle access to a database for multiple clients. They must handle a variety of edge cases, all while squeezing out as much efficiency from the machine without crashing, a far cry from our philosophers having their quaint little dinner. After all, multi-threading is feared by many for good reason.

Appendix A: Thoughts on Pantheon

Pantheon has been a very interesting ride so far and perhaps deserves its own separate blog post one day. In short, it’s an animated sci-fi show centered around and exploring the implications of digitizing and emulating the human brain as a software program.

I think what has especially resonated with me about it is the show’s technical subject (a software developer loves a show about software who would’ve guessed) and the clear inspirations it takes from late 90s and early 2000s anime (think Serial Experiments Lain, Neon Genesis Evangelion, Paranoia Agent, Ghost in the Shell). The anime influences are especially telling from its liberal references to these sources of inspiration. One only needs to look at Maddie’s laptop stickers to see parodies of the NERV logo from NGE and Maromi from Paranoia Agent.

One character even blatantly re-enacts the famous cyborg typing scene from Ghost in the Shell later on in season 1! (so cool!)

Despite this, I don’t think Pantheon lets its predecessors completely define it either. The execution and story feel like a fresh take on the ideas and atmosphere pioneered by its forebearers, remixing them rather than just creating a weak rehash. The only criticisms I have is that I felt the show kinda did a genre swap from a psychological horror to a more general thriller. It also gets a bit difficult to manage near the end of season 1 with the viewer having to juggle several, often changing, character motivations. The voice acting can also feel a bit wooden at times, especially in earlier episodes. Of course, these are minor nits. I still love this show.

Pantheon itself is based off of a series of short stories. I really want to read them, especially after I found out that they were authored by Ken Liu, the English translator for my favorite sci-fi novel of all time, Death’s End. However, at least one person online has said the original stories are quite dry and may not be as interesting as the show. This I can believe after attempting to drag myself through more “sciencey” sci-fi, such as Permutation City or Red Mars, and finding myself losing focus.

One thing I do wish is that I had gotten into the show earlier. I had never heard of it until this month despite it being nearly 2 years old at the time of writing. I think one of the main contributing factors to this is its distribution. It was apparently locked behind AMC+ and HIDIVE, only making it to Netflix this year, so no one I knew was talking about it. Season 2 is also still only available in Australia and New Zealand which is both extremely odd and kind of a shame. I think the show is amazing and doesn’t deserve to die in obscurity. Hopefully the Netflix release will prevent that.

Appendix B: Thoughts on Ken Liu and others

Ken Liu is an interesting person. He started his career as a Microsoft engineer. Then he became a lawyer. Then he became a writer, and quite an accomplished one at that, having written for Star Wars and Love Death + Robots (and of course translating Cixin Liu’s Death’s End, the final work of his Three Body Problem trilogy).

Even more surprising is that Ken Liu as someone who made it in both tech and literature circles is not unique either. Andy Weir of Martian fame is another programmer who later became a sci-fi writer. I also recently learned of another person in the tech space writing sci-fi, Ted Chiang, whose Tower of Babylon I’ve been wanting to read.

The upshot is not really to fanboy about random famous people but more so to say that it never ceases to amaze me knowing how many people out there not only have multifaceted, cross-discipline talents but are also actively applying them to accomplish their dreams.

Ruining a Professor Layton Puzzle with Linear Algebra

Thu, 07 Nov 2024 00:00:00 +0000

Capcom recently re-released most of the Ace Attorney series on Switch which made me want to play those games again. I ended up digging up my copy of Professor Layton vs Phoenix Wright: Ace Attorney and brought it along for a plane ride a few weeks ago where I encountered a particular puzzle that could be solved using mathemagics.

The puzzle in question was Puzzle #36: Decipher the Door and appears in the game as a challenge that Layton and Luke must solve before they can proceed to meeting with the Storyteller of Labyinthia.

The puzzle itself involves opening a door by solving a combo lock. The number slots range from 1 to 6 and can only be changed via buttons that modify multiple slots at once.

Ah ha, a classic linear algebra problem.

Converting both the combination lock and the buttons into vectors, the problem reduces down to needing to find the right linear combination that will produce the vector representing the target state for the combination lock.

In other words, finding $\vec{x}$ where:

\[\begin{bmatrix} 0 \\ 1 \\ 1 \\ 0 \\ \end{bmatrix} x_1 + \begin{bmatrix} 1 \\ -1 \\ 1 \\ -1 \\ \end{bmatrix} x_2 + \begin{bmatrix} -2 \\ 0 \\ 0 \\ 2 \\ \end{bmatrix} x_3 + \begin{bmatrix} 0 \\ 0 \\ 1 \\ 3 \\ \end{bmatrix} x_4 + \begin{bmatrix} -4 \\ -3 \\ -2 \\ -1 \\ \end{bmatrix} x_5 + \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \\ \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \\ 3 \\ 3 \\ \end{bmatrix}\]

Doing some algebra and converting everything to a matrix, this becomes:

\[\begin{bmatrix} 0 & 1 & -2 & 0 & -4 \\ 1 & -1 & 0 & 0 & -3 \\ 1 & 1 & 0 & 1 & -2 \\ 0 & -1 & 2 & 3 & -1 \\ \end{bmatrix} \vec{x} = \begin{bmatrix} 2 \\ 1 \\ 0 \\ -1 \\ \end{bmatrix}\]

Which means the matrix we need to row-reduce is:

\[\begin{bmatrix} 0 & 1 & -2 & 0 & -4 & 2 \\ 1 & -1 & 0 & 0 & -3 & 1 \\ 1 & 1 & 0 & 1 & -2 & 0 \\ 0 & -1 & 2 & 3 & -1 & -1 \\ \end{bmatrix}\]

I ended up doing the row-reduction in Python which yielded this:

import numpy as np
import sympy

m = np.array([
	[ 0,  1, -2,  0, -4,  2],
	[ 1, -1,  0,  0, -3,  1],
	[ 1,  1,  0,  1, -2,  0],
	[ 0, -1,  2,  3, -1, -1],
])

print(sympy.Matrix(m).rref())

# (Matrix([
# [1, 0, 0, 0, -5/3,  1/3],
# [0, 1, 0, 0,  4/3, -2/3],
# [0, 0, 1, 0,  8/3, -4/3],
# [0, 0, 0, 1, -5/3,  1/3]]), (0, 1, 2, 3))

Or in other words:

\[\begin{bmatrix} 1 & 0 & 0 & 0 & -5/3 \\ 0 & 1 & 0 & 0 & 4/3 \\ 0 & 0 & 1 & 0 & 8/3 \\ 0 & 0 & 0 & 1 & -5/3 \\ \end{bmatrix} \vec{x} = \begin{bmatrix} 1/3 \\ -2/3 \\ -4/3 \\ 1/3 \\ \end{bmatrix}\]

Cool. Now we have a problem: this matrix is not square.

What this means is that we have a free variable, $x_5$. We need to make a guess for it. Obviously we can’t hit a button a fraction of a time so we also need to adjust $x_5$ such that all the elements of $\vec{x}$ are whole numbers. Thankfully, $x_5 = 1$ happens to just work out.

This gives us:

\[\begin{align} & x_1 = 2 \\ & x_2 = -2 \\ & x_3 = -4 \\ & x_4 = 2 \\ & x_5 = 1 \\ \end{align}\]

This is where we encounter our second problem: we also can’t hit a button a negative number of times.

In the puzzle, the combo lock numbers range from 1 to 6. Once a number goes beyond 6, it loops back to 1. What this means is that we’re actually working in base 6. We’ll have to convert the negative numbers to base 6 which can be done by just adding 6 to them until they’re in the range ¹.

This gives us a final answer of:

\[\begin{align} & x_1 = 2 \\ & x_2 = -2 + 6 = 4 \\ & x_3 = -4 + 6 = 2 \\ & x_4 = 2 \\ & x_5 = 1 \\ \end{align}\]

With vector in tow, I tapped the buttons one by one accordingly and voila, puzzle solved!

Overkill? Definitely. Fun? Debatable. Efficient? Not really (the best solution is apparently: [0, 2, 1, 0, 1]).

Long story short, I did not, in fact, perform Gaussian elimination by hand 40,000 feet in the sky next to a crying baby. I actually just ended up pressing buttons randomly until the right combination lined up :P

Why does this work? Don’t ask me to prove it, but intuitively, it’s similar to moving the hands on an analog clock. The hours of a clock are in base 12. If the hour hand is at 12, moving it clockwise by 3 hours puts it at 3. The same thing can be achieved by moving it counter-clockwise by 9 hours. So 3 is the same as -9 in base 12. What we’re doing in the puzzle is similar to this² except the clock only has 6 hours and we’re moving the hour hands of multiple clocks all at once. ↩
This is a lie. We’re actually applying this clock intuition on the number of times we do an action, not the actual number in the slot. However, this is still okay since our entire equation is linear which means mod can be applied early since $(a(x \mod m)+b) \mod m = (ax+b) \mod m$. Don’t ask me to prove this one either. ↩

A Follow-Up to the Magic of --follow

Wed, 11 Sep 2024 00:00:00 +0000

Nearly half a year ago, I wrote a mini blog post detailing my exploits working with large outputs. I was trying to find a way to see both the output logs of a test and to get a copy of the outputs in a file to share around which eventually led me to a solution using tail -f. Then a few weeks ago, while my friends and I were searching each other up on the Internet, that very blog post was dug up again. This in turn spawned a conversation that went something like this:

“Jason, have you ever heard of tee?”

“You mean for assembly, yeah?”

“No for shell.”

“Oh. Um. No.”

“What does it do in assembly?”

“It stores a value and loads it at the same time…I think.”

“Imagine that for files.”

Apparently I had been doing things the hard way. Linux comes with a command called tee that does exactly what I was trying to do with tail -f: it displays to stdout while also writing to a file at the same time.

Funny enough, it was at this time that I found myself also having to go back and work with some tests again. I tried out my newfound knowledge this time:

./run-tests.sh | tee ~/test-results.txt

…And to no one’s surprise it worked like a charm! Now I could see both the output and save a copy of it to a file with only one command!

Perhaps, tail -f was a bad solution and the blog post from before is just junk now. However, an optimist would probably say that this in fact shows growth and that there are many approaches to solving a problem while a pragmatist would probably wonder why I spent so many words explaining what could have simply been an amendment to the original blog post. Then the super duper shell guru would show up and tell us all that there’s an even better way to do this.

Anyways, until next time!

Status 599

Thu, 15 Aug 2024 00:00:00 +0000

Recently, I’ve been seeing ads on the Metro targeting the military-industrial complex. While I didn’t find them terribly interesting per se ¹, they did do their job and somehow still ended up catching my attention.

The Ad

Each ad prominently features an HTTP error message plastered on a backdrop of military personnel along with some rhetoric for the viewer to buy the product/service advertised or else face implied dire technical consequences.

I’ve only seen two overall variants so far. The first one features the standard Status 404 that everyone knows and loves ².

The second one, however, features an error code I have never seen before: Status 599.

What in the world is Status 599?

Despite working with web APIs every day, this was the first time I had ever heard of a 599 error code. What exactly is it? And why did the makers of this ad choose that code specifically?

For starters, Status 599 seems to be non-standard. It does not appear to be documented in any RFC and neither Rust nor Go include it as a constant in their list of status codes.

A cursory Google search indicates that Status 599 is supposedly shown when a request made to a proxy gets timed out by the upstream service that the proxy is calling. However, the truth of this is difficult to verify as several sources making this claim this seem to circularly refer to each other.

Becoming ever more curious, I dug a bit deeper and tried to find exactly what software products and what kind of situations people encountered Status 599 in. One source says that Status 599 has been seen among Microsoft HTTP proxies but did not mention any specific products. Searches across forums also show this error showing up in a few other products including SAP Crystal Reports, Cisco CUCM, Shadow PC, Proxmox, and OwnCloud. A few of these do seem to corroborate the proxy timeout definition, relating to connectivity issues with dependencies. Others, not so much. OwnCloud appears to respond with Status 599 when something has gone so horribly wrong that the server can’t even log the error that got thrown. Cisco CUCM’s usage is even more odd, showing Status 599 when the AXL version used by the client is unsupported, a client error that doesn’t seem even remotely related to timeouts or 5xx server errors at all.

What exactly Status 599 means remains ambiguous, and its varied usage across actual software products doesn’t help clear much of that fog. Status 599 still seems like an enigma to me, and I feel like that it is precisely the reason why it was chosen for this ad. Just imagine paying for a piece of software, getting an error message that is effectively gibberish while using it, spending week after week digging through forums and customer support to try to figure out what is wrong, and finally turning up with absolutely nothing. That is extremely frustrating, and perhaps it is that very frustration that this ad hopes to conjure up, rudely resurfacing dreadful memories buried deep in the minds of random IT folks during their otherwise uneventful commutes, all to get their attention.

Appendix: Proxy vs Reverse Proxy

Other than sounding like an epic battle of grand proportions concocted and debated by only the biggest and baddest of nerds, proxy vs reverse proxy is a question I often find myself asking and answering again and again (mostly due to my poor memory).

In short, both are man-in-the-middle components used for forwarding requests. The main difference is that a proxy forwards requests from a private intranet out to the Internet while a reverse proxy forwards requests from the Internet to a private intranet.

A proxy makes sense when you have callers sending requests out into the Internet. For example, let’s say you are a web administrator for a public school. You want to prevent kids from watching anime on their school laptops. You could install a content filter on the laptops that intercepts every outbound HTTP request, checks the origin, and blocks the request if it matches a blocklisted origin. In this case, the content filter is a proxy. It controls which requests from the student can make it out into the external Internet.

On the other hand, a reverse proxy makes sense when you have services that receive requests within your intranet. For example, if you are hosting a server for a web API, you most likely want to have something like NGINX only route requests for certain paths to be handled by your application. In this case, NGINX is a reverse proxy. It controls which requests from the external Internet can make it into your application.

…or fear. I once encountered an ad featuring a business woman that was placed and oriented in just the right way so that she would make eye-contact with passengers sitting down. ↩
Writing this, I was reminded of The Best 404 Page Ever where I spent much of my after school life before Flash got murdered. It apparently lives on here thanks to Ruffle. ↩

The Magic of --follow

Wed, 13 Mar 2024 00:00:00 +0000

I recently discovered the joy of using the --follow flag. Let me tell you a little bit more about what I mean.

One of the tools at my work involves running a test script that both (1) takes a long time to run and (2) generates a lot of output.

When I first started working with it, I naively ran the script as-is:

./run-tests.sh

I came back 1 hour later to find that it had printed a huge mess of a test report to stdout. The report had so many lines to the point that the oldest ones had already been truncated by my terminal. This was not very useful to me nor my senior who was asking me where the hell the test results were.

On my second try, I tried a slightly smarter approach. I redirected the script’s output to a file instead of stdout this time:

./run-tests.sh > ~/test-results.txt

Now the script would write the test results to a file for safe-keeping. I re-ran it and left for another hour. When I came back, it was a simple matter of grepping the TXT file for any failed test cases and transferring the file off of my remote desktop to send to my team:

grep -i fail ~/test-results.txt

scp me@host.example.com:/path/to/test-results.txt ~/test-results.txt

This worked out, but now I had a new problem: Unlike in the naive approach, I couldn’t see the script output in real-time anymore. This made it difficult to figure out what was actually happening over the course of that 1 hour. To get the output, I could repeatedly tail the results file but that gets kind of annoying after a while. That’s when I discovered the magic of --follow. Here’s what I did:

I re-ran the original command in my terminal:

./run-tests.sh > ~/test-results.txt

And in another terminal, I ran:

tail -f ~/test-results.txt

The -f flag is short-hand for --follow which continuously updates the output with the latest lines at the end of a file. Now I had the best of both worlds: I could monitor the progress of the test script in real-time and have those results saved to a file that I could analyze in post.

Perhaps something this simple doesn’t merit a full-blown blog post, but I still feel proud of my little discovery, no matter how much it pales in comparison to the arcane commands of the shell gurus. Also, I haven’t written a post in over half a year, so now is as good a time as any to start again. Until next time!

The Microsoft OAuth2 SPA Experience

Sat, 28 Oct 2023 00:00:00 +0000

Boo! It’s October, so I’ve decided to rise from the dead with a new post. I’ve recently had the pleasure of messing around with the Microsoft Identity platform and would like to recount my experience here.

The Microsoft Identity platform is a service that Microsoft uses to handle authorization into several of its web APIs, most notably the Graph API which gives developers programmatic access to various Microsoft cloud services, such as Outlook, Teams, OneDrive, etc. Of course, the platform uses everyone’s favorite authorization standard, OAuth2. Microsoft supports a rarer application of OAuth2 for entirely client-side applications which makes this a bit more interesting to write about.

Creating an OAuth2 application

The first thing that needs to be done for any integration into an OAuth2-authorized API is to create an OAuth2 app on the resource vendor’s site. For Microsoft, this setup happens to live on Azure. I made an Azure account and after wandering through a forest of UI panels and widgets, I finally found myself on the Applications page.

When creating an OAuth2 app, you are given several options for which type of app to create. Since I was working on a React app in Vite, I chose SPA. I also provided a redirect URI for Microsoft to send users to at the end of the OAuth2 handshake.

Typically most OAuth2 apps also come with a client secret. This is usually used by the client app to prove to the resource server that the client is legitimate when it redeems an access token. However, we don’t need one for the SPA route (since I guess the web UI a user is using can just be thought of as an extension of that user), so this was skipped. That being said, Microsoft’s client secret generation is a bit different from most OAuth2 apps I’ve seen before in that it allows devs to (1) have multiple secrets on one OAuth2 app (yay?) and (2) it also forces those secrets to expire which means that they must be manually rotated every so often (aww…).

Authorization in a SPA

With my shiny new OAuth2 app, it was time to get building. The SPA workflow on Microsoft is slightly different from 3-legged OAuth2 which most vendors implement. The general flow is still:

Authorize the user and get a code
Redeem that code for an access token

However, since this all takes place in a web browser, there are only really two agents involved in this flow, the resource server and the user, rather than the 3 involved in 3-legged OAuth2 which would also include a server from the 3rd party client that owns the OAuth2 app. This simplifies some things (no need for a client secret since no client service), but it also makes things more complicated (so how do you prove that the redemption request is legitimate without a client secret then?). This leads us to one of the more unique (and infuriating) parts of this experience, PKCE.

PKCE, or RFC 7636, is a way to prevent malicious clients from using stolen OAuth2 codes. Consider this scenario: Eve, an attacker, is able to see all incoming HTTP requests for Alice, a normal user, (either through malware installed on the Alice’s device or by sniffing network requests). Alice decides she wants to use a legitimate client application and authorizes against a resource server through that client’s OAuth2 app. The authorization is successful, and the resource server responds to Alice with a code. However, since Eve is watching all HTTP responses for Alice, Eve sniffs the code from the query params of the URL for that response. Eve then attempts to redeem the stolen code through her own OAuth2 app. The resource server receives the redemption request from Eve’s malicious app, and since the resource server thinks it’s a legitimate request, it responds to Eve with an access token. Now Eve can use this token to access all of Alice’s data! 😭

This is very bad. The problem is that we need some way to allow the resource server to prove that the agent who originally generated the request for the code is also the same person who ends up redeeming it for an access token. To do this, PKCE asks the user to first generate a high-entropy secret (read-as: unguessable UUID). This secret is hashed and encoded before being sent as part of the authorization request to the resource server who keeps it on file. When a client later tries to redeem the code for an access token, the client needs to provide the original secret in the redemption request. The resource server receives the redemption request, and does the same hashing procedure on the secret. If the hash from the redemption request matches the original hash from the authorization request, then the user has successfully proven themselves as the original requester and an access token is sent back to them. If the hash is mismatched or not provided at all, an error is sent back instead.

Once I digested all of this, it was time to put my new-found knowledge to practice. The code challenge scheme used by PKCE is SHA-256, so I found a way to generate a cryptographically-secure UUID, hash it using SHA-256, and encode it in base64:

const codeVerifier = crypto.randomUUID();
const encoder = new TextEncoder();
const hash = await crypto.subtle.digest("SHA-256", encoder.encode(codeVerifier));
const codeChallenge = btoa(String.fromCharCode(...new Uint8Array(hash)));

However, sending this to the authorization server resulted in an error:

AADSTS501491: Invalid size of Code_Challenge parameter.

Thanks, Microsoft. A quick search and I found a few others who ran into the same issue before. Apparently, base64 pads encodings to multiples of 4, so a SHA-256 hash would end up with a 44 character-long encoding. However, Microsoft doesn’t expect the challenge to be padded and only accepts encodings with character length 43. This meant that I needed to manually trim off the extra padding in the base64 output. I also found out that base64 encoding as-is is not URL-safe, so I needed to replace some symbols as well. This is the final code I ended up with:

const codeVerifier = crypto.randomUUID();
const encoder = new TextEncoder();
const hash = await crypto.subtle.digest("SHA-256", encoder.encode(codeVerifier));
// Remove trailing ='s and make b64 url safe
const codeChallenge = btoa(String.fromCharCode(...new Uint8Array(hash)))
  .replace(/=/g, "")
  .replace(/\+/g, "-")
  .replace(/\//g, "_");

Whew! After all that, I finally got my app to start redirecting with the code. Now pausing for some questions.

Yeah, you kid in the back.

Uh-huh.

Ok so the question is: “Why do this stuff with redeeming codes if the resource server could theoretically just give you an access token instead of a code at the authorization step?”

This is actually one of the options for authorization from a SPA on Microsoft and is known as an implicit grant. Essentially, instead of the resource server sending the user to the redirect URI with a code, they just send the user to the redirect URI with the access token straight up after authorization. While this is a simpler method for authorizing in SPAs, implicit grants are considered insecure since the token is available in the response URL (which leads to risk of getting sniffed as we saw before in the PKCE section), and implicit grants also don’t vend out refresh tokens. The OAuth2 spec also puts the access token for implicit grants in a URL fragment identifier for some reason which kinda scared me.

Fun with CORS

So far, I felt I was doing pretty well on authorization after PKCE. Then I tried to redeem the codes in the SPA and got this:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://login.microsoftonline.com/common/oauth2/v2.0/token. (Reason: CORS header 'Access-Control-Allow-Origin' missing

Okay…I had seen this type of error before. It’s a CORS error that usually happens to prevent the web browser from calling out to any arbitrary server. But Microsoft had advertised this method as working for SPAs which run in web browsers, so we really shouldn’t be getting these, right? What’s the deal?

Unfulfilled, I set off on a long journey of discovery (read-as: I started searching the error up on the Internet) and found that many users had also run into the same issue. However, none of their solutions worked for me. As a sanity check, I even tried redeeming the code outside of a browser by sending the redemption request in curl and watched it drop an access token in my lap. Something else was up.

Throughout this journey, I had also come to learn of the Microsoft Authentication Library for Javascript, or MSAL.js. Apparently, this is an SDK developed by Microsoft for devs integrating with their identity platform. It streamlines a bunch of complications with orchestrating the OAuth2 handshake (like opening up login popups and constructing URLs). I had initially ignored it, refusing to build on top of another layer of abstraction, but another hour passed, and I caved, integrating my app with MSAL just to see what their secret sauce was.

I walked myself through the login popup, the authorization, the code, and then…the access token?!

What? It worked?! How was MSAL doing it?

I opened the network tab in my browser, and sure enough, the token redemption request was going through and it was using…a form data body? Wait. Microsoft’s documentation had this POST request written with a URL-encoded form which is what I had been using as well. I switched my form construction code over and then there it was:

The access token.

Oh god. The service is conditionally setting an allow origin header depending on the format of the POST body ¹. I screamed and then moved on.

At this point, I had a working integration with Microsoft Identity in a SPA. I could do anything now.

And so…

I chose to stop. I had initially envisioned making a one-stop shop for calling into a bunch of different SaaS APIs (which would’ve solved a pain point from my day job), but then I discovered the Graph Explorer, a website miles beyond whatever I had finally managed to scrap together over those past few days, and realized I had a long road ahead of me. I was already tired from debugging OAuth2 and wanted to do something else, so I set this one down.

Thus the story stops here for now. Maybe one day I’ll pick this project back up again, but I wanted to rest at this point. Perhaps I just needed a spa day (a real one, not one of those cursed ones where you lock up 5 devs in a room with food and water, shake the room for 3 days, and hope a functioning SPA pops out at the end).

It looks like this has been fixed since then so now both form data and URL-encoded forms can be used for token redemption from a SPA. Either that or I had a fever dream hallucination sometime in early October. ↩

Fixing a D-Force Pad

Sun, 19 Feb 2023 00:00:00 +0000

This past year, I’ve been trying to get into DDR. One thing led to another, however, and instead of getting good at dancing, I ended up dissecting my soft pad instead.

I first got into the game since it seemed like the next logical jump for me. I played a lot of Bang Dream and osu! in the past, so I felt like DDR would be another rhythm game I could easily pick up. Plus I don’t get enough exercise, so I was also planning to use this as a motivator to move around more.

The easiest way to play DDR on a PC is through StepMania, an open-source DDR clone. I downloaded it and a few song packs, and once I had gotten the hang of the game on the keyboard, I felt it was time to move to a real dance pad. Scouring the Internet, I ended up deciding on a D-Force soft pad for reasons I don’t particularly remember. After getting the pad and some head-banging with the controller settings (I didn’t realize I had to download Joy2Key to map the controller to keyboard inputs), I eventually got it connected to StepMania and was up and grooving away.

Things Fall Apart

That lasted about a month. At some point, the pad’s right button stopped responding consistently. At first, I chalked it up to my foot just missing the button (I found out I’m actually pretty bad at DDR once I started using the pad). However, after playing more maps, it slowly dawned on me that the pad was actually broken.

“Okay, maybe this one was just defective.”

I returned the pad and got it replaced. However, the next one only lasted for two weeks before the down button ran into the exact same problem. Frustrated, I rolled up my pad up into a corner and ended up quitting the game for nearly a month.

Open Pad Surgery

Eventually MLK day rolled around. Desiring the dance floor once again, I dusted off the pad and began taking it apart. My plan was simple: the D-Force pad has a matrix of 9 buttons. I’ve mapped 4 to hit arrows in the game and 1 to confirm selections. This left the remaining 4 buttons unused. If I could re-wire one of these unused buttons to the controller’s down button input, then everything would be solved!

Opening up the pad, I found two plastic sheets coated with some sort of conductive paint separated by a layer of foam. The foam layer has a bunch of holes where each button is. It seems that the pad works by getting the circuits of the two layers close enough to each other in order for electricity to flow between the two sides. When I flipped the pad around, I noticed that the plastic sheet on the bottom had a crease in the bus line leading to the down button. Suspecting that this to be the cause of my woes, I grabbed some scissors and scotch tape and began the operation.

The plan was to cut out both the down button portion and the unused bottom right button portion of the plastic sheet. Then I was going to install the bottom right button as the new down button with the help of some tape and extra conductive components that I cut out.

This is what it ended up looking like, all said and done (buttons are flipped horizontally since this is the bottom sheet).

I stacked all the layers up and stapled my frankenstein pad together. Loading up a song in StepMania, I apprehensively started to play, and to my surprise, it seemed to be working.

The Relapse

Unfortunately, this time of love and joy did not last very long. Later in the night, as I was playing a longer session, the down button became unresponsive again. I removed the staples and opened up the pad once again to a new mystery.

However, this time, I intended to use a more scientific method in my investigation instead of just flat intuition. I busted out my multimeter and started to measure parts of the circuit. Multimeters have a mode called “diode test” which is really useful for seeing if there is an electronic connection between two points or not. The multimeter displays a voltage drop if there is a connection, otherwise it just displays a “1” which means that the circuit is overloaded. When I pointed the terminals at the down button and its bus line, I got a 1. It seemed like my first fix did not last very long (whoops). Just as a sanity check, I also tested the connection between the down button and other buttons. Since they wouldn’t form a circuit with the multimeter, I got a 1 as expected.

In an attempt to fix the connection, I ended up cannibalizing a larger piece of the leftover down button and attaching it higher in the down button’s bus line.

Measuring again, the voltage drop was pretty bad, but at least there’s some connection now.

I slapped the button a few times with Joy2Key running, and it seemed responsive again, so I sandwiched the sheets back together again and returned to stomping on them for the rest of the night.

Closing Thoughts

Since then, the D-Force pad has been working quite smoothly. I suspect it will break again in the future, but at least I know how to fix it now. I still wish I had a more reliable dance pad, but basically the next upgrade I could get is a $400 hard pad from Poland. I’m not that invested into the game yet, so I figure I’ll make do with my current setup for now.

Overall, this was an interesting experience. I don’t know much about hardware, so even learning a small thing like how to use a multimeter in a practical situation feels like a big step up to me. That being said, the outcome was definitely more satisfying than the journey on this one. After all, I’ve finally fixed my pad! Now there is nothing to do but dance (or fail miserably trying to do so)!

A Eulogy for Heroku Free Tier

Sun, 27 Nov 2022 00:00:00 +0000

We are at the end of November and with it, the final hour of all of my Heroku free dynos. When I first heard that Heroku was slashing their free tier back in August, I remember feeling a bit of a bittersweet nostalgia. Like many other folks in software, Heroku had been my gateway into doing more web development.

I first found out about Heroku in high school. I was a junior stuck in an elective course called computer graphics which covered just about everything except its namesake. One of the class’s modules was on web development and briefly mentioned something called Ruby on Rails, a funny name that piqued my interest at the time. One thing led to another, and I soon found myself working through a Rails hello world tutorial after school. At the very end, the instructions described deploying the site to this thing called Heroku.

“What in tarnation is a Heroku?”

I went to the website, made an account, and cautiously followed each step from the tutorial. Suddenly, the terminal spat out a bunch of colorful logs, and I was told that the site was deployed on something or another dot herokuapp dot com. I pointed Firefox to the enigmatic URL, waited a bit, and then there it was: my shifty beginner project, on the Internet. My eyes glimmered and the gears in my head started turning. I remember immediately adapting the tutorial project into PhamBash, a clone of Bash but with a uniquely inane focus on memorializing quotes from my high school’s eccentric chemistry teacher. Right as it deployed, I tweeted the link at him from an anonymous account. The next day in class, he had the site displayed on the projector, shown up at the front of the room for all to see. He surveyed the 15 students present in the room before stopping to look at me, the kid seated at the middle table, grinning sheepishly.

“You make this?”

“Yeah.”

Awkward silence. Then a smile.

“Okay guy let’s start class.”

After playing around in Rails, I quickly found out about Django and soon became inspired to make more apps. During this time, Heroku would become my testing ground for all things web.

I remember my next project which was a Facebook Messenger bot that responded to messages by babbling using a Markov chain generated from text written by yours truly. I had foolishly chosen to use Django to build the bot, overkill for a server that was basically just responding to webhook requests. Eventually, I did manage to get it all working locally (with ngrok tunneling so FB could hit my server).

Then I came to the part where I had to deploy it. I remember having some trouble getting the build to work on Heroku and being generally mystified by the Heroku Django guide introducing yet another server, gunicorn (which was also a blackbox for me since it does not run on Windows). I did figure it all out eventually though: I had been using the wrong buildpack (whoops).

Later, when I started learning about Docker, Heroku was where I went to experiment with it. My first Docker-based dyno was a small site to commemorate birthdays for anime characters by letting users submit doodles on their birthdays. While the site itself had a short life and was nothing terribly grand (especially given the amount of phallic creations my friends submitted to it), building it got me more used to working with Docker. I learned a lot of new things, like multi-stage Docker builds and the pain of watching Docker get into fistfights with VMWare before WSL2 existed.

Of course, Heroku’s free tier was not without many grievances. Cold start always annoyed me to no end, and many a time I would find myself on the dashboard with my trigger finger on the delete button, mulling over which one of my 5 dynos to sack so I could deploy my next project.

Despite these limitations, free users still always find their way around. For example, one trick for cold start was to just never let the app fall asleep. I remember I had trained a ResNet-18 classifier to tell the difference between pictures of Shamiko and Momo from Machikado Mazoku. I built a web application around it and wanted to show off the app to others but knew no one would be patient enough to sit and wait through the cold start. To combat this, I subscribed the endpoint to Kaffeine, a service that periodically pings Heroku endpoints to keep them awake. After that, my app never fell asleep, and the only hassle I got was getting a monthly email from Heroku, telling me that my free dyno hours were running out at the end of the month.

As I write this, I remember the free dynos I still have running in my account. One of them is Stapler-kun, my faithful Discord bot, born out of fatigue from a friend who would always declare social plans in our Discord server before following up with the message “someone pin that”. I used to pin them for him. Now the bot does it for me.

I had hosted Stapler-kun as a worker on Heroku with a free Postgres database to persist which messages had been pinned. I remember getting giddy when I first got it working. A personal little robot to do my bidding. Built by me. All hosted for free.

Now Stapler has a new home. Or perhaps, I should say, a new body. I ended up building out Stapler-kun v2, switching it from a discord.py bot to a Discord Interactions app running on a free Cloudflare worker. I had a lot of fun with the port and even got the motivation to add more functionality to the bot while rebuilding it, yet I still feel uneasy. From one free tier to another. What happens when this all too, ceases to be?

Gripes aside though, I had a lot of fun with Heroku’s free tier throughout these years. It gave me a risk-free chance to practice web dev, and it fuelled many of my good, bad, and plain crazy ideas. Now that the hour is nigh, I suppose it is time to put a close to reminiscing and for the Heroku free tier to depart once and for all. To all the things I learned with you and all the adolescent fun you provided: Thank you and goodbye, Heroku free tier.

Running Elixir on Lambda

Sun, 21 Aug 2022 00:00:00 +0000

A while back, I discovered that Johns Hopkins publishes daily COVID case and death counts for all 50 US states. The graphs reminded me a lot of metric graphs for monitoring services, and I began to imagine a system of mirroring the data into CloudWatch and paging myself whenever they breached a certain threshold.

This idea ended up evolving into CovidPager, and while I never did get as far as paging myself (PagerDuty kinda expects me to be a company), I do want to talk about the most interesting part of the project: the fact that it ended up as an Elixir application running on Lambda!

Run Whatever You Want

The beginning of this journey starts with Lambda and its custom runtime. Although your typical Lambda function would use an officially supported runtime, ever since 2018, Lambda has also begun providing custom runtimes. This is our ticket into running an Elixir Lambda handler.

A handler running in a custom runtime interfaces with Lambda by interacting with the Lambda Runtime API. The runtime API is available as an HTTP service and is quite simple. It’s made up of four operations: get the next invocation, respond to an invocation, report a service initialization error, and report an invocation error.

In order for Elixir to run on Lambda, at minimum, we would need to write an Elixir application with an HTTP client that is able to poll requests from the next invocation endpoint, process the requests, and send responses to the response endpoint. This is a lot of boilerplate work. Thankfully, someone else has done it for us already in the form of the AWS Lambda Elixir Runtime! Building on top of the runtime implementation is as simple as adding :aws_lambda_elixir_runtime as a dependency in our Mix project, writing the Lambda handler, and deploying a custom runtime Lambda function that targets the handler using its identifier.

Run In Wherever You Like

If you are like me and followed the AWS Lambda Elixir Runtime guide by deploying the release artifacts directly to Lambda, there is a high chance you ran into an error related to a dependency for beam.smp. The reason behind this is that Elixir itself depends on the BEAM virtual machine which was compiled for your machine’s environment as part of the prerequisite Erlang installation for Elixir. Your machine is most likely not configured exactly the same as Amazon Linux 2 (AL2), the environment that the custom runtime Lambda function runs in, which will result in incompatibilities. In my case, BEAM needed glibc version 2.29 while Amazon Linux 2 has version 2.26 at the time of writing. The only real solution for this would be to spin up an EC2 instance running AL2, re-download and re-compile Erlang and Elixir on it, and then rebuild my application. I took one look at my wallet, one look at the time, and decided nah.

Normally, this would be where the experiment ends, but back in 2020, Lambda released support for container images. What this means is that our function’s environment does not have to be tied to Amazon Linux anymore. We can just ~~steal~~ acquire a pre-existing Docker image for Elixir and build our application on top of that! Lambda will run our container and any dependency issues between our machine and Amazon Linux disappears since we can just model our environment via Docker instead! For CovidPager, I ended up using an Elixir Alpine image and set up Docker to build and run the application binary.

# Dockerfile

FROM elixir:1.13-alpine

WORKDIR /app

RUN mix do local.hex --force, local.rebar --force

ENV MIX_ENV=prod

COPY mix.exs mix.lock ./
COPY config config
RUN mix deps.get

COPY lib lib
RUN mix release

ADD aws-lambda-rie-x86_64 /usr/local/bin/aws-lambda-rie
COPY ./entry_script.sh /entry_script.sh

ENTRYPOINT [ "/entry_script.sh" ]
CMD [ "Elixir.CovidPager:handler" ]

You may have noticed that the Dockerfile also contains an ENTRYPOINT that points to a shell file called entry_script.sh. This is to support locally testing the container image function. To do this, AWS provides a Runtime Interface Emulate (RIE) which is a stand-in for the real Lambda Runtime Interface web server. We can include the RIE binary in our image and then configure the entrypoint of the image to a program (in this case, entry_script.sh) which will conditionally run the service using RIE or the actual binary depending on whether the Lambda runtime is present. If deployed locally, RIE allows the Lambda function to be triggerable via HTTP POST to the local Docker endpoint.

# entry_script.sh

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
  exec /usr/local/bin/aws-lambda-rie _build/prod/rel/covid_pager/bin/covid_pager start $@
else
  exec _build/prod/rel/covid_pager/bin/covid_pager start $@
fi

Having the ability to do this kind of local testing is powerful since it means that we can test our service as if it were truly deployed instead of only being able to test through unit tests. Unit tests can’t test entire integrations since they usually mock out key dependencies, such as external services, so being able to see a service’s real behavior during local development helps to uncover any sort of integration failure cases as early as possible.

Once I was able to test locally, it was time to deploy for real. I uploaded the image to ECR and spun up a Lambda container image function that would consume the uploaded image. The infrastructure specification ended up being just a single function in CDK.

// deployment-stack.ts

const fn = new lambda.DockerImageFunction(this, 'ECRFunction', {
  functionName: 'CovidPager',
  code: lambda.DockerImageCode.fromImageAsset('/PATH/TO/PROJECT/ROOT'),
  timeout: cdk.Duration.seconds(30),
});

But…Why Run Like This At All?

Question time. Yes, you in the back.

…

Ok um, so the question was: why in the world would anyone want to do this? That’s a great question. I’m not really sure.

One of Elixir’s major selling points is its ability to deal with faults. An Elixir application does this with supervisors that watch child processes. If a child fails, supervisors can take further action based on a supervisor’s strategy. For example, a one-for-one strategy will restart just the failing child if a child fails while a one-for-all strategy will restart all children under the supervisor if a child fails. A more complex application may have a supervisor tree with higher-level supervisors that watch lower-level supervisors that themselves watch the actual worker processes.

The supervisor design works best for long-running applications since it allows for a service to recover from faults by only restarting parts of the application as opposed to the whole in order to keep the overall service up and running. However, this isn’t really a great fit for Lambda which is meant for only short-lived tasks (currently, a Lambda function’s max execution time is 15 minutes).

Moreover, with serverless design, it feels that fault tolerance is intended to be taken care of by the infrastructure rather than by the actual service. Because of this, AWS services end up doing most of the heavy lifting to recover from or avoid faults (ex. ensuring a service is healthy in ECS by automatically restarting unhealthy containers or ensuring requests are processed by a service by fronting them with an SQS queue). All in all, this seems to make Elixir’s fault-tolerant design feel redundant when paired with Lambda.

In my opinion, the greatest benefit of running Elixir on Lambda may simply be access to the Elixir language itself. Choosing Lambda for infrastructure does not mean that Elixir is off the table as a choice for language and vice versa. Moreover, as a functional language, Elixir generally feels like a good fit for writing Lambda functions since both are designed for stateless execution.

Closing Thoughts

The upshot is that running Elixir in Lambda is not only theoretically possible, but actually quite accessible through the AWS Lambda Elixir Runtime and Lambda’s support for containers. While Elixir’s fault-tolerance benefits are lost within a Lambda function, it’s still neat that Lambda can run Elixir, and it opens up the door for Elixir development on Lambda from anyone who wants to try.

That being said, running Elixir in Lambda still poses a risk since it is not officially supported by Lambda. Even though some well-known companies are already using Elixir (Heroku and Discord to name names), the language itself is not mainstream like Python or Java. If Elixir and functional programming as a whole really do reach widespread popularity in the coming years, it would be interesting to see its effect on how developers design services in the future as well as how cloud providers will react to those shifting design paradigms.

Dunning-Kruger

Mon, 18 Apr 2022 00:00:00 +0000

I recently read a blog post arguing that the Dunning-Kruger effect was not a psychological phenomenon but, rather, a statistical artifact.

For some context, the Dunning-Kruger effect is a well-known cognitive bias discovered and discussed by David Dunning and Justin Kruger in a 1999 paper. In short, the idea of the effect is that poor performers tend to overestimate their ability while skilled performers tend to underestimate their ability. The effect seems to have received wide-acceptance over time due to the ease with which their results have been reproduced in other areas. It also kind of just makes sense anecdotally; a lot of us have either been or seen that overconfident person learning a new subject.

On the other hand, the blog post referenced above strongly pushes against the psychological conclusions of Dunning and Kruger’s results, making the argument that the Dunning-Kruger effect is actually a statistical artifact. The general idea from that post is that Dunning and Kruger’s results only arise because we are trying to correlate some random variable, X, with a function of itself, Y-X. In the case of Dunning-Kruger, X were test scores subjects scored, and Y were test scores subjects think they scored. Y-X is the discrepancy between the predicted and actual scores which were observed to have trended downwards as subjects went from overestimating to underestimating as skill increased.

The author argues that because of the way these calculations were designed, we are guaranteed to observe some trend between the two variables we are comparing (X vs Y-X) due to the inherent correlation that exists between them rather than because there is some psychological effect at play (e.g. humans overestimating their abilities). The author then goes on to provide an experiment and some graphs to support his point.

This seemed like a very reasonable explanation, but I still had trouble accepting that the DK effect that many others, including myself, have taken to heart was just the result of some statistics. I wanted to try and recreate the author’s experiment and see it with my own eyes. Plus I wanted to see if I still had it in me to write some R.

An Experiment

We needed to setup a scenario with two independent random variables, X and Y (I ended up just making them standard normal distributions). Dunning and Kruger broke their participants up into quartiles based on their test scores, so I have done the same on the X variable.

library(dplyr)
library(ggplot2)

n <- 1000
x <- rnorm(n, 0, 1)
y <- rnorm(n, 0, 1)
x_quartiles <- cut(x, breaks=4, labels=c(1, 2, 3, 4))

After this, I calculated the averages for X and Y by quartile. R treats the quartiles as factors, so I also needed to convert them to a numeric vector.

df <- data.frame(x, y, x_quartiles)
avgs <- df %>%
  group_by(x_quartiles) %>%
  summarise_each(funs(mean))
avgs$x_quartiles <- as.numeric(avgs$x_quartiles)

Now it’s time to make some graphs.

These are what the data points looked like. X and Y were not correlated, so graphing them just creates a cloud of points. X and Y-X have a negative correlation, so we see a downwards trend with some noise (produced by Y).

After the data points were grouped by quartile, I graphed a boxplot for each quartile. Since X is correlated with itself, the averages in the boxplots display a linear trend moving across quartiles. On the other hand, since X and Y are not correlated at all, the boxplots in each quartile look about the same compared to each other.

Graphing only the averages in each quartile and combining the graphs, we finally see something that is eerily similar to both what the author of the blog post and what Dunning and Kruger produced.

This graph roughly shows the correlation of X and Y to X (since the quartiles are just X split by group). The black line increases over the quartiles because X is effectively being graphed against X. Meanwhile, the red line appears flat since, because Y has no correlation with X. In each X quartile, the distribution of Y values remains roughly the same (quartiles 1 and 4 just represent fewer points since the probability of landing at the tails of a normal distribution is less likely).

A Statistical Artifact

The idea of DK being a statistical phenomenon is not new. The Dunning-Kruger effect Wikipedia page outright states that one of the criticisms for the Dunning-Kruger effect is regression to the mean.

One idea from regression to the mean is that after an initial sampling, performing a second sampling from the most extreme data points will still result in a mean that is close to the mean from the initial sample. For example, suppose I am interested in the distribution of test scores on a randomized true-false exam. If I made 100 students take a randomized true-false test, the expected mean score is 50%. If I then took the top 10 students from the 1st exam and made them take a different randomized true-false test, the expected score on the 2nd exam is still 50%. Since the answers are randomized, even though I took the top performers and made them take another test, the mean from the second testing is not extreme but, rather, still close to the mean from the first testing.

This is roughly what is happening with our experiment. We performed an initial sampling of a distribution (X) and broke the data points into quartiles where the 1st and 4th quartiles could be considered “extreme” data points. We then performed a second sampling on the same distribution (Y) and calculated the averages per group. Due to regression to the mean, the averages of Y look the same across all groups of X, despite some groups being more extreme than others in terms of X. This is because, in each group, the group averages of Y are regressing towards the overall mean of Y.

The real world implication is that, perhaps, the scores people get and the scores people guess they got have a discrepancy, not because of a difference in mental models between poor and skilled performers, but because these two variables are not really related at all. And why should they be related? All we know is that some people are very good at whatever test they took, not that they can make good judgments on how well they did.

So What Does It All Mean?

Do the Dunning-Kruger effect and its psychological conclusions really exist or not? I used to believe so, that it was because most people new to a field were overconfident, but after running this experiment, I’m not so sure. Maybe we all just suck at judgment. Even if a person is good at taking a test, what incentive (other than maybe avoiding hubris) do they have to accurately assess how well they did? When we self-assess our ability, how do we know that we aren’t just randomly picking numbers from a distribution in our head and that we are actually making a grounded and logical analysis?

Of course, when I questioned all of this to a friend, all they said was, “You’re thinking about this too much. People are just overconfident. I’ve seen them.”

And perhaps they are right. Dunning-Kruger effect or not, psychologically-based or statistically-based, humans are still humans, one more unique than the next. One can use broad experiments to draw conclusions about data points overall, but it doesn’t mean you’ve discovered something about how an individual acts. I suppose the only real way to do that is to go out there and talk to some folks.

Thoughts: Wei Wei Daigakusei

Sun, 27 Feb 2022 00:00:00 +0000

I had previously written a separate post briefly talking about Yabai T. Most of their songs don’t have English translations, so I thought I would try my best to translate them and talk a bit about what I learned in the process. I’ve also been wanting to keep up with my Japanese, so it’s a win-win. To preface this post (and the rest of them if they come): I’m a hobbyist. I’m missing a lot of linguistic and cultural experience relating to Japan. I’ll try my best, but the translations are not going to be or even aiming to be perfect. This is mostly a learning exercise for me. Anyways, onto the fun stuff!

First on the chopping block is Wei Wei Daigakusei which was released in 2016 as part of Yabai T’s album, We love Tank-top. The song describes a group of 5 college students in the Kansai area living a rowdy and free life instead of studying hard at school. While the premise isn’t exactly unique, Yabai T frames it with a uniquely Kansai spin, including a few callouts to locations that have ties back to Osaka and the Kansai area in general.

Translation Notes

The first few lines of Wei Wei Daigakusei mention two phrases, 産近甲龍 (sankinkouryuu) and 関関同立 (kankandouritsu). These are both acronyms for two groups of famous private universities in the Kansai area.

Sankinkouryuu:

San: Kyoto Sangyo University
Kin: Kinki University
Kou: Kounan University
Ryuu: Ryuukoku University

Kankandouritsu:

Kan: Kansai University
Kan: Kwansei Gakuin University
Dou: Doshisha University
Ritsu: Ritsumeikan University

The Sankinkouryuu are a group of 4 semi-elite universities while the Kankandouritsu are a group of 4 elite universities. All 8 are considered quite prestigious and tough to get into though, hence their notability. Of course, these rankings are just for Kansai private universities. Kantou has their own pyramid of prestigious private schools, although I was not able to find definitive information about exactly which ones they were. Beyond these pyramids, there are even more elite schools since Japanese public universities are viewed as more prestigious than private ones. The schools making up what were the imperial universities would probably be considered the most prestigious in the nation.¹

Later on, the song also mentions 鳥貴族 (torikizoku) which is a famous izakaya chain. Torikizoku originated in Osaka but has spread out around Hyogo and Tokyo. It appears their specialty is chicken skewers as well as other izakaya fare like alcohol and small bites. Naturally, no Torikizoku exist in the US as far as I’m aware, but the abundance of chicken skewer photos I found both on their Twitter and on other various English-language blogs has started to make my mouth drool. If I ever get the chance to visit Osaka down the line, I’m going to check them out.

Another location mentioned is the スポッチャ (supoccha) or Spo-cha, a portmanteau of the words “sports” and “challenge.” Spo-cha is an indoor, all-you-can-eat (play?) sports complex aimed at both adults and children offered at various Round 1 entertainment centers. They seem to offer a wide variety of activities, from traditional sports like basketball and soccer, to more unorthodox activities like go-kart and full-body-inflatable-ball-thing(?). Some US Round 1’s also have Spo-cha as well with nearly the same list of activities which means this is something that I could realistically try out someday.

Closing Thoughts

Wei Wei Daigakusei was one of my favorite Yabai T songs, even before I dived into the translation, purely based on its tune and theme. There’s something amusing about a high-energy song featuring a bunch of college students going all out with their lives. It also spoke to something that I wish I had during college. Like many others, I spent the latter years of uni stuck at home because of the pandemic. Being able to listen to a fun song about college students and pretend I was with them having a fun campus life during that time was a bit therapeutic in a way.

Not every imperial university was in today’s Japan since the Empire of Japan had territory outside of the modern-day country. Two of them actually became the modern-day National Taiwan University and Seoul National University, both of which are still very prestigious. It’s interesting to trace how the past, whether positive or negative, ends up shaping the future like this. ↩

Yabai T-Shirts Yasan

Sat, 12 Feb 2022 00:00:00 +0000

I’m going to try to write on a non-project related topic and see how it goes.

One of my favorite J-pop bands is Yabai T-Shirts Yasan (lit. The Bad T-Shirt Shop), or Yabai T for short. I first heard about them through one of their songs, KawaE, which was used as the credits theme for the Nisekoi live action, and have been enthralled with their work ever since.

Yabai T is a 3 member rock band founded in Osaka that’s been active since 2014 to present. Their 3 members are Takuya Koyama (guitar, vocals), Aribobo Shibata (real name Arisa Shibata; bass, vocals)¹, and Mori Morimoto (real name Iketa Morimoto; drums).

In terms of their appeal, the band’s music is fast-paced and screams flippant young person energy. Naturally, their primary audience seems to be young adults (age range 18-35 if I had to guess). Yabai T is unique in their often unorthodox and out-of-left-field choices for song topics (ex. wanting to feed a cat, wanting to buy a drone, Morimoto’s name being Morimoto, etc). Songs often make use of word play and Kansai dialect which gives them an air of youth. However, this may also be one of the biggest factors for their lack of presence in the West despite seeing large popularity at home; their works are not really aimed for anyone outside of Japan.

While Yabai T’s primary audience is limited, this does not discount their phenomenal sound. Yabai T’s performances almost always get you pumped up, especially with their faster songs. Takuya and Aribobo tend to duet in most of them, with Takuya screaming into the mic and switching off to Aribobo’s high-pitch, more shrill voice. This fun, life’s-not-so-serious noise is what makes this band special to me. Yabai T feels fun and relatable (even to an American an ocean away). I think that’s part of what makes me and a whole slew of other fans love them so much.

According to an interview, Aribobo actually lived in the US for a while! Supposedly she also lived in Maryland, but JP Wikipedia has scrubbed that bit of detail. An old revision’s citation leads to a 2017 radio interview that might just simply be lost to time. ↩

Method Missing

Sun, 17 Oct 2021 00:00:00 +0000

Let’s try a shorter-form blog post today. Suppose you are bored and want to learn more about a Ruby library. You do the sensible thing: read the source, starting from a class the library’s clients would consume. You expect to find a plethora of method definitions or at least some mixins or class inheritance that might point you in the right direction. Instead you get the following:

class Foo
  @bar = ...
  def method_missing(method, *args)
    @bar.send(method, *args)
  end
end

If you are like me, then the first question you may have is, “Author. Where. Are. My. METHODS?”

Then the old wise-sounding voice in your head would probably echo something vague in response like:

Seek not what lies before your eyes.

And you would thus be enlightened.

Method Missing

The secret to this mystery lies, of course, in Ruby’s method_missing method. Normally, when a method call is made to an object that does not define that method, NoMethodError is thrown. However, by implementing method_missing, one is able to essentially catch calls before they fall by specifying behavior that gets executed dynamically at runtime. For instance, in our toy example, method_missing catches method calls in Foo, gets the method’s id, and uses send to redirect the method call to @bar instead.

A major use of method_missing seems to be in providing concise implementations of delegation (our snippet above is actually an example of this). Delegation usually is more verbose. For instance, in Java, one would have to manually redefine every method being delegated in the parent’s class to achieve the same functionality. Ruby’s method_missing seems to make the whole affair shorter to write and easier to maintain.

Another interesting application of method_missing is in creating domain-specific languages (DSLs) by parsing method names. As an example, Kim Bekkelund has a very concise XML DSL implementation using method_missing.

method_missing is also more generally a case of metaprogramming, a feature of some languages that allows code to be treated as data. While metaprogramming is not limited to Ruby, Ruby seems to be particularly (in)famous for it.

A Hypothetical Pitfall

Like all things method_missing can be overused and abused. For one, method_missing has very high reach since it can respond to any undefined method. Sometimes, this can lead to unexpected behavior.

Let’s contrive an example. Suppose we are making a dictionary API. We love method_missing for some reason and decide to use it to write our dictionary:

class DictionaryAPI
  def method_missing(method, *args)
    method_name = method.to_s
    if method_name.match(/^define_/)
      term = method_name.sub('define_', '')
      return {
        :term => term,
        :definition => @external_source.retrieve_definition(term),
      }
    else
      raise NoMethodError
    end
  end
end

Our dictionary parses terms from method calls of the form define_{TERM} and then retrieves the definitions from an external source before returning a response.

This works fine for most common words:

d = Dictionary.new
puts d.define_dog

# { :term => "dog", :definition => ... }

But what if someone decides they want to find the definition for the term “singleton_method”?:

puts d.define_singleton_method

# dictionary.rb:18:in `define_singleton_method': wrong number of arguments (given 0, expected 1..2) (ArgumentError)
#        from dictionary.rb:18:in `<main>'

Oops, define_singleton_method is a method that already exists for all objects. Back to the drawing board…

This example is admittedly contrived. In this case, Ruby has also stopped us very quickly because the function arity differs, so little harm was done. However, what if this API existed at a larger scale? Perhaps DictionaryAPI inherits from a long line of ancestors, and we had a method name that silently collided with a name from one of the ancestors. Perhaps there is no collision now, but there will be one in the future due to a change in an ancestor.

Furthermore, it would be hard to catch this collision in testing since most people would just test common words like “dog” and see expected functionality. And what happens once someone does eventually discover the collision? The whole API needs to change. What if we end up colliding again with something else after the changes?

The main issue is that although name collision itself is not specific to method_missing, by using method_missing, we open up the potential to collide with an infinite number of function names rather than just one we manually defined.

Because of this, I feel that method_missing and metaprogramming overall are double-edged swords. Metaprogramming opens a lot of doors. On one hand, it allows for a crazy amount of functionality to be implemented with very little code. On the other hand, using metaprogramming haphazardly can lead to a lot of unforeseen issues, such as the method name collision scenario. This means that much more care and attention needs to be taken when using it, and these mental acrobatics may not be worth the benefits of metaprogramming at the end of the day.

The Nand2Tetris Experience

Wed, 18 Aug 2021 00:00:00 +0000

I recently went through the Nand2Tetris course. The motivating goal behind the material is to learn about the foundational hardware and software components of a computer by building one from scratch. As the name suggests, the course starts you off with nothing but a NAND chip and, through a series of projects, helps you build your way up to being able to hypothetically program Tetris. Along the way, you get bite-sized tours of all the glue in between, everything from chip design to virtual machines to compilers, even to a very bare-bones operating system!

Chips Chips Galore!

The first half of the course consisted of defining the machine hardware. The general idea of these sections was to incrementally take previously-fabricated chips and build new, more advanced ones, ultimately culminating into our computer’s hardware.

Starting with the NAND chip, we were able to design and obtain a few basic logic chips. These chips, along with a pulsing clock, opened the door to several chains of fabrication that would lead us to a few necessary higher-order components. For instance, one chain went from adders to an ALU to a CPU, and another chain took us from flip-flops to registers to RAM. Once these higher-order components were made, we could finally assemble them as our machine, a very basic computer called Hack which could read and execute sequential instructions from a ROM. To make programming Hack easier, we also wrote an assembler to translate Hack assembly mnemonics into the actual program binary that gets loaded into ROM.

Looking back, despite the entire hardware section being a new experience for me, I was surprised to find that much of the chip designing felt natural and methodical. Once I had made additional logic chips from the starting NAND chip, many of the more complex components came together like pieces to a puzzle. Full adders could be chained together to make larger n-bit adders, and RAM could be doubled by recursively applying the same technique used to build the previous RAM chip.

Coming from the software side, there were many nuances to hardware that also took some getting used to. One thing that became clear very quickly was that patterns learned from software do not always translate to hardware. As an example, say you have 8 individually addressable slots of memory, and you want to write a bit to a specific slot. Suppose in hardware, each slot takes two signals, one that indicates an incoming value (new_value) and one that locks the slot, indicating whether it should update its current value to the incoming value or not (should_write). In addition, suppose all the slots are bused together. In hardware, to write to a certain slot, you would turn off should_write for all slots except the target slot and then flood the incoming value to new_value for all slots. From a software perspective, however, these 8 slots of memory would normally be represented as an array. All a software programmer needs to do in such a scenario is target and write to the slot via array indexing. Any notion of flooding or turning on and off locks from the hardware implementation becomes lost to the software programmer.

Having to change my mindset to work in a world of wires rather than bytes took some getting used to. I found that it was almost easier to run with the “electricity is like water” analogy and think about the circuits as plumbing rather than algorithms. Signal was water that I wanted to get to certain places and not others, and each gate changed the way the water flowed.

In the end, though, I finally managed to build my way up through all the hardware components to obtain a shiny, new virtual computer. With the hardware and assembler built, I could now begin the software section of the course.

Machine-to-Machine

The first thing we needed to address was the fact that our machine’s assembly language was still much too primitive. Our software suite was in dire need of a few more features, which leads us to the virtual machine (VM). In brief, the VM is an extra layer that sits directly on top of the Hack machine with the sole purpose of providing a few additional tools to make it easier to support higher level languages on our machine. For our VM, these tools were mainly a stack-based language, user-defined functions, and the concept of pointers.

Overall, the VM is important, primarily because it brings more ease with managing complexity. While we could have written a compiler that translates from a high level language directly to Hack assembly, building one without an intermediary is tougher and more error-prone than doing so with the VM layer in between.

Building a VM also has the added benefit of easy software portability. Suppose our VM happens to be a popular standard that was implemented on other computers as well. Software written for these other machines and compiled to the VM language can simply run on our VM implementation without any additional work¹ which makes it easy to add new software to our ecosystem. This kind of portability benefit is perhaps best seen in a language like Java. All a new device needs to do to be able to run Java is implement the JVM for its hardware. Once that is done, as long as the specification for the JVM and Java bytecode do not change, the device can effectively run Java programs, even as Java itself continues to update.

Of course, getting a VM is not free either. We needed to implement all of our VM’s functionality in Hack which was done in the form of a VM language to Hack assembly translator. Once that was done, the next step was creating the final tool in the compilation toolchain: a high level language compiler.

The Compiler

In order to write programs for our computer in a language that was a little more user-friendly, we needed a compiler to translate the course’s high level language, Jack, into VM code. The Jack language provides a much more reasonable programming interface, resembling Java and C to some degree and providing familiar concepts such as types and classes.

Construction of the compiler was split into two parts: a lexer, which converts text into tokens, and a parser, which converts tokens into semantic meaning. The parser we had to make in particular was a LL(0) parser² which means it parses tokens left-to-right, derives the leftmost non-terminal token, and looks ahead 0 tokens when determining how to proceed. Usually a parser generates an abstract syntax tree which is then passed off to other modules for compile-time checks, optimizations, and, ultimately, compilation. However, the scope of the course seemed to only ask us to generate code directly.

Working through it, the compiler ended up being quite a reasonable section despite its foreboding name. I had done similar projects for school, so this part was actually old news for me. Many of the VM’s abstractions also off-loaded the logic for the code that the compiler needed to generate by providing a stack and taking care of copying values to it during function calls. The biggest problems I ran into when writing the compiler were dealing with pointers and an issue where certain identifiers would be lexed as keywords. It turns out that the latter problem occurred because I was not lexing greedily which was easily fixed by comparing the lengths of potential tokenizations and proceeding with the one that consumed the most text.

Remembering to Forget

With a compiler for a high level language added to our tool belt, the final part of Nand2Tetris was to write an operating system (OS). Yes, my friends. A fully-fledged operating system just like Microsoft Windo-. In reality, all we needed at this stage for our operating system were a few foundational libraries to supplement the Jack language, handling basic features such as more advanced math, screen and keyboard IO, and memory management.

Of the modules I needed to write, I found the memory management one to be the most interesting. OS memory management was something I didn’t really have a clear understanding of before, so having the chance to finally write something that actually did it was quite exciting.

We were introduced to two ways to manage memory. The first way is a naive method where memory is never recycled. A pointer starts at the beginning of the heap and is incremented anytime memory is allocated. This, of course, will exhaust the heap very quickly over time.

The improved method asks us to, instead, allocate memory via blocks on a linked list. The list is implemented directly on the heap and consists of nodes that hold runs of memory. Whenever memory is needed, we find a block that contains enough space and split off a necessary amount of it for the requester. When memory is freed, the freed block is appended to the linked list, allowing it to be re-used on subsequent allocations. A program starts off with a list that contains one node representing the entirety of the heap. As memory gets allocated, the biggest node shrinks as blocks are split off, and, as memory gets returned, the list grows as broken off blocks are appended back on.

One problem with allocating blocks of memory like this is that you eventually run into memory fragmentation. The heap ends up becoming split into blocks of very small sizes, and it becomes impossible to allocate memory for bigger structures because no blocks are big enough to hold them. There appear to be a variety of methods to defragment memory by compacting free blocks in the heap, but doing so efficiently seems to be a complex endeavor.

While the memory module felt like the most complex part of the operating system, the other modules were non-trivial as well. Eventually, I managed to write out and test every module in the operating system, finally bringing my Nand2Tetris journey to an end.

Thoughts on the Course

Overall, I found Nand2Tetris to be fun and enlightening. It was interesting to see a lot of the lower level computing I generally don’t get to encounter, and the course gave me a much greater appreciation for the kind of effort that goes into making and maintaining lower level software (see appendix A).

Being able to see how a computer is made from the ground up was a very eye-opening experience for me. One of the most difficult concepts to grasp when I first started learning to program was understanding how computers worked from top to bottom. For all intents and purposes back then, the computer was magic in a box that could run Python and play Youtube. How my computer managed such complexity was unfathomable. Later in school, I had the chance to explore lower level languages like C and assembly as well as compilers. I learned about the compilation process and memory allocation which got me a little closer to seeing the bottom but still left me with many questions about how these lower level tools were made, and what existed that was even lower.

Although I am still peering down into that abyss today, I think that Nand2Tetris helped clear up a lot of the haze. The biggest benefit of the course is that it starts you at the bottom and asks you to build an alternative to what we have today. As you work your way up, you begin to see familiar sights, parallels between your toy projects and real-world tools on the other side. And from these parallels, you slowly gain a better idea about what goes into those other once-enigmatic pieces of software and hardware.

Appendix A: Big Bugs

One thing that became readily apparent the deeper I got into the course was the difficulty of testing projects at higher levels in a holistic manner. The course, thankfully, only expects you to complete projects piecemeal rather than compiling entire programs down to binary. To compensate for this, there are many many tests your programs need to pass at each stage. Yet, in spite of all the testing, I always had a gnawing fear throughout the course that something I wrote 2 projects ago might still have a latent bug in it somewhere. If I were to realistically use this entire toolchain I wrote and something broke that was not supposed to, the bug could be layers deep or perhaps even the result of two bugs at different layers interacting in an unforeseen manner. In short, this would be a nightmare to debug. To combat this fear, industrial-grade pieces of software have a lot more attention and rigor thrust on them, and in the space of compilers, much effort goes not only into making the compiler but also proving that a compiler is correct. It is still, however, nerve-wracking to think about a bug existing at such a foundational level that it compromises all the software running above it.

While I have only been musing about accidental bugs so far, perhaps more scary are intentional ones. Ken Thompson famously wrote a lecture, Reflections on Trusting Trust, where he describes the potential threat of discreetly compromising the UNIX C compiler with a backdoor for the UNIX login command. The bugged compiler was designed to be able to replicate the backdoor even on subsequent compilations of clean compiler source. As long as one managed to set the bugged compiler up as the official C compiler, it would be nearly impossible to detect the bug without tracing the assembly output due to having circularly trusted the C compiler binary that was used to compile the compiler in the first place. Of course, this trust bubbles up. Anything that trusts the C compiler is compromised. Anything that trusts anything that trusts the C compiler is also compromised. Scary stuff.

Except for anything that interfaces with libraries that rely on specific hardware. ↩
The authors acknowledged it actually needed to look ahead 1 token sometimes, but they still considered it LL(0). ↩

Maze Contest: A Quick Analysis

Sat, 10 Apr 2021 00:00:00 +0000

One of my CS classes this semester runs a semester-long contest to see who can write the fastest concurrent maze solver. Submitted programs are tested on a battery of 7 mazes that may or may not be solvable and a public scoreboard is updated quasi-daily with each competitor’s runtime performance on all the mazes as well as their rank and score. With such a treasure trove of data before my eyes, I could not resist and set out to run some analysis on it.

Parsing and Cleaning the Data

As it is, the scoreboard is simply an HTML table on a web server which needed to be parsed. The good news is that this part of the project was already done. I had been working on a separate project before this to visualize the data from our contest and had been regularly parsing and saving snapshots of the scoreboard with a Cloudflare Worker for a few weeks already, so getting the data was a simple matter of downloading the latest snapshot my worker had saved. However, my worker stores the scoreboard snapshots as serialized JSON objects which are not too convenient for running statistics on. I ended up writing a small Python script to transform all the data into a CSV, ending up with the following table:

#	name	rank	maze1	maze2	maze3	maze4	maze5	maze6	maze7	score
1	*	1	1.100	0.108	1.468	2.469	0.330	0.380	1.159	139
2	*	2	0.987	0.210	1.600	2.431	0.288	0.339	1.057	152
3	*	3	3.742	0.098	10.209	6.144	0.375	1.697	5.770	168
4	*	4	2.202	0.268	2.587	5.421	0.363	0.494	1.859	176
5	*	5	3.785	0.096	10.326	6.110	0.374	1.685	5.912	202
6	*	6	3.761	0.097	10.242	6.120	0.380	1.682	5.981	210
…	…	…	…	…	…	…	…	…	…	…

At this point, the data still needed to be cleaned. Not all programs on the scoreboard ran correctly for all mazes, so these entries needed to be filtered out. Since these failing entries also caused the columns end up with non-numerical data, each maze column and the score column also had to be be converted to numerical data.

# Read and filter data
data <- read.csv("./out.csv") %>%
  na_if('err') %>%
  na_if('bad') %>%
  na_if('ovr') %>%
  na_if('FAILED') %>%
  drop_na()

for (i in 1:7) {
  maze <- sprintf('maze%d', i)
  data[[maze]] <- as.numeric(as.character(data[[maze]]))
}
data$score <- as.numeric(as.character(data$score))

Filtering the data brought the number of entries from 155 down to 132 points. Now that we had some pretty clean data, it was time to dive into some statistics!

Maze Runtime Statistics

Exploratory data analysis was first performed on the data. This consisted of graphing a histogram and generating some statistics for the runtimes of all 7 mazes.

Maze #	Mean (s)	SD (s)
1	3.857	0.757
2	0.105	0.0234
3	10.262	1.860
4	6.367	1.623
5	0.385	0.037
6	1.697	0.291
7	5.947	1.147

In general, it seems that mazes with greater average runtimes displayed greater standard deviations. This is probably because mazes with greater runtimes are likely larger, unsolvable mazes. This means more variability in how programs may proceed with searching the maze over time and thus more variability in the amount of time a program would take to solve the maze or determine that it is unsolvable.

Despite this, the histograms and standard deviations seem to show that, overall, there is not too much spread in the runtimes for each maze amongst competitors. This seems to indicate that most programs are performing quite similarly to each other on all of the mazes.

While these histograms give a good initial look at the runtimes, it also divorces each program from its performance on all 7 mazes as a whole. To see how each competitor’s program was related to one another, we need to cluster the data.

Clustering the Competition

To make the clusters, each competitor’s maze runtimes was treated as a 7D vector with the runtime for each maze occupying a dimension. The vectors were fed into K-means with 3 centroids which gave the following:

runtime_data <- data %>% select(starts_with('maze'))

# Kmeans
kmeans_res <- kmeans(runtime_data, centers=3)
kmeans_res$centers

Cluster centers:

#	maze1	maze2	maze3	maze4	maze5	maze6	maze7
1	1.453400	0.1904000	1.96160	3.55420	0.3530000	0.472400	1.377400
2	3.909024	0.1013333	10.59048	6.35104	0.3845714	1.745048	6.128159
3	9.255000	0.1880000	10.32500	22.46000	0.6390000	1.706000	6.007000

Each data point was also transformed into 2D space for visualization by applying PCA and graphing the first two principal components of each transformed point.

# PCA
pca <- prcomp(runtime_data, center=TRUE, scale.=TRUE)
summary(pca)

Importance of components:

metric	PC1	PC2	PC3	PC4	PC5	PC6	PC7
Standard deviation	1.9748	1.4750	0.65062	0.47248	0.43497	0.24521	0.17016
Proportion of Variance	0.5571	0.3108	0.06047	0.03189	0.02703	0.00859	0.00414
Cumulative Proportion	0.5571	0.8679	0.92835	0.96025	0.98727	0.99586	1.00000

The PCA summary shows that the first two PCs capture about 87% of the variance so the transformation still retains most of the information from when the points were in 7D.

From here, we can begin to characterize our 3 clusters:

Cluster 1: Gotta Go Fast

Cluster 1 consists of 5 points that are spread decently far apart. The cluster’s center shows faster runtimes compared to all other centers on each maze except maze 2.

Overall, this group seems to be comprised of the fastest programs. Due to their spread, the creators of these programs probably took different approaches from each other to design but ultimately still managed to create programs that were able to out-perform most of the other solvers on most of the mazes.

Cluster 2: Most of Us+

Cluster 2 is the largest cluster at 126 data points and also the most densely clustered with only 6 points deviating significantly from the main blob. Compared to the cluster 1 center, cluster 2’s center displays about 2-5x slower runtimes with the greatest difference being in the maze 3 performance (1.96s vs 10.59s).

Almost all of the submitted programs fall in cluster 2 and nearly all of those fall within the main blob. This seems to support the earlier observation from the histograms that most programs in the contest perform very similarly. While the performances of most programs are similar, it is difficult to say whether this similarity in performance is because of similar implementation or because most of us were hitting the same barriers to improvement despite having different methodologies.

Cluster 3: Pluto

The final cluster consists of a single data point, so nothing can be said about spread within the cluster. The centroid cluster shows similar maze performance to that of cluster 2 with 2-4x slower runtimes on mazes 1, 2, 4, and 5.

While this point is far from the cluster 2 blob, it is only the tip of the iceberg. There are 23 failing programs that were initially filtered out and not considered in this analysis. If we had included those data points, setting each point’s missing runtimes to an arbitrarily large value, the graph might have instead shown a fan of points stretching out beyond this one.

Closing Remarks

Ultimately, the data analysis yields a very boring conclusion: most everyone is about the same. While we do have a few outliers, they do not amount to many; over 70% of the class is contained within a single, dense blob.

I also haven’t mentioned how programs are ranked. While you would think it’s based on fastest average time across all mazes, it’s actually based on smallest cumulative rank across all mazes. A competitor’s rank for each maze is summed with smallest summed value corresponding to highest ranker. This means that even if your program significantly out-performs everyone else’s on a large maze, if it is not optimized for a small maze and runs 0.1 seconds slower than everyone else’s, you are ranked last for that maze and drop rank significantly. Ultimately, this means that while our clustering says something about the performance of the programs, it does not say much about their ranking in the contest. While many members of cluster 1 are also high rankers, being in cluster 1 does not guarantee high rank.

Of additional note is that this analysis is only based on a snapshot of the scoreboard at around the time of writing. The contest runs until the end of the semester, and we are free to improve our programs however we like. It is very possible for the clusters and rankings to change as competitors find smarter and leaner ways to solve mazes. Perhaps the everyone blob will scatter; perhaps we will all end up in cluster 1. Only time will tell, but until then, let the maze solver tournament arc continue!

UPDATE 08/19/21: Convert HTML tables and code output to Markdown tables.

Misaka Net

Mon, 16 Nov 2020 00:00:00 +0000

Since my personal site went through a recent re-haul (I can’t believe we’re on v4 now…), I figured now would be a good a time as any to, hopefully, start up writing this blog again.

As you can all tell from the title, today’s topic will be: the Misaka Network! Or rather, the strange and haphazard distributed computing project I made named after it.

Background

Side A: The Misaka Network

When the COVID-19 pandemic hit, I suddenly found myself with a lot more extra time which I wisely invested, as any young adult would, into games and video entertainment. Specifically, I recently dived into the fantasy sci-fi world of A Certain Magical Index (broadly known as the Toaru franchise) and have now unfortunately found myself neck-deep in the fandom.

Of the more interesting characters in the series are the Sisters, a group of 10,000 some genetic clones copied from a certain high-level electromaster, Misaka Mikoto. The Sisters’ main power is that, being electric-based espers, they are capable of linking their brainwaves through EM waves, allowing them to form a massive distributed computational network known as the Misaka Network.

When I first heard about this concept, it intrigued me. I was only tangentially familiar with the concept of distributed systems, so as the series continued to develop on the idea of the Misaka Network further, introducing details such as a central controller for the entire network and having plot points centered around the network being compromised, it got me into thinking about how the Misaka Network would work in real life.

Side B: TIS-100

Now, let’s jump a bit back in time:

I had gotten my first taste of assembly when I started writing 6502 for small NES homebrew projects around 2018. During that time, I was also recommended the game, TIS-100, an assembly puzzle game created by Zachtronics. The game is based around a fictional distributed assembly architecture made up of several low-memory nodes. While one can only write very small assembly programs on each node, the main gimmick is that the nodes can communicate with each other, forcing the player to have to split the logic of their programs into a network of code snippets. As you play further into the game, juggling nodes and their limited program memory becomes more challenging, especially as the problems get more difficult and stack nodes get introduced.

While I have yet to finish TIS-100 since it feels just a little bit too much like school work, the idea of a distributed assembly system piqued my interest back then.

sideA.join(); sideB.join();

So exactly where do these two unrelated areas lead to? Well, I suppose I was feeling a little more excitable this year because I woke up one morning thinking to myself:

What if we could use the concepts behind the Misaka Network to build a TIS-100 distributed system in real life …and then deploy it all with Docker! ahahaha-!

Now a few months later, I live to tell the tale of what happened next.

Laying Plans: Can It Actually Be Done?

The first thing I needed to do was figure out whether this was even possible or not which naturally involved learning a little about distributed systems and how they work. Cursory research brought me to this blog post from Digital Wing describing how to build a distributed MapReduce. Scanning through, it was evident that the system would be made up of different types of nodes, mostly worker nodes that respond to a central master node. The example was also both written in Go and used HTTP for communication between nodes, and as I wanted more experience with Go and was relatively comfortable debugging HTTP requests, both seemed like good choices to start out with in my own project. While the DigitalWing example was deployed on Kubernetes which I really wanted to try, I figured that I may have had too much on my plate already and opted to see if I could use something simple like Docker Compose instead. After all, the distributed system was only going to be half the project. I would need to figure out how to handle TIS-100 assembly as well.

Adventures in Parsing ASM

The second thing I needed to do was figure out how to interpret TIS-100 assembly. Thankfully, yours truly happened to take Introduction to Compilers instead of Operating Systems this semester, so he has some idea of how to lex and parse code. Since TIS-100 is assembly to begin with anyways, there was actually not much work to be done to parse it. The two biggest challenges I faced with processing the assembly ended up being figuring out how to deal with jumps and how to store each instruction.

The former was not too hard but certainly different from what I was used to. With languages where whitespace doesn’t matter, such as C or Java, code being on a certain line doesn’t really play a role in code execution, but with assembly, I needed to keep track of the line an instruction was on in order to implement jumping. This meant that while usually a program written in a higher-level language gets parsed into an abstract syntax tree, my assembly instructions ended up just being stored in an array where each instruction’s line number was tied to its position in the array. Moreover, in TIS-100, one can also specify labels to tag a line to jump to, so when I parsed the assembly, I also needed to create a lookup table, mapping labels to indices in the instruction array which could be used during interpretation to figure out where to jump to for a specific label.

The latter challenge was a bit confusing at first. Of the two languages I had used to write parsers in before for school, OCaml had algebraic data types (ADTs) which offered wide flexibility with nesting and Racket was dynamically typed so I wasn’t constrained with the type of structure I could store in other structures. With Go, however, I was stuck in a statically typed world without inheritance to even try to cheese ADTs! In the beginning, I thought maybe I could represent each instruction type as a different structure and parse them into an array of interface{}, but I quickly realized that would end up as a giant casting mess. After sitting dumbfounded for a while, it dawned on me that I had overcomplicated the situation. Each assembly instruction is quite simple: it has an instruction type followed by a fixed number of arguments based on the instruction type. If that was the case, why not just store everything as an array of string where the 1st element was the instruction type and the rest were the instruction’s arguments, each in string form. So TIS-100 assembly that was originally:

ADD ACC
JMP START

…would end up looking like:

[
  ["ADD", "ACC"],
  ["JMP", "START"],
]

When interpreting, it was simply a matter of switching on the instruction type which was guaranteed to always exist, and when the right instruction was found, process the following arguments. In a sense, I was actually just lexing the assembly into lines of tokens, so perhaps to say I was parsing anything at the end of the day is just slightly overselling the work I did.

Building the Network

Once I was able to deal with TIS-100 assembly, I could move on to taking down the Goliath of this project: creating the actual network. The basis of Misaka Net is a collection of independently functioning nodes, each presumably with its own IP and hostname, where each node can talk to any other node on the network. A node can be one of three different types: program, stack, or master. The master node controls all the nodes on the network, broadcasting commands, loading programs onto nodes, and acting as the center for the network’s input/output. Meanwhile, the stack and program nodes act as workers, with the program nodes executing assembly instructions and the stack nodes acting as stack storage.

While implementing the nodes boiled down to copying the functionality of TIS-100, I still needed to make some modifications in design to get my network working.

Architecture Changes

One difference in architecture between my design and the original TIS-100 deals with how nodes are connected together. A TIS-100 program node has registers in up to four directions that connect it to other nodes. Misaka Net, on the other hand, doesn’t have any concept of this restraint since as long as a node has another node’s hostname, it can communicate with it. This meant that the MOV instruction for node-to-node communication needed to be adjusted. I opted to keep the idea of having four registers, but now these registers were more like one-way mailboxes rather than shared memory. When a node wanted to send a value to another one, it would do so now with a MOV, specifying a value to move and the target location and register to send it to. On the other side, the target node would also use MOV to retrieve the value, specifying the targetted register as the input source.

For example, something in TIS-100 like:

+--------------+     +---------------+
| MOV 3, RIGHT | ==> | MOV LEFT, ACC |
|              | <== |               |
+--------------+     +---------------+

…would now look like this in Misaka Net:

+-misaka1-----------+     +-misaka2-----+
| MOV 3, misaka2:R1 | ==> | MOV R1, ACC |
|                   | <== |             |
+-------------------+     +-------------+

New instructions were also added throughout the process to reconcile design choices made in Misaka Net with the TIS-100 architecture. Notably, PUSH and POP as well as IN and OUT needed to be added to deal with communicating with stack and master nodes respectively since program nodes did not have information about the other nodes on the network.

Architecture changes were not the only challenges. Program nodes also had to deal with interpreting assembly while simultaneously running an HTTP server to respond to requests. This meant that designing the network was beginning to head into an uncharted territory for myself: concurrency.

Juggling Concurrency and Getting Blocked

While I didn’t expect this project to have been my first practical project writing concurrent code, I ended up having to learn a lot about it. Thankfully, Go having concurrency as a first-class citizen of the language made it much easier to reason about many of these concurrent processes throughout the course of the project. That being said, I still ran into a lot of issues involving concurrency and blocking, from needing a thread-safe stack data structure to finding strategic ways to get Goroutines to sleep and avoid eating up CPU usage.

One of the major areas where concurrency came into play was running a program node’s interpreter alongside an HTTP server. Designing this essentially boiled down to sticking an infinite loop inside a Goroutine that would get called before starting the server. The loop emulated the node’s CPU clock, which would interpret TIS-100 instructions line-by-line and update the program counter as long as the program node was in a running state. While this worked fine for the most part, it broke when the node was commanded to reset its assembly execution. I had made the program nodes block execution when sending values to other nodes. However, this meant that they could also never be interrupted while in the middle of trying to send a value, leaving them stuck in limbo, even when commanded to stop execution.

I was in dire need of something that could somehow cancel the HTTP send requests when the program node received certain commands. While I had initially tried setting up more Goroutines to handle cancellations inside my program loop, it was starting to get very messy and cumbersome. I probably should have sat down and considered the name of what I was trying to do since that happened to lead to the exact thing that I was looking for: context cancellation. I ended up learning about contexts and the fact that I could generate a context and cancel function pair, ctx, cancel, which I could carry into the program node struct. Anytime I did a blocking operation, I would pass in ctx into the operation, and then whenever I received a command to stop program node execution, all I needed to do was call cancel which would cause any operation using ctx get cancelled, freeing up the program node! While this meant that the blocking functions I wrote would need some adjustment in the form of a select statement to jump out on cancellation, this was 100 times easier and cleaner than having to juggle more Goroutines.

With the the program node’s biggest problem out of the way, I was moving ever closer to the final goal of Misaka Net: linking multiple nodes together into a network.

Deploying the Network

At this point, I had mostly been testing nodes as single machines and querying them manually through Postman, but the big question now was whether or not they would work together. I imagine in “ye olden days” this would be where the project would have ended, and I would just have had to hope that my code worked or somehow bought and hooked up a bunch of servers to test on bare metal. However, today we have container virtualization and container orchestration tools all within the grasp of the average layperson, and so the show went on.

Docker Compose was chosen for orchestration as it would be the fastest way to setup everything and see it in action. One of the nice things Compose does is set up hostnames for the services defined which meant that referring to any node was as simple as calling to it using the name of the service. Once I had specified all the nodes in my network as services with a few minor hiccups, I started up all the containers. No errors, that was a surprise. I sent a run command to the master node…and surprise number 2, it actually worked! The network I set up was just something simple to ping-pong values back and forth and it had somehow worked without any problems.

At the time of the first test, all I had implemented were the program nodes and parts of the master node, so there was still a lot more work that had to be done. From there, I ended up finishing implementation for all the node types and added a way to interface with the client to do input/output as well. Running the finished network and seeing it actually compute something made me overjoyed. Somehow, my originally half-baked idea was now running for real.

Bonus Rounds

Once I had finished implementing the rest of the network and could actually compute values with it, it was time to add some extra features.

Switch! It’s gRPC time!

The first order of business was converting all intranet communication from HTTP to gRPC. While HTTP allowed for easy debugging early on, gRPC uses protocol buffers which would allow for smaller message payloads and faster communication, both huge benefits for Misaka Net which relies on constant communication between nodes. Also I simply wanted to get some more experience making gRPC applications.

Integrating gRPC into Go was as simple as following the quickstart guide. From there, I needed to create a proto with all my service definitions. I had haphazardly prototyped the routes for when writing the HTTP server, so explicitly laying out the services actually gave me the chance to re-organize the code for my endpoints and figure out exactly what endpoints every node offered along with the data going in and out. This ended up working out very cleanly since each node type just ended up with its own RPC service.

Once I had my proto definitions, I needed to auto-generate the Go code to interface with gRPC and replace each node’s HTTP server with its correspoding gRPC service server. This is where I ended up running into a minor issue with the master node. Since HTTP only cares about the names of endpoints, I didn’t have to worry about the type of worker node the master node was broadcasting to originally. This was not the case with gRPC. I needed a different client to communicate with a program node versus a stack node which meant that the master node ended up needing to know not just the names of all the nodes on the network but their node types as well. From there, it was a small tweak to the broadcast function, allowing for the correct client to be used when sending a command to a node.

Locking Down on Security

While Misaka Net would probably be on a walled-off internal network if was hypothetically used in production, God forbid, I still wanted to add some form of security to it. Following the gRPC authentication guide, I settled on securing connections using SSL. With the help of Nicolas Leiva’s guide on generating certificates and server keys for gRPC, I managed to get myself shiny, new, very untrustworthy, self-signed certificate, with some extra fun in added of course.

However, after setting up gRPC to use SSL and plugging in the certificate and private key, the net was now stalling when I tried to run it. Strange. Fiddling around, I found out that I had attached grpc.WithBlock() as a dial option which was the culprit of the stalling. Removing it led me to the real problem: the client connections were getting refused because my certificate was not valid for any names. Turns out I had not added all the hostnames in Compose to the certificate’s alt_names (whoops). Giving it a second whirl, the net was up and running again as before but now with encryption.

While this ended up working, it felt a little wrong to have a single certificate being used for multiple services. Perhaps it may have been better to generate an SSL certificate for each node registered on the network, and choose the right certificate to attach when communicating with a certain node, but I was ready to put the project down at this point.

Thoughts and Conclusion

At the end of the day, Misaka Net was another weird project and learning experience rolled into one. The design of the network is unfortunately stuck with a few issues, from potential data races with multiple pop requests on a stack node to a low tolerance for node failure to difficulties with scaling when adding new nodes. For some of these problems, simple solutions may be engineerable but others, such as properly dealing with node failure, are not something that I want to tackle anytime soon.

In a sense, Misaka Net ended up being similar to its namesake: both were, at their core, experiments. Although my network wasn’t anything close to a work of mad science, it did teach me a lot about working with concurrency in Go and was my first taste of what a distributed system (albeit a very naive one) may look like.

Appendix A: Fun with Spriting

I happened upon a T-shirt design of some pixel art of the Sisters and Last Order while working on Misaka Net. The sprites were simple enough to re-create, and I thought it would be neat to work them into the repo README in a similar fashion to how a lot of popular projects make liberal use of emojis in their headings.

Appendix B: Thoughts on Toaru and Sci-Fi

One of the things I think a lot of fans find compelling about the Toaru series is its dedication towards trying to back fictional phenomenon with real-world principles. The series actually gets viewers asking, “How would this thing work in real life?” or, “What are the supporting principles behind this ability?” instead of just letting us shrug it off as a fantastical construct.

For example, in the light novel, the esper ability, teleportation, is mentioned to involve 11D calculations because it’s based off of 11D spacetime and creates an ability where, presumably, teleporters are able to move objects through 11D spacetime. Later on, we are later introduced to the most powerful esper in Academy City, Accelerator, who can control vectors which allows him to reflect any attack. But what about teleportation-based attacks? The light novel explains that, well, since teleportation involves an 11D vector, Accelerator still has control over that.

While this kind of world-building is very fun and thought-provoking, it’s not perfect and does lead to a lot head-scratching when holes appear and is, of course, harder to keep coherent because of the extra work needed to find a believable basis for your fiction. That being said, I like fiction like this that tries to explain itself since it goes a long way to preserve one’s suspension of disbelief. Perhaps this is also why something like Jules Verne’s science-backed sci-fi caught on and is so fun to read because the fictional world you peer into feels like it could really exist.

VS Code Bolt

Sun, 29 Mar 2020 00:00:00 +0000

Recently, I had started experimenting with Firebase’s Realtime Database. Then one thing led to another, and I eventually found myself working on a language extension for a rules language, but I’m getting ahead of myself.

Background

Taking a step back and providing some much-needed background: the Firebase Realtime Database is a JSON-based, key-value store which makes it easy to start with but also quick to trip up in since data doesn’t have a well-defined structure. Moreover, while a traditional database would be interfaced by a backend acting as a middleman to authenticate users and limit the scope of possible queries, Firebase seems to be intended to directly interact with the client. This means that security becomes a major problem as it has to be enforced directly on the database’s end.

Firebase’s solution to both of these problems is a rule system. Database rules are written into another JSON file and allow one to specify the conditions for data read, write, and validation. The rule system solves everything except it also introduces a new problem: giant, unreadable JSON files. It seems that the people at Google got tired of writing these rules and made a compiler for a new rules language called Bolt which compiles down to the JSON rules.

When my nose led me to Bolt, I latched onto it immediately, having also tired of writing a growing, unmanageable set of rules. Jumping into Bolt, my first instinct was to find an extension for syntax highlighting, but, unfortunately, none of the available ones on the marketplace met my expectations. I made like any sane and normal person and dived into making my own!

Making Language Extensions

While I develop a lot IN VS Code, I had no actual experience developing FOR VS Code. Surprisingly, getting started was quite simple: I was able to bootstrap a new project with the Yeoman VS Code extension generator and all a language extension needed after that was a TextMate file (.tmLanguage.json) where all the parsing rules for the grammar would be written.

So how to write the parsing rules?

The generator thankfully provided a dummy grammar to provide some ground, and I had also just taken a programming languages course at university so writing grammars wasn’t uncharted territory. However, figuring out how to write these rules and finding a place to start was still a daunting task. Looking at the grammars defined by both my Bolt extension predecessors as well as the JS grammar file for VS Code itself, I began to get a grasp of how to write grammars in TextMate.

TextMate works by matching constructs using regular expressions. Constructs can be either matched using a single regex or by trying to match regexes for the beginning and end of a construct. The latter kind of match is especially useful for constructs that enclose things such as strings.

As an example, this regex was used to match the beginning of function declarations:

(function\\s)?\\s*([a-zA-Z_$]\\w*)\\s*(\\()((?:[a-zA-Z_$]\\w*)(?:,\\s*(?:[a-zA-Z_$]\\w*))*)?(\\))\\s*{

…which looks like a complete mess at first until it gets broken down:

The regex above uses a lot of captures groups since TextMate uses those to assign scopes to parts of a construct which determines how text gets highlighted. In addition, capture groups can also be used to nest patterns for more constructs as seen in capture group 4 below which tries to find and match argument names in the function header.

"beginCaptures": {
	"1": {
		"name": "storage.type.function.bolt"
	},
	"2": {
		"name": "entity.name.function.bolt"
	},
	"3": {
		"name": "punctuation.definition.parameters.begin.bolt"
	},
	"4": {
		"patterns": [{
			"name": "variable.parameter.function.bolt",
			"match": "\\b(?:[a-zA-Z_$]\\w*)"
		}]
	},
	"5": {
		"name": "punctuation.definition.parameters.end.bolt"
	}
},

References to defined constructs can also be included inside other constructs which makes it really easy to design a grammar top-down. The patterns shown below match the in-between text of a function declaration block, saying that the inside of a block can either include another function-declaration or an expression (expr).

"patterns": [
	{
		"include": "#function-declaration"
	},
	{
		"include": "#expr"
	}
]

The rest of the project boiled down to writing regex rules to match key constructs and placing them properly into my hierarchy.

The one difference with TextMate grammar rules and the ones I had written for school was that I had to parse entire constructs in one go. The parsers I was used to had much more flexibility, using lookaheads to determine nested constructs. The best I could do in TextMate was to mimic this by specifying the patterns in the in-between of an enclosed construct.

Debugging the extension throughout was also a nightmare. Writing the regexes was already bad enough since they had to be written using escapes, but TextMate’s limitations meant that more work for parsing fell on the regexes, leading to very long and complicated expressions. Trudging through, I eventually got the extension to a place where most of the language constructs were written into the TextMate grammar.

Opening Up Shop

At this point I was ready to release it to the marketplace. Packaging up the extension for release was simple with vsce but my experience with other markplaces told me that publishing my extension was likely going to be a hassle.

Turns out I was wrong.

It was surprisingly easy to publish to the VS Code Marketplace: all I had to do was get an Azure personal access token and publish the package. From there, it was a matter of doing some aesthetic clean up and adding some badges and my Bolt language extension was ready for anyone to use!

Improvements and Conclusion

While the syntax highlighter works quite nicely overall, more could still be done with it. One issue is that I did not name scopes for all language structures so while highlighting may look acceptable with the default theme, other themes might render syntax differently.

It also appears that a lot of JavaScript functionality is supported in Bolt which was not encoded in my grammar. While I had initially tested what JS I could write by compiling arbitrary statements and seeing if they worked, definitively solving this problem would require a deep dive into the Bolt compiler’s source which I am not going to do as of now. However, for the most part, the highlighter covers the kind of Bolt that most people will probably be writing (or so I hope).

In the end, I am surprisingly proud of this project despite not having written any code for it. I didn’t expect to ever use any of the parsing stuff I learned when I took programming languages, having written it off as too low-level and faraway from the world of app development, but here I am today, a wiser man. I also find it ironic that in an effort to not have to write large, unreadable JSON files, I ended up making this project which is, at its core, a large, complicated, and very unreadable JSON file.

Life is full of many mysteries and contradictions.

Appendix A: Fun with FOSS

At the start, I had originally just intended to make a pull request to one of the other Bolt extensions, but when I peeked at their repositories, I found that both projects had actually done a straight one-to-one adaption of a set of rules originating from a package written years ago for Sublime by a Googler who, in turn, had lifted parts of his grammar off of someone else’s personal JS Sublime language package!

This long chain of forking was the primary reason why I had hesitated to make a pull request to begin with. The grammars appeared to have been machine converted from the original XML into JSON which made them somewhat of a nightmare to read and modify. I was also not sure what both creators treated as the ground truth for their grammars and decided it was just easier to make my own grammar from scratch.

On the other hand, it’s amazing to see code(?) work its way up to the present. Some rules from the original JS language package date as far back to 2012 and have haphazardly made their way across different projects for different editors, residing in different forms, to end up being used by thousands today and scrutinized by a flakey college student 8 years later. I think it’s one thing when talking about old code in a long-running project but it’s another to see code move across different projects and contexts over time.

UPDATE 05/25/20: Grammar examples moved to Gist, minor changes to text

UPDATE 01/17/21: Grammar examples moved back as code

adxtools

Fri, 24 Jan 2020 00:00:00 +0000

A little over half a year ago, I got it in my head to make a character mod for the 3DS game Persona Q. While I did not make it very far, I did dive far enough into the sound files to switch tracks and start working on making my own encoder and decoder for them in Go.

Background

Persona Q uses a proprietary file format developed by Criware called ADX for storing compressed audio. The ADX file format has been in use as far back as 1998 in Burning Rangers for the Sega Saturn, and more recently, games from the Persona franchise and others continue to use the format as audio middleware, likely due to its favorable compression-to-quality ratio and looping capabilities.

Strangely enough, the most centralized wealth of knowledge on ADX is actually the Wikipedia page. The ADX file header contains metadata such as the encoding type, number of audio channels, bitdepth, sample rate, looping data, etc. The header also points to the start of the sample data.

Normally an uncompressed file like WAV stores its samples directly, storing each sample as a bitdepth-sized chunk of data. For example, a WAV file with bitdepth of 16-bits could have a sample be an integer anywhere from 32767 to -32768. While this allows for high resolution storage of sound waves, it costs quite a bit of memory. In order to cope with this trade-off, ADX uses two techniques in compression: First, samples are stored as errors from a predicted value derived from previously decoded samples as opposed to an exact value which supposedly allows for higher-quality encoding.

Second, lengths of sample errors are chunked into blocks with a scale value used as a multiplier to recover each error value during decoding. The scales are calculated in such a way that all unscaled values can be stored as tightly-packed 4-bit nibbles instead of 16-bit integers. As a whole, this means that ADX files are able to preserve audio to a high quality while reducing the storage size by 4x.

The Decoder

I started on the decoder in May of last year. I had just started out in Go and wanted to try it out for another project. Writing an ADX decoder seemed to be something within reach while remaining challenging enough to not be a complete cake walk. Since pseudocode for a decoder was already on the Wikipedia page, I would mostly be dealing with figuring out how to implement it in Go, hopefully learning some file IO. I also ended up tacking Cobra on the project to get some experience with a command-line library.

The first trouble I ran into ended up being in Cobra. I had wanted to use Go modules, but at the time I could not initialize a Cobra project outside of the GOPATH. Not wanting to find a workaround, I ditched modules and went straight to implementing the pseudocode. Ideally, I had imagined I would come to understand the file specs by building the implementation, but I ended up only being able to glean a fraction of what the code was doing. Instead of stopping, however, I opted to write now and analyze later. When the time came to test the decoder, I ran a file that I had ripped from Persona Q through it and, lo and behold, it worked!

Well only kinda.

I could recognize the music but the quality was beyond awful bringing us to my second issue. I scoured the code but could not find what was wrong. I had translated the pseudocode perfectly (or so I thought at the time). Because I could recognize the song, I was not doing anything wrong with picking up the samples, but then the problem could be anywhere in the decoding process. Logically, if the algorithm was implemented perfectly then perhaps there was some floating-point error or bits were getting truncated somewhere. It was at this point, I had had enough and put the project on hold.

The Encoder

As more time passed, I became ever more reluctant to work on adxtools. After contributing to aimacode for Google Summer of Code, I was immediately thrust into a new semester with no time to review the ADX file specs or polish my Go skills. adxtools got shelved as one of those eternal WIP projects on my GitHub, that is, until winter of this year.

I had gotten the sudden urge to start working more with Go, started coding up a big project, got burned out as usual, and shelved it before immediately trying to start a new one. However, this time, I also became uncomfortably aware of the high number of WIP repos on my GitHub.

I should finish one of these.

Scrolling through them I saw adxtools once again, an unfinished Go project.

Perfect. I dived right in.

The first point of business was resolving what had stumped me in the decoder so many months ago. Within the first few seconds of reviewing my code, something caught my eye:

a := math.Cos(2)*math.Pi*...

Hold up. What? Why is the pi outside the trig function? I check the pseudocode. It’s supposed to be inside. I fix it and run the decoder for the first time in months, not expecting much. Then I play the file.

It’s perfect. You gotta be kidding me!

Feeling the full strides of success, I figure I may as well write the encoder while I still have time. Initially thinking it would simply be reversing the order of execution from the decoder, I quickly ran into a problem: How does one determine the scale? Our only source of authority, the Wikipedia page, says nothing about it and most other sources on ADX are only focused on describing decoders. Stopping to think, I reasoned that since the scale is merely a multiplier for a block of data, perhaps I could produce it by finding the greatest value in the block, treat it as the 4-bit max and scale the rest of the values against it. Tentatively writing this out, I managed to reverse the algorithm with a few changes and simplifications, such as choosing to encode by block instead of trying to encode starting from anywhere.

Testing the encoder, I ended up with ADX files that once again, had recognizable audio but were very noisy. On the first round, I decoded my test file and ran it through Audacity.

Strange. I was only encoding one channel.

At first I thought I may have been corrupting blocks from the first channel when encoding the second, but hunting down the bug, I found I had entirely missed encoding the second channel at all! Naturally, fixing it didn’t alleviate the noise but in fact made it worse. I continued find and fix bugs in my encoder but at no point was I able to figure out where the noise was coming from.

Truly stumped for the second time on this project, I ended up re-writing the decoder to ensure that I did not have a problem in my understanding of ADX. Re-writing the decoder had the small benefit of optimizing it as the Wikipedia version assumes streaming the audio from anywhere, but since all I needed to do was convert it over, I ended up getting away with fewer file operations, saving some time.

After 3 days of trial-and-error and debug printing, I traced my problem to the one original algorithm I wrote: scale generation. Testing the encoder on a decoded file that was originally ADX, I found my scales to be similar but slightly off to the stored ones. What could be the cause? Did I need to re-think my entire process? Turns out the solutions to many of these big stumps end up being quite simple for some reason. On a whim, I changed my max 4-bit number from 8 to 7, and, suddenly the sound is crystal clear. I was dumbfounded. The 4-bit integer range is from -8 to 7. I had wanted to use 8 under the impression that it was a more conservative max but I was actually scaling some numbers out of range. Apparently this was enough to create noticable noise. With this tiny fix, both the encoder and decoder were finally finished. I merged my branch and wiped the WIP title off of adxtools, feeling quite satsified.

Conclusion

adxtools, despite working, is nowhere near complete. The encoder and decoder work for a small subset of ADX files, specifically those with two channels and 4-bit bitdepths, and the encoder does not support looping. I would, however, like to turn it into a complete encoder and decoder at some point. Criware have also developed several other file formats such as ADX2 which would be worth adding to adxtools in the future.

The greatest benefit of this project for myself was more experience working with Go. I got more comfortable with doing file IO as well as working with modules and got the added challenge of having to optimize my code. While I still do not understand much of the math used in the encoding techniques behind ADX, I did learn a bit of audio file storage from this project as well. At the end of the day, I know ADX is not a popular file format, only seeing prevalence in modding communities. ADX also has been around for long enough that mature tools have already been developed for it over a decade ago, but I think adxtools was, regardless, still an interesting project, albeit mildly infuriating to debug at times.

UPDATE 02/06/20: Incorrect integer range has been fixed.

Speed Trig

Sat, 14 Dec 2019 00:00:00 +0000

At the time of writing, the app is available for Android on Google Play here!

UPDATE (12/29/2019): Now available on iOS as well here

One of my fondest memories from high school were a set of re-takeable quizzes from pre-calculus collectively called “Speed Trig”. The idea was that you had to solve a battery of rote trig problems under time pressure and graded on an all or nothing basis.

While Speed Trig ranks among one of the more stress-inducing experiences from my public education, I do appreciate it for drilling trig skills into me. My problem throughout Speed Trig, however, was that I had no organized way to practice for it. Since the quizzes were re-administered repeatedly, I ended up using the actual quizzes to benchmark my progress. This was not ideal and looking back, I would have appreciated an app for practicing problems. Come years later, while brainstorming project ideas to explore mobile development, I ended up digging up my past and making a simple Speed Trig app.

Background

Foremost, this idea is not original: our high school had a Smartphone Programming Club (SPC) which developed a Speed Trig app years earlier (apparently at the whim of their math teacher club sponser).

Compared to the app developed by SPC, mine was going to be a functionally simpler app. From the outset, I wanted this app to be much more “game” than anything else. Because of this, I chose to make it a multiple choice, timed quiz which would be more appealing to the player than inputting answers directly. I also went in with a heavier focus on trying to get the aesthetics and feel of the game down.

Mobile and Trying to Render Math

Starting development, the first roadblock hit with was rendering the math formulas needed for displaying questions and answer choices. Working in React Native, there were nearly no reliable libraries for rendering formulas, and I was not about to go in and try to write a custom formula renderer. My only solution at this point was to render the formula in an HTML page with MathJax and display that in a WebView, and while it worked, I could not manage to get the WebView to center as I wanted and MathJax rendering also had a very noticable lag. All of this made the app look clunky, so I ended up scrapping the entire thing and just pulling math symbols from Unicode. Although this was a setback, the math I needed to display was not too complicated, so the app did not suffer much. That being said, a fast formula renderer for React Native might be an interesting project for someone to tackle in the future.

Designing for Looks

Coding up the app, the logic ended up being quite simple. All the app does is repeatedly select a trig problem and display it with 4 possible answer choices. A point is rewarded for a correct answer and a heart is lost for a wrong answer. The user tries to answer as many questions as possible, and when either time is up or the user loses all hearts, the final score is displayed.

I was able to write all of this in about a day’s time which meant that most of the work for this project actually went towards making it look nicer. I opted to go for a geometrical, vector art look, mostly because that’s the only type of art I can make reasonably fast. I also wanted animated feedback for answering questions but wasn’t ready to step into the world of React Native animations. Looking for a quicker solution, I eventually found Joel Arvidsson’s wonderful react-native-animatable library. Animations were easy to work in after that, and in no time, I had a very basic Speed Trig app.

Making this Speed Trig app was also a chance for me to try a new thing: adding ads to my apps. An entrepreneuring friend of mine had suggested the idea in the past and this seemed to be a good place to test it out. Using Invertase’s react-native-firebase library to interface with Google AdMob, I was able to quickly add two banner ads. Being the first time using AdMob, I did run into a little bit of trouble displaying ad units, but this was easily solved by just verifying my account. Initially, I had some concern that the banner ads would end up being a major eyesore, but playing around with the app, I found that they only ended up being a minor annoyance. The react-native-firebase library also provides access to other Firebase features like analytics and Firestore which are worth looking into in the future.

Problems and Improvements

Speed Trig was meant to be a fast and simple project, so it does leave a lot to be desired. One of the biggest problems with Speed Trig is the way it stores its questions. The entire question set is a JSON file storing a list of functions which each contains a list of input-output pairs. Although this results in a very inflexible question system as each possible input has to be manually entered, it keeps the question selection implementation relatively clean. However, should I choose to change the game down the road and add arbitrary inputs, for instance, this entire system will have to be ripped out.

In the future, I would also like to add more features like competitive play, rankings, adjustable settings, and perhaps even alternative game modes ~~like kart racing~~, but these are all considerations for another day. Adding some of these features also has the additional overhead of jumping through a lot of app store bureaucratic hoops and red tape which I’m not keen on going through.

Conclusion

Ultimately, I am quite happy with this project. Granted, while it was mostly all design work with minimal coding, I am still proud of the end product. Making a Speed Trig app was always my joke suggestion when brainstorming for ideas at hackathons, so it’s a little weird now that I’ve actually made it. Either way, I got a good amount of React Native and mobile app deployment experience from this project and managed to ship a decent app at the end of the day.

DQNsort

Fri, 23 Aug 2019 00:00:00 +0000

A famous saying goes: There are many ways to skin a cat. These are words that apply especially well to sorting arrays. When asked to sort one, your sane person may immediately go with bubble sort, insertion sort, or perhaps the very lovely and very fast quicksort. The show-offs like to call out radix sort and the jokers always point out bogosort and Stalinsort. Today, however, I present to you something new, pulled entirely out of my limited machine learning knowledge. Yes, ladies and gentlemen, it’s DQNsort!

Background

Reddit’s r/programminghumor was holding a hackathon with the theme “Overengineering”. I had long dreamed of getting into reinforcement learning after watching many of Siraj Raval’s videos on the subject, so this was the perfect time to try to play around with it, and what a better thing to overengineer than a sorting algorithm! It does look like, however, that sorting using reinforcement learning is not a new idea. A paper has already been written describing RPsort which appears to use a more advanced version of RL that generates algorithms in a process called reinforcement programming, but that’s beside the point.

RL in a Jiffy

Reinforcement learning, in general, trains an AI agent to make optimal decisions in an environment. An environment has a state and the agent is allowed to make actions to change this state. The environment gives the agent some amount of reward based on the agent’s choosen action. The goal of RL is to train an agent to learn a policy, or a function that chooses the most optimal action given a state.

A naive agent may have a policy that simply picks whatever action has the max immediate reward at every state, but this is not ideal since this agent becomes too short-sighted and won’t try to go for better long-term rewards. Reinforcement learning agents usually use a better method of learning policies called called Q-learning. Q-learning uses a Q table that acts like a large reference sheet, keeping track of a score for the action taken at each state called a Q-value. Q-values are learned values that reflect both the reward for an immediate action as well as those taken in the future. The agent is able to learn these Q-values by interacting with the environment and updating the Q table.

The problem with traditional Q-learning is that these tables take up a lot of space. Imagine you’re trying to build one for chess: There are millions of possible board states and tens of actions that could be made at each of them! It would be really nice if we could just create a function that took in a state and told us the Q-values of each action, and thanks to deep learning, we can do just that. Deep reinforcement learning is like traditional RL except this time, we’ve swapped the Q table for a neural net called a deep Q network or DQN that allows us to input a state and estimates the Q-values for each action.

Sorting with a Net

The DQNsort agent works by using a DQN to decide which two elements in an array to switch to get it closer to being sorted. The DQN is a multilayer-perceptron that takes an n-length array as a vector for its state and spits out a n^2-long vector that encodes the Q-values for all possible switches that can be made.

I had initially played around with multiple reward functions that did not rely on knowing that a certain element belonged in a certain place, such as rewarding based on the number of ascending elements, but these did not perform very well. I eventually settled on rewarding the agent if it made a change that put an element in the right place, penalizing it for doing the opposite, and adding an additional reward for sorting the array. Visdom was also added early on since I wanted a way to visualize the sorting process and graph the loss from the DQN in real-time.

With everything set, I started testing my agent, and, to no-one’s surprise, it was awful. Although the DQN’s loss went down over time, DQNsort never actually managed to sort anything, opting to repeatedly switch two elements. Searching online for solutions, I came to learn about replay memory. Instead of plugging in one state vector into our DQN, replay memory allows us to do batch training by keeping track of past states from which we sample from and feed into our DQN. However, even after adding replay memory, DQNsort was still doing the same thing. Something was still amiss.

I spent several days double checking my implementation and trying out new reward functions. Scouring the internet once again, I found a possible reason for my poor results: Lack of exploration. Training an RL agent comes in two phases: Exploration and exploitation. Exploration lets the agent randomly choose actions, giving it time to hit many states. Once the agent has explored enough, it switches over to exploitation, greedily choosing the action with the highest Q-value. My sorting agent did not strictly have these two phases. Instead, I had it set up to explore with a constant probability which was likely not allowing it to hit as many states as I needed. Doing a little bit of work, I changed DQNsort to start off with a high probability of exploration, dropping down to a low constant rate over time. However, DQNsort still did not improve after this change.

On a whim, I lowered the number of elements in the array from 10 to 5 and jacked the number of iterations to 10 million. The program started out training slowly as usual, but, after the 30th epoch, it suddenly shot through 50 more of them in an instant. No doubt about it. The only way it could do this was if it had actually learned how to sort! Training finished in under a few minutes, and I loaded the model into my test script with bated breath. It was still getting stuck on a few permutations, but for most cases, the agent had finally learned how to sort! The only changes I made after that were a few tweaks to some hyperparameters and the reward function to try to make it more reliable.

Problems and Improvements

The most tedious part of this project was figuring out what parameters DQNsort needed and the limits it could work within. Even after tuning it for several days, DQNsort is still not that robust.

A big problem with Q-learning in general is that it assumes an agent can visit every state and take every action an infinite number of times to refine the DQN. Reaching all 5 element permutations may have been fine for my agent, but scaling up to 10 elements was nearly impossible. Even at the current 5 elements, DQNsort still often runs into states that cause it to repeatedly take the same action, presumably because it simply has not seen that array configuration before.

DQNsort also rewards based on if the element’s value and its index in the array match. This means that it is only able to sort permutations of (0, 1, 2, 3, 4). This is not terrible if you treat sorting arrays as just another game, but compared to other sorting algorithms, DQNsort is not flexible at all.

Game of the Amazons

Sat, 15 Jun 2019 00:00:00 +0000

Game of the Amazons is a lot like chess except all pieces cannot be killed, move like queens, and shoot fire arrows that also move like queens.

Ok, maybe it’s not like chess at all.

In short, Game of the Amazons pits two players on a grid with an equal number of pieces or “Amazons.” Each player takes turns moving their Amazon and firing an arrow from it, both actions functioning like the queen’s movement in vanilla chess. When an arrow is fired onto a square, the square “catches fire” and becomes a barrier, blocking future movement and arrow firings. The game goes until one of the players cannot move anymore of their Amazons.

I started getting interested in the game after watching a Numberphile video on it a while back. Mathematically, it seems to be a very interesting and difficult game to solve. I wanted to experiment around with something more than the toy example given in the video and started scouring the internet for a playable version. I must’ve been really shot in the head that day because I could not find anything other than Michael Keller’s version of the game for Windows 3. Getting that to run was an adventure all on its own (see Appendix A).

Disappointed at the relative lack of Amazons online, I did the only reasonble thing a bored college student would do: Write my own!

Design

My goal was to build something similar to Keller’s implementation. I wanted to make it easily accessible and experiment with learning React as well so making it in the browser was the natural choice.

I also knew I wanted to separate out my UI code and my game logic from the start. The Amazons game was going to be abstracted out into a set of possible actions that the UI code would interface the game with.

Starting React

Before this, my experience with React was next to none, so most of this was an uphill fight. I also wanted to make this solely a browser project which meant avoiding Node as much as possible. I ended up getting React as a CDN and started testing out some sample JSX… But hold up! The browser doesn’t understand JSX! A quick search taught me about the Babel compiler which among many things can apparently transform JSX into browser-compatible Javascript.

I did the Babel transformation in the browser which popped up a warning telling me to transform before serving in production. I ignored it at first but later in the project, I did some digging and found out I could compile and bundle JS using Webpack first and simply include a single, bundled script file. Of course this meant going back near Node since I was using npm again. However, it seemed that Webpack just transformed and bundled the scripts without involving anything server-side. I think my fears were mostly in wrongly associating Node with being exclusively linked to doing server work.

Anyways, with JSX working, I could finally go my merry way and get started.

Amazons Game Logic

Game logic was contained within a single class that would keep track of the state of the game and provide a set of interfacable actions for the game.

I split the functionality of the game into 3 different actions: choosing a piece, moving a piece, and firing an arrow. Each action would accomodate for the state of the game and alter the board accordingly, usually setting for the next action to take place. This scheme worked very well for player vs player games as it made every action easy to link to some event that happened on the UI side. However, as I started adding rudimentary AI later, I found it to be quite troublesome, needing to save the state of the Amazons instance as actions would directly alter the board.

I also needed a way of highlighting where the pieces could move and fire. Since they follow the movement rules of queens, I had thought to originally just highlight the 4 lines going through the piece. However, this presented a problem: I was unable to detect when a barrier was blocking movement. In the end, I switched over to doing a very watered-down raycast. Normally, raycast is something seen in graphics where, a vector is incrementally traveled along until it ends up within or on some mesh. Here, however, I simply needed to travel along the eight directions on the grid, stopping when I hit a barrier or the edge of the grid.

Wrapping Up

The rest of the project was mainly linking up the UI with the game logic, making sure users couldn’t pull off illegal moves and break the game. I also added some extra features like a board reset, a dialog that displayed the state of the game, and a log of player moves.

I also wanted to try adding AI but realized that it was its own can of worms. Moreover, the possible moves in Amazons is very large in the opening making the AI very slow as I tried to add in a heuristic. Perhaps with the advent of web assembly and more time, I could properly sit down and try to write something that would work and run reasonably fast but that’s for another time. For now, the AI option simply has the agent move randomly until the number of possible moves it can make thins down. At that point it switches over to greedily using the relative territory heuristic that tries to maximize square ownership.

Conclusion

In retrospect, doing this write-up and retracing my steps and thoughts, I feel like this was a project where a few things were done right but many could have been done better. For one, I severely underestimated the number of implementations of Amazons floating out there. I think a decent number of people got the same idea as me after watching the Numberphile video and set out to make their own versions. Additionally, instead of taking the path of least resistance, I ended up installing Windows 3 to play Amazons to begin with and then re-invented the wheel to make my own JS version. The journey itself was fun and I frankly learned a lot from simply finishing this project, but it’s also something that shows I should be more aware of my own naivities and perhaps do better reconnaissance before starting things.

On the other hand, this project was my gateway into a larger browser JS project along with learning more of the language and its accompanying resources. Amazons itself often served as a small reprieve from school. I would often boot-up my VM and play Keller’s implementation with the AI or friends.

Currently, the project is live here to play around with.

Appendix A: Fun with Windows

For some reason I was really intent on getting Michael Keller’s Game of the Amazons implementation up and running. The oldest Windows system I had ever used was XP so anything older was truly a mystery. Internet searches convinced me Keller’s game was supposed to run on Windows 3 (although it may have been fine on Windows 95).

Windows 3 runs on top of MS-DOS. At first, I got DOSBox but couldn’t figure how to install anything on top of it. The only other way I could think of to get Windows 3 was through a virtual machine. I created a new MS-DOS VM in VMware, but when I went to find images of MS-DOS and Windows 3, I found myself greeted with about 3 to 7 floppy disk images. Vintage. Turns out I can choose to add a floppy drive to my VM as well, so pretending to be from a bygone era, I loaded the images into my virtual floppy drive, switching them out when prompted to to obtain my version of MS-DOS. Then doing the same procedure, I installed Windows 3 on top of it.

How was I going to transfer my copy of Amazons to my VM though? I figured well since the floppy drive worked so well, why not just burn it on an floppy disk image like everything else and load it in? Sure enough it worked. Missing a DLL? Do the same thing and copy it to where it needed to go.

As someone who thought VHS tapes were ancient when I was young, seeing this kind of old tech was a little exciting. Floppy disks are only around half a century old at the time of this writing, so it really makes you think about how fast technology advances. We take a lot for granted these days too. First booting up Windows 3 after installation, I was asked if I wanted to know how a mouse worked. A mouse. Of course everyone knows how a mouse works! But not those who grew up on a terminal right? Windows 3 also had some different widgets on their windows. We’re mostly used to the standard minimize, expand, and exit widgets today but on Windows 3, I was trying to make heads and tails of these minus signs and up and down arrows!

I’m sure older developers are rolling their eyes as they read this, but that’s how I really felt. We treat a lot of technology as “duh” and “it’s always been this way” but the truth is that there were many prototypes and versions that came before our obvious standards today. It’s quite fascinating and also makes me slightly afraid that in the future, I will be one those people complaining how, “Yer new-fangled 3D desktop don’t make no sense! In my day, we had 2D windows” or something like that.

UPDATE 11/15/20: Update with new link to demo

Protobowl VR

Thu, 24 Jan 2019 00:00:00 +0000

Late last year, after a session of my school’s XR club, I had, on a whim, proposed to a friend my grand idea of putting quizbowl in VR. As per usual, my absurd scheme was quickly shot down, and we moved to other points of discussion. Later at the outset of winter break with nothing better to do, I told myself: Why not? and got to work.

Design

The goal of this project was to emulate quizbowl to the best of my ability in VR which evolved into building a VR client for Protobowl (PB). At its base, the program was going to be have the user with a realistic buzzer and several other avatars representing the other players in an immaculate room. The program would properly translate the events that took place in the web client PB room into actions in the 3D scene. As for the platform, I chose to target Oculus Rift and write the project in Unity.

Rewriting and Porting the Protobowl Client

To start off, I had to implement a Protobowl client in C#. I had already written an incomplete Python client a few years ago, so I ended up porting that code and subsituting dependencies with C# libraries and scripts from online. One major problem was working with the JSON responses which were dictionaries in the Python version. While Python’s dynamic typing makes its dictionaries very flexible, it also makes them near impossible to translate directly as dictionaries in a statically typed language. After trying to write a data structure for each type of object in the JSON responses, I ended up just giving up and copying the code for SimpleJSON off the Unity wiki which handled all the JSON parsing. Walking through my old PB client code was a bit hectic but eventually I got it to at least have the question text scrolling in sync with the web client and buzz from inside the Unity app.

Moving to Oculus

On the VR side, I started by getting Unity’s Oculus Integration and putting together a simple scene with an avatar that could pick up objects through Frankenstein-ing prefabs from the provided demo scenes. Then I populated the scene with a table and a buzzer prefab I had modeled earlier to get a sense of how the environment felt. This buzzer in particular was a hassle as I had wanted to give it a cord that reacted to physics. Following some online tutorials and copying code gave varied but ultimately insuficient results. On the verge of giving up, I came across a Unity Answers post where the asker had designed a rope with a line renderer and a chain of character joints. I ended up copying the design by chaining a string of empty gameobjects connected with character joints and then making a line renderer draw lines connecting their positions. Lo and behold it worked. I was afraid that putting it into VR would suddenly break it but it worked better than expected.

The Long Haul of Making UI

Now came the Herculean task of writing UI in VR. The first thing to came to mind was making something similar to the menus in Sword Art Online. In SAO, the menu has a main body of options that can expand outwards. The player can bring up this menu by pointing and swiping down which materializes the menu from top to bottom.

I chose to simplify swipe detection, assuming the user would always summon the menu standing up. Detecting a swipe was done by checking if the user was making a pointing gesture using the touch controllers and then checking if the vector representing the change in position of their hand was both directionally “close enough” to down by dotting the direction unit vector with the down unit vector as well as large enough in magnitude to register as an intentional swipe. If summoned, the menu was spawned in and would follow a recorded animation, allowing it to appear to materialize before the user.

User input was also going to be a hurdle. I was hoping to use a canned VR keyboard prefab, but the Oculus Integration didn’t have one. The only other options were buying one off the asset store for $15 or making my own. I opted for the latter and hacked together a very simple QWERTY keyboard made out of buttons. Additionally, in order to actually use the keyboard, I took a gaze pointer from one of the demo scenes and modified it to work with the player’s right hand. While it was pretty crude and didn’t support input from both hands at the same time, for the most part, the keyboard served its purpose effectively.

I had also naively hoped that the Unity UI elements would all transfer flawlessly into VR, but that was not the case. Chiefly, I needed a scrolling option menu and while Unity’s default UI prefabs has one, the scroll bar was too hard to work with in VR. I ended up having to write my own scroll wheel script which simply implemented several drag handler interfaces that allowed a panel of options to be dragged vertically. The whole panel would then snap to a selected option upon the end of a drag.

Adding Voice Recognition

I had considered adding a voice recognition input option early on, but it readily became apparent that this was going to be the primary method of input as typing in an answer through the VR keyboard was too slow for the allocated answer time that Protobowl gives you. I had planned on using an online service such as voice recognition from Google Cloud or IBM’s Watson Speech to Text but ended up deciding it was too much trouble given that Unity provides access to Window’s voice recognition directly. After putting a simple script together and hooking up the proper settings, voice recognition turned out to work surprisingly smoothly albeit with a few hiccups that persisted into the finished product.

One major problem is that voice recognition would often stop becoming responsive and shutting down the game in Unity afterwards would result in a noticable freeze before stopping. I had suspected this was a problem with starting and stopping the dictation recognizer from the warnings to the console, and while fixing that did resolve most of the hiccups, the voice recognizer is still one of the more finicky parts of the app.

Fixing What’s Broke

While connecting the PB client to the actual VR game, I also had to expand upon the client itself as the Python client was incomplete. Among the many things added were:

Detection of who claimed a buzz
Keeping a list of users for a room
Implementing a countdown bar for various timed events
Keeping an answer and event log
Allowing the local client to let the player keep a buzz after a question goes overtime
Handling prompts
Tracking pause and prompt states

Late into development, I ran into a problem where the client worked perfectly in a personal game room but would randomly desync when testing in the main hsquizbowl room by going into a pausing state which was especially strange since it is impossible to pause in that room. I couldn’t find what was triggering the pause state and ended up monkey patching the code to only register pauses in a room where pause was enabled.

However, while the pause desync was fixed, playing in hsquizbowl still had a problem where upon buzzing, the local client didn’t claim the buzz. I initially thought this was a problem with data being lost because processing during a websocket receive took too long and would get interrupted by the next receive. A high traffic room like this one would have a barrage of server responses, so I thought my theory was certainly plausible and tried to fix this by only adding JSON response strings from the server to a queue during a websocket receive and moving response processing to a separate thread. However, the problem persisted. Strangely enough, when trying to simulate a high traffic room with a bunch of bots in my test room, the client performed flawlessly; somehow there was a phantom problem in the hsquizbowl room…

Frustrated, I eventually gave up to play a little bit in hsquizbowl when something caught my attention: 400+ logged users in the sidebar… Trying to get into the game, I immediately realized my problem: hsquizbowl was lagging beyond belief! A quick check in the traffic monitor told me that the problem was not that there were too many responses but that the server was sending them too slowly, evidently because it was overburdened by keeping track of all of its idle users. In order to detect who claims a buzz, my program needs to wait for the server to send back a response. I had hardcoded the timing for this which had worked in normal rooms but was failing now due to the lag. A messy rewrite of the process with a boolean to keep track of when the client was awaiting a response for who had the buzz fixed the problem. To my surprise, after the fix, my VR client actually ran smoother than Protobowl’s web client in the overburdened room (humble brag lol..).

Avatars, Polish, and Shenanigans

While it would have been possible to play quizbowl with just an event log, the point of doing it in VR would be to have quasi-realistic quizbowl experience. Because of this, I wanted to have the other players in the room represented as avatars. As I’m not very good at art, especially in 3D, simple was going to be better. The avatars are made up of disconnected geometry, similar to Miis. Faces were done by simply swapping textures out, making them expressive while simple to design. Certain animations, such as buzzing, were activated by events in game, but other than that, the avatars just randomly activate sets of recorded animations. The avatars were originally supposed to just be opponents, but I ended up populating rooms with them as background characters to make the spaces feel less lonely.

This project was originally going to be graphically bare, but for some reason, the idea of making a grand main hub wormed its way into my head. My modeling skills are still very amateurish, so I thought this would also be a good opportunity to practice. To design the main hub, I pulled up some pictures of Grand Central Station and started going at it. I also ended up doing the same with the game rooms, turning what were originally empty rooms into outdoor rooftop pavilions at night. This was also a particularly good chance to make some prettier 3D scenes. I ended up lighting most of the scenes with baked lighting from emissive materials which looked quite good. The hub also has a reflection probe which gave the tiled floor a nice, freshly-waxed look.

Going into the endgame, all that was left was doing some polishing. Among the additions were:

Plants were put into the game room to give the space more life
A clock was put on an empty wall in the main hub
Sound effects added for buzzing and typing
A grabbable clipboard to check the answer log in the game room
UI was made to be more round

At this point, it also seemed like playing quizbowl was the least appealing thing to do in this app. Notably, with the answer log clipboard, it is possible to place an object at the end of the board and send it launching with a flick of the wrist at no cost the user as the torque exhibited by the object in-game is obviously not felt in real life. Sending the buzzer launching was particularly amusing as it would mess with the character joints, causing the cord to spaz out.

TL;DR Demo Video

After Thoughts

All in all, this was a very satisfying project and also my first large project with VR. It is still not a perfect product, but I feel it is one of the more polished things I’ve managed to push out in Unity. Practically speaking, Protobowl VR is sort of useless as it’s easier to get a bunch of mates together, pull up a packet, and play quizbowl in real life. VR lends itself to be particularly useful for hard to emulate scenarios like gunning down robots on a battlefield or flying around the Earth, but, nevertheless, I am happy that I somehow managed to implement some crazy idea I spat out one day.

Additionally, I may be biased in this regard since I wrote it, but there is now a functional Unity PB client in C# floating around on Github. Perhaps now is the time to bring back Protobowl mobile, but I digress. I imagine there is a reason why the old mobile client for iOS at least has disappeared from the app store. However, with Unity, the platform indpendence doesn’t really end. If someone were crazy enough, we could even get Protobowl for 3DS or something!

That being said, I also think I’ve done way too many quizbowl and websocket related projects. I may come back to websockets in the future as async is very useful for games, but as someone whose “quizbowl career” peaked (and that’s a low peak mind you!) in high school and has never even gone back to play since then in real life, perhaps I am hanging on to the past with these programs. My hope is to move on to a new area and stop doing quizbowl stuff for a while at least.

Check out the code here or if you have an Oculus, download the build from Itch here.

Bubbles

Fri, 14 Dec 2018 00:00:00 +0000

Among many things brought out into the cold of December 1st, 2018 were four nerds with nothing better to do on a Saturday morning than participate in their school’s local hack day. While I cannot speak for the experiences of my companions, I do recall being awoken at 2 in the morning earlier that day by the finest drunks on our floor. In my drowsy state of quasi-conciousness, I came to a sudden premonition:

Beatsaber. We should make 2D Beatsaber with OpenCV.

A lackluster slumber followed, leading to the events of the day proper. The hackathon itself was comparably short, clocking in at 12 hours, meaning we could not go overboard with crazy ideas. While we had initially proposed making a motion-responsive color picker, this was deemed too simple as it could have been easily accomplished by hacking together some scripts in Unity and exporting to a phone with a gyroscope. When 2D Beatsaber was brought up, there seemed to at least be some amount of consensus on the project, and so we took it and ran.

The following account reflects my own experiences and thus will focus a considerable amount on the code side of the project. The work of my teammates was pivotal to the completion of the hack, but I am afraid I would not be able to give anything more than a superficial description of their workflows.

Design

The general gist of our hack was to detect players using a webcam and overlay them into a rhythm game where they would be able to reach out and pop bubbles with their bodies in sync to music. In that sense, the program was more similar to Fruit Ninja on Xbox Kinect than Beatsaber, but the old name stuck. From the outset, we also chose to use OpenCV for object detection as it is relatively straight-forward to use. The game itself was split into a game manager to handle logic and a renderer to generate images from game data, both operating within a game loop.

Process

The first hurdle was, of course, figuring out how to detect our player. As training any sort of algorithm to detect features was out of our time scope, we were left with two options:

Use broad and traditional computer vision tactics to detect the player
Steal object detection models from somewhere else

A brief search for OpenCV classifiers to detect hands gave us nothing, so we opted for the former option. In order to single out the players, the image was first background subtracted and then contour detection was run on it which left us with a reasonable player outline given that there was not too much movement behind the user.

Once we extracted all the points representing the player from finding the contour, it was a matter of sending this data to be processed by the game. The design was fairly tame with a central manager that kept track of, among many things, points representing the player and the locations of the bubbles. The game manager was also passed into a renderer that used PIL to stitch together an output image based on game data which was then sent out to be displayed. Music remained difficult to incorporate and had to be hardcoded in with stolen internet code, separate from the main models. Later on, a crude implementation of animated sprites was also added that simply gave each bubble a sprite index that incremented over time.

Among the many challenges faced in that period were:

Embarrassingly poor use of git. We will branch and merge next time…
Difficulties incorporating sound. This ended up being messily written and running on a separate thread.
Problems synching bubbles with music. The game was supposed to be 4 minutes long. It is now 8 minutes long.
A slew of image-related problems including troubles with color-encoding, transparency, flipped images, and resizing
Issues optimizing collision detection

Specifically regarding collision detection, collision with a bubble is done by checking if a player-representing point is contained within a radius of the bubble. While an exhaustive search initially worked with a small set of bubbles, the game quickly started to lag after scaling up. A simple solution was made, limiting only bubbles present on screen to be collidable, and while this greatly reduced lag, it did not entirely fix the game from stuttering. Potentially, in addition to only considering on-screen bubbles, a smarter solution may have been to bucket the player points by a position range to lower the search space for each detection.

After Thoughts

Nearing the end of the hackathon, we integrated proper art assets and a completed beatmap into the game, ending up with something half-way decent. While admittedly some parts of the code were cheesed (cough strategic use of eval cough), the program as a whole was moreorless functional and worked to our satisfaction. Despite this, there was still much that could be done with this project. For one, the game would probably run nicer had it been written in a more optimized language like C as Python is very slow. It would also have been nice to have pluggable songs and beatmaps as well as a proper UI since the game currently goes directly to playing our one hardcoded song out of the box.

In any case, in spite of its shortcomings, at the end of the day, the program did win us some socks so no complaints from me! We walked off that night, feeling chipper than ever all whilst spiriting away an orphaned liter of Diet Pepsi and a stack of cups from the catering.

Acknowledgements

Thanks to my friends who spent all Saturday building this hack with me
The song used in-project was BUBBLES by Tokyo Machine

The repo for the project is available here at the time of writing.

Kuiperbowl

Mon, 26 Nov 2018 00:00:00 +0000

As of the time of writing, Kuiperbowl is live here!

Another old idea that had been swimming in my head for a few years was creating a Protobowl clone. Since I owe the more impressive parts of my largely unimpressive trivia knowledge base to rote practice on Protobowl, it has always been a small dream of mine to make something similar in functionality (and bot the site lol).

Design

Roughly, Kuiperbowl is made up of a Javascript client with a websocket that pings a main game server to stay in sync as well as requesting information and commands. The information stored is comprised of four models:

Question: Toss-up quizbowl question
Room: Describes a room. Made of players, messages, a running question and keeps track of actions taken in the room.
Player: Describes a player. Keeps track of points and buzzes.
Message: Timestamped messages with a purpose and content.

The bulk of the project server-side is actually just a giant if-else that responds with the proper requests. Surprisingly, I feel the client’s design took more thought as aside from coordinating UI elements, the client also had to be kept in sync with the server.

Process

I knew I wanted to make the site in Django using Channels to handle websockets since I had built a real-time chatroom following tutorials doing the same thing. The first problem I ran into was when I tried to re-serve the old chatroom project, however, I suddenly ran into errors. Turns out I had updated my version of Django which was not compatible with Channels anymore. Switching gears, I scoured the internet and finally found a setup that worked, re-configured the chatroom project, and got everything back up and running smoothly as before.

Now that I had an async setup that actually worked and some database models, it was time to start writing crappy Javascript! The client itself has a looping timed update that locally keeps track of the room state. It also pings the server every 5 seconds to keep in sync. For actions that require immediate synchronization, the client sends a websocket command to the server which then sends out a group update at that instant instead of relying on the 5 second sync to update the other clients.

One of the biggest challenges was figuring out how buzzing in for an answer would work. The first implementation would designate a buzzed player who would be contending the question and lock out the room for everyone else until the contender answered the question. However, this did not account for the fact that time would still pass while the buzz was taking place; the problem remained of how to coordinate buzzing among all the clients. I ended up drawing a little diagram and came up with what I affectionately call the “King Crimson” approach, drawing inspiration from JoJo’s Bizzare Adventure:

Essentially, upon a buzz, a buzzing start time is recorded. The contesting player spends some amount of time on the buzz and when the buzz is done, the duration of the buzz is calculated and both the start and end times are moved by forward by that duration. This effectively erases any time the buzz took up from the question reading time, hence the name, King Crimson approach (Additionally, this also serves as a nice explanation as to how King Crimson works I feel…).

Afterwards, it was a matter of sprucing the client up so that the local and client game states stayed in sync. One issue in particular was that while buzzing near the end of a question, the question would prematurely end, interrupting the buzz. At first, I thought it was a flaw in the King Crimson solution itself, but it turned out to be another coordination issue. As the time erasure doesn’t occur server-side until the contesting player finishes buzzing, it’s possible for other clients to declare the question done before the player finishes answering. Thankfully, I had another bug where I was checking for the end of the question even while the player was in a contesting mode, making the first bug easy to find. The solution, of course, was to fix both the game flow of the client by preventing premature question ending as well as write in constraints for the server that would prevent ending the question while in contested mode. For the server, pings would also trigger room updates, making sure that in addition with syncing the clients, the rooms were also all up-to-date.

Figuring out the King Crimson approach remained the most challenging design aspect of the project. Most of the rest of the work was adding small features like theming and messages as well as cleaning up client-side bugs, such as one that didn’t unlock player buzzes after going to the next question. A good deal of the project was, unfortunately, front-end design which boiled down to a small eternity of learning how to style and align HTML elements until things looked nice…

While the project could have ended with the implementation, I figured since I had some AWS credits, I would go the full mile and move the project into production. This was honestly a small hell in itself. Part of the problem was figuring out production in async Django as using the default server is poor practice. After some time, I managed to get the project running with Daphne, but without the default server, I could not serve any static files which is where all my client code was. The patchwork jank solution was to open up an S3 bucket and pull files from there but updating the bucket slowly became a real pain. So through some trial and error and after accidently deleting a bunch of files for Apache, I came to fathom how to use Nginx. Nginx was configured to serve static files as usual but used a reverse proxy to Daphne for HTTP requests. The last stretch was linking up a domain name and SSL certificate. The domain name linking went smoothly as I had used Route 53 before, but the SSL certificate was a hassle. Eventually I figured out I could generate certificates for multiple domain names and then just had Nginx redirect HTTP to HTTPS.

After Thoughts

Once again, like most of my projects. The implementation is not the cleanest it could have been. The most glaring monstrosities in this project are the huge if-else that comprises the server and the spaghetti code that coordinates game states for the client. Nevertheless, functionally, it works, and I’m quite happy that I managed to make something this close to Protobowl. This was another project that I had started a while back but never finished because I couldn’t figure out how to design it. I had gone as far as dumping a bunch of quizbowl questions and making a question reading client back then but coordinating multiplayer was something I could not figure out. I had wanted a server-centric system that would have the server keeping time and updating the clients continuously which was frankly not possible. Looking back again, this time, I came up with a much more client-centric approach where a lot of the updates take place locally and the server is only talked to as necessary for consensus which I feel is a better design as a lot more work is off-loaded to the client.

While an extension of this project is to move it into other mediums, possibly making mobile clients or maybe even some sort of quizbowl VR app, I do want to step away from this project for a while. I felt like I got a lot of experience doing some web work for this project (mostly the production configurations…). This also is still an inferior clone of Protobowl so I had also thought of including some additional features like global rankings with automated tournaments. One colleague even suggested some sort of card game-trivia bowl hybrid. I believe Protobowl also operates in a similar manner to my design so perhaps this might also be setup for writing a mobile app to finally interface with Protobowl once again…

…but that’s for another time!

Thank you for reading and hopefully I remember to update the SSL certificate for Kuiperbowl next month…

CHIP-8

Sun, 04 Nov 2018 00:00:00 +0000

I haven’t had fish ‘n chips in a while but that’s beside the point. Way back, I started a CHIP-8 emulator project in Java and then dropped it soon after I realized my lack of experience with low-level programming. However, since delving into NESASM, I decided this would be prime time to pick CHIP-8 back up again. I also chose to make it in C++ this time since, prior, I had nearly zero experience with low level programming languages (aside from 6502 assembly), and I also wanted to get some more experience with SDL.

Overview

There are many good guides out there detailing the CHIP-8 instruction set, but, roughly speaking, the CHIP-8 has 4096 bytes of memory, 16 general purpose VX registers, an I register for holding memory locations, two registers for delay and sound, and a call stack that stores address locations for subroutines.

Additionally, the emulator had to implement a program counter (PC) as well as arrays for holding graphics data and keyboard presses.

The emulator naturally has to read in a rom and then correctly execute the proper opcodes. This one uses SDL for displaying graphics and taking in input.

Process and Design

Much of the project was kickstarted thanks in large part to many guides detailing the process of building a CHIP-8 emulator. The CHIP-8 CPU itself simply initializes and indefinitely runs cycles. Each iteration gets and executes the opcode where PC is currently at and then moves PC 2 bytes forward. The reason for the 2 byte step is that the opcodes are 16 bits and thus require 2 bytes for each one.

In code, the opcodes were stiched together with bitwise operations and then sent through a chain of if statements that would determine the command, parse the arguments, and execute the proper functionality.

Afterwards, the process of building the emulator was relatively straight-forward as it was largely translating the functionality described online into code. Perhaps the most troublesome opcodes were the draw command and subroutine call.

Roughly, the draw command takes in a coordinate and height as arguments and, starting from the address in the I register, draws the bit representation of the data in memory as a row by XOR-ing the bits to the screen and then continues to move down in both memory and rows until it has drawn to the proper height. While decoding drawing the bits into the graphics array was enough of a pain as it was, the draw command also needs to take into account sprites that overflow off the edge of the screen. The standard way of dealing with overflow is to wrap-around both vertically and horizontally, but some emulators (and roms) don’t account for this. Moreover, the CHIP-8 also detects collision by setting VF if a draw command changes a pixel from 1 to 0. For the longest time, many roms were broken simply because the emulator was doing collision detection wrong as it had been both only detecting a collision on the last pixel drawn as well as detecting the wrong condition for collision since I had assumed I had already XOR-ed the two values.

Meanwhile, the subroutine call was a hassle, not because it was difficult per se, but rather due to my lack of experience with C++, the difficulty of tracing multiple subroutines, and even discovering that the subroutine call was buggy to begin with. In general, subroutines work by taking the current address location pointed to by PC, pushing it to the call stack, and jumping to another memory location. When a subroutine return opcode is called, the PC is set to the address at the top of the stack and the address is itself popped off. Early on, I would get memory exception errors for certain roms, not knowing where they came from. Eventually, on a whim, I tried to pop from an empty stack, and, lo and behold, there was that error. Turns out bugs in other parts of the emulator were messing up program flow and would sometimes call a subroutine return without having called a subroutine before. However, even within a subroutine, it seemed like returning would not jump back to the pushed address. This time, the problem was a conceptual one as I had been pushing the address locations into the stack as bytes and not shorts, resulting in the first byte of the address being cut off.

While these two were the biggest hurdles, some other run-ins included:

Problems with casting to unsigned char
Making SDL draw faster
Confusing indices with stored values in reg load and dump
Skipping instructions when jumping to address locations due to incrementing PC

In retrospect, the SDL drawing component was not as bad as I had originally anticipated (probably since I took most of it from online lol). However, as is the opinion of many I believe, sound in SDL is horrific. I had planned to use the sound tutorial in Lazy Foo but after finding out that I had to get more libraries linked properly, I promptly gave up and took code to run WAV files from elsewhere.

I had also considered implementing the SCHIP-48 instruction set from the outset but was not sure how easily it could be done. For one, documentation on SCHIP is more sparse than CHIP-8 which made it slightly more difficult. In the end, however, SCHIP-48 was only a hop, a skip, and some shady monkey-patching away, and I managed to get the few new opcodes written relatively painlessly while preserving compatibility with CHIP-8 roms.

After Thoughts

In reality, this project is not the best-written CHIP-8 emulator out there (especially considering that this is a common beginner’s project…). Most difficulties were due to being unfamiliar with SDL and C++ going in as well as an overall lack of understanding with Visual Studio. However, I am also quite proud of this project as I had long dreamed of making an emulator since my middle school days looking up incomplete NES emulator guides. I had given up on this project 2 years ago when I started it in Java, and I guess it means a lot now that I managed to pick it up and actually make it work.

Plus, I got to play a bunch of games while making it!

Acknowledgements

This project was a hurdle for me and I got a lot of help out of these resources (also on project readme):

Roms for testing retrieved from Zophar’s Domain
Made with heavy reference from to instruction sets on Wikipedia and Cowgod
multigesture.net blog post for starting tips
DZone for SDL pixel drawing code
armornick’s SDL sound code

CMSC Undergrad Calendar

Thu, 27 Sep 2018 00:00:00 +0000

Going into college, one of my primary goals was to get my act together. I’ve always been awful at managing events, so recently I’ve started trying to actively use Google Calendar. Integrating it into my livelihood was very straight-forward:

Add calendars for bulk events
Manually add smaller, one-off events

Then I happened upon the behemoth that is the CMSC Undergrad calendar. Not only could I not find a Google Calendar mirror, but calendars for subsequent months seemed to be getting pulled via some ajax magic. I wanted to just scrape the entire thing but the whole site was so arcane to navigate that it was much more trouble than it was worth.

I was about to give up and just manually add all 50+ some events when the words of an old friend came back to me:

“Webscraping? Pfft that’s easy.”

In a sense, he was right because while trying to hack together a script and figuring out what parameters to shove into a POST request takes a lot of time (at least for me anyways…), there are smarter ways to do webscraping.

The solution I turned to was Selenium, mostly because this project gave me an excuse to start messing around with it.

Process

One of the first things I did was figure out how to get around with Selenium. Once I could successfully navigate through the calendars via button click and pull the source, the project became mostly a parsing game. I threw all the HTML into BeautifulSoup and looked for event hyperlinks which essentially just got me a list of all the files under “event” directory. I then went through every event page and did the same thing, this time trying to pull out the event title, location, and date if I could and processing it into a reasonable form.

Now I had all the event data I needed but still no way to put it into my Google Calendar. Thankfully, Google provides a nice API to their service plus some base example code which I copied and modified. Running my little script, I now had a handy way of mirroring the undergrad calendar over to Google except for one small problem. For some reason, the times were off by an hour starting from November. How curious! I thought I had messed up some timezone issue, but then I remembered: Daylight saving time. Ugh. Google Calendar was automatically accounting for DST for some reason. To be honest, I just monkey-patched the code to change the time offset based on whether DST was in place or not.

After Thoughts

This project was not that complicated, and I’m sure there were better ways to do what I had set out to accomplish. For all I know, I may have just entirely missed an “Add to Google Calendar” button on the undergrad site which would have saved me a lot of trouble.

One of the end goals was to put this script onto a server and run it every month to update the list of events, but I had some trouble getting Google to authenticate me on my EC2 instance and decided that this was enough since I could run the script monthly myself anyways.

As for its use, many big events in the computer science department get hyped up days before they happen anyways, making this thing largely useless. That being said, the calendar still has some use for reminding me of smaller events and if anything, I learned how to prototype webscrapers faster with it through the Selenium experience.

The calendar itself can be accessed here although it is likely not up-to-date.

Spooter

Wed, 08 Aug 2018 00:00:00 +0000

Fresh out of the Nerdy Nights Pong tutorial, I had vowed never to write any assembly for the NES ever again.

As you can guess, that didn’t happen.

Background

While on vacation visiting my grandparents, I found out about a game jam called Floppy Jam. The gimmick of the jam was that all entries must theoretically fit on the 1.44MB alloted to a floppy disk, hence the name of the jam. Naturally, this meant that most orthodox game engines would not meet the cut as they create too much overhead.

This also meant that I was down to only a few options if I chose to participate:

Finally learn OpenGL and pray that I can write something in C/C++
Hack a thing together with various JS libraries and a canvas
Go back to NESASM hell

I might also mention at this point that my grandparents have limited access to the Internet, and being on vacation, I wanted to see sights, not screens. So, with a few cached NESDev wiki pages and the 6502 instruction manual in hand, I jumped back into the fray of writing really bad assembly code.

Process

For the longest time, I had dreamed of making an even shittier version of Hong Kong 97, a bootleg SNES game starring Jackie Chan vs China. While this was the impetus for the project, Spooter very quickly morphed into a traditional space shooter.

Starting off, I copied the Pong basecode I had written following Nerdy Nights last year, leaving a lot of the hardware shenanigans intact while deleting just enough Pong functionality for the assembly to still compile and run. After fiddling a little bit more with the code, I managed to get a space ship sprite to load and went from there.

One of the greatest hurdles in designing Spooter was controlling groups of objects (lasers, aliens, explosions). Initially, I thought this could be done naively by hardcoding the functionality of every object individually. However, as I started adding functionality for the aliens, this readily became infeasable. As you might expect, since every alien needed to be checked against every laser, I would be re-writing upwards of 12 blocks of code. This kinda ticked me off. I had avoided anything too complicated before this, but now I braced myself for the inevitable and wrote what may be called in the most primitive sense, a “nested for-loop”. After somehow successfully doing that, I started to get into the groove of things and began navigating and building a jungle of jumps, loops, and register juggling, even working in a system of activating and deactivating groups of objects with a bitmask where the bits in a byte would dictate active members of a group (so a bitmask of 00001001 would mean that the 4th and 1st object were active).

By far, the largest time sink with NESASM was fixing bugs. Often this was a slip-up like forgetting to load newly-added sprites or writing to the wrong area of memory. Other times, bugs were a result of inexperience with writing for the system such as not knowing to turn off NMI before loading a new BG.

A few of these many endless gripes include:

Labels that would occasionally get out of range. This was solved by hacking sections of code out and putting them into subroutines which worked somehow…
More than enough bugs arose simply from bad math in hex and writing to the wrong addresses
Nine-out-of-ten, a bug somehow got fixed by clearing/setting the carry bit
For the longest time, I thought I was bit-shifting right when I was really going left
The score is an 8-bit value and, as such, will overflow once the player gets a score of 256. This will remain a feature that I now dub, “Spaceship Reincarnation”
The game over screen is made of BG tiles and flickers since the only working method I have of writing to the background is turning off NMI, writing to an address, and turning it back on…

Here is the finished game. The final product ended up being a relatively simple shoot ‘em up where enemy floppies would float down, get shot, and respawn ad infinitum.

After Thoughts

Several goals remain unaccomplished for this project.

For one, I had wanted to add music and sound-effects into the game but lack of experience with the APU and time made this a hard endeavor. Another early dream was also to parallel the boss of Hong Kong 97 with a large boss made of BG tiles. This was also not possible due to a lack of experience.

The box art was also drawn up in MSPaint in the literal hour immediately after the game’s completion and leaves much to be desired.

Here it is in all its glory.

Protobot

Wed, 27 Dec 2017 00:00:00 +0000

Winter break’s here, so I decided to spend some time off making a way to programmatically interface with Protobowl

The main goal of this was to prototype and then port code to Swift as iOS 10 has broken mobile Protobowl. Also out of some morbid desire to beat my quizbowl teammates at their game, an auto-answer bot was developed alongside API code.

Of course, I’ve already tried this multiple times in the past with varying levels of success:

The first one: found PB source then gave up
Do it through HTTP: yeaaahhh…
Found ProtoBot in Java: promptly gave up afterwards

I’m a bit pragmatic as you can see, mostly since I wasn’t going to burn a week on this kind of thing-whoops.

From what I understand, PB upgrades HTTP requests to the websocket protocol and then sends JSON files between client and server. I was able to see this using Chrome’s Network tab, but the problem remained: How to upgrade HTTP to websocket?

On a whim, I requested a socket from PB using HTTP and tried plugging the websocket address into a websocket client which…worked. Huh. Cool.

Over the next few days, it was a matter of implementing many PB commands to be sent as JSON. Thankfully, Python’s json library dumps JSON nicely and PB’s not that strict with parsing, so I can just ignore the command headers for the most part.

Sample run of the bot:

Among various minor annoyances and roadblocks include:

PB disconnects idle users, so I need to constantly ping the site
PB sends packets as general “sync” packets which just contain a bunch of info. Of course, these things are a a mess to identify and dig through.
Parsing data received is still incomplete. While unnecessary for an auto-answer bot, I want it for a future client.

The plan as of now is to focus on porting to Swift and then figure out how to best parse incoming websocket data. Xcode grievances are worth another day though.

Acknowledgements

Much of this project owes itself to ProtoBot which was a huge help in reference.