Published: August 5, 2025
Summary
This blog post documents my full process and learnings while fine-tuning a GPT-2 Medium language model from scratch using the Alpaca instruction dataset.
It includes:
- My own dataloader implementation built to work with a from-scratch GPT-2 model following Andrej Karpathy’s Zero-to-Hero series.
- Training a dataset of ~51,000 samples on an NVIDIA A100 for 2 epochs, reaching a validation loss of about 1.7.
- Insights about how I handled special tokens, padding, and attention masking.
- A walkthrough of resizing GPT-2 model layers to accommodate new tokens.
- Example generations before and after fine-tuning to highlight the transformation from base model to assistant-style responses.
All code is available here.
Introduction
The goal of this project was to build and fine-tune a GPT-2 Medium model (355M parameters) from scratch, using a curated dataset with instruction-following examples. I was specifically interested in exploring supervised fine-tuning (SFT) — a key …
Originally published June 2015
The majority of my career has been spent building software for marketers to track you online. More often than not, it’s so that we can know everything about you to sell a product to you as effectively as possible.
For the past few months I've been using Ghostery, a Google Chrome extension that identifies third-party trackers, and I noticed the extent at which sites use tracking companies to measure everything about my visit. Things like who I am, my last visit, my computer, where I’m located, and much more are distributed via many, many tracker URLs (Huffington Post has over 160, as I’ll show) to different companies to store and analyze my information.
I became curious about how often major sites and brands measure our web visits, and wrote a program to measure it.
The program does the following:
Originally published in July 2014
There has been recent buzz about a Google recruiting method called Foobar. Supposedly if you search enough programming-related issues like "python mutex lock," your account is flagged and you're offered to play a game with a series of programming challenges. If you do well, people claim you're invited for an interview.
Well, I wasn't offered to play this game. Thankfully, my friend was, and he shared one of the challenges below:
Peculiar balance
Can we save them? Beta Rabbit is trying to break into a lab that contains the only known zombie cure - but there's an obstacle. The door will only open if a challenge is solved correctly. The future of the zombified rabbit population is at stake, so Beta reads the challenge:
There is a scale with an object on the left-hand side, whose mass is given in some number of units. Predictably, the …
Originally published June 2014
I'm not very good at internal communication. As a product manager, this should be one of my strong suits: refine technical issues into clear points and spread these to sales, marketing, and service teams.
It's not that I'm incapable of doing this, it's that I just don't enjoy it very much. I also find that this can become an area that demands the greatest time, as opposed to working directly on making the product better.
Once the startup company I was working at was acquired by Oracle, this problem became pronounced. If you’ve worked at a large company before, you quickly realize a portion of your success is tied to your ability to communicate through email. There are teams spanning multiple departments, each dispersed across different geographies and timezones and all expecting information to be in a system that is unique to their teams. These …
Originally published June 2014
I wrote a quick Node app to scrape submitted challenges from Coderbyte and to save them locally. Find it here: https://github.com/rparang/scrape_coderbyte
If you've spent time on challenges through the browser and realized it may be nice to have these saved for other purposes, this may be helpful.
Keep it mind, since this scrapes the site by parsing the returned HTML and doesn't use a dedicated Coderbyte API, it may not be stable.
See here for the solutions to the challenges in Javascript (pulled using the Scrape Coderbyte app) that I've managed to solve. There are few remaining in "Medium Difficulty" and all in the hard category as of the time of this writing.
Completed Challenges:
- AB_Check.js
- Alphabet_Soup.js
- Arith_Geo.js
- Arith_Geo_II.js
- Array_Addition.js
- Array_Addition_I.js
- Binary_Converter.js
- Bracket_Matcher.js
- Caesar_Cipher.js
- Check_Nums.js
- Consecutive.js
- Counting_Minutes.js
- Counting_Minutes_I.js
- Dash_Insert …
Originally published June 2014
Recently I've been attempting the programming challenges on Coderbyte. Coderbyte is a site that has a handful of questions that allots points given how 1) fast you complete the challenge and 2) how correct your answer is.
Once you solve a challenge correctly, it shows you solutions from other users, which might be its most addicting part. Typically I'll either scoff at someone's approach or look at their answer in amazement.
For one challenge though, I wasn't sure.
The challenge is:
Have the function BracketMatcher(str) take the str parameter being passed and return 1 if the brackets are correctly matched and each one is accounted for. Otherwise return 0. For example: if str is "(hello (world))", then the output should be 1, but if str is "((hello (world))" then the output should be 0 because the brackets do not correctly match up. Only "(" and ")" will …
Originally published May 2014
Paint Hip Hop is a hip hop music app. It was the first application I built and was influential in terms of my experience with web development and my confidence in front of code.
I’m telling you, I'm as cold as windows down in the winter
And I be riding rims if my tires any thinner
Airport stunting, flying charters overseas
Full of Dom Perignon and the water for the D's
A user shares a song which builds playlists for those who follow that user. It uses a basic Twitter follow model.
Popular tracks earn users rep points (check the shades). Rep points are achieved by people liking the shared song.
When Paint Hip Hop launched in mid 2012, it had a fair bit of traffic (at least what I was expecting for a side project) that reached roughly 1,000 unique visitors for the …
Originally published May 2014
Old School Kool is a collection of beautiful photos from the past. The images come from Reddit's Old School Cool subreddit, a community described as "a pictorial and video celebration of history's coolest kids, everything from beatniks to bikers, mods to rude boys, hippies to ravers. And everything in between."
I built Old School Kool after running into an app that claimed to have a perfect image layout: an attempt at using 100% of the browser’s viewport to display a series of images without cropping or modifying their proportions.
It's common among popular photo applications like Facebook and Twitter to have photos cropped into a square to show an organized list view. Often, as a result, much of the context of the image is removed (like your face, for example).
The folks at Crispy Mountain, a web development firm, had open-sourced their attempt at the …