Ubuntu installer slideshow customization, with pictures!

I have escaped the perilous grasp of Perl, and I bring news:

If you make a localized Ubuntu ISO like the Italian CD, you can customize the installer slideshow, with translated screenshots and more!

How? Follow me!

First, you need to be running Ubuntu 12.10 with the latest version of the Ubuntu defaults builder — 0.35. I assume you already know about the defaults builder, since you are making a localized ISO. If you do not, start with this explanation about localized images. Don’t worry: the defaults builder is a great tool that makes this very easy!

The defaults builder template has been updated to include a “ubiquity-slideshow” folder. Let’s open a terminal and create a new defaults package to play around:

ubuntu-defaults-template ubuntuway-defaults

That will create a fresh defaults source package in a folder named “ubuntuway-defaults”. Open that, then open “ubiquity-slideshow”.

Of course, you can always add ubiquity-slideshow to an existing defaults package.

As you might have guessed, you get to customize the slides! Open “slides.”

Yay, an empty screen, glimmering with potential!

We put slides in this folder, in almost exactly the way they are organized with ubiquity-slideshow itself. Now would be a good time to install ubiquity-slideshow-ubuntu so you can use the base files as reference. Make sure you have at least version 61, because things have changed since Ubuntu 12.04.

With the defaults package, you get to extend /usr/share/ubiquity-slideshow/slides/l10n.

In the ubiquity-slideshow package, each folder contains slides for a particular locale. Note that each folders’ contents look a lot like /usr/share/ubiquity-slideshow/slides.

Back in our defaults package, let’s start by adding some locales. We are going to add translated text and screenshots for en@piglatin. To start, we create a new folder: “en@piglatin.”

On this topic, there should actually be a pig latin option in the locale chooser. It would be sooo cool.

The way this works is the slideshow will search the extra locale folder for a language and, if a particular file is available for that language, it will use it instead of the default. For example, we could replace welcome.jpg for the “it” locale, so when someone installs in Italian that picture will appear instead of the original welcome.jpg.

Let’s replace welcome.jpg for the en@piglatin locale! To replace a file, the easiest thing is to copy where it is in /usr/share/ubiquity-slideshow/slides. You can see there’s a screenshots folder there, with a bunch of screenshots. So, in our own “en@piglatin” folder, we’ll add a screenshots folder. Of course, what you would normally do is look at the original welcome.jpg and create something similar that works for your locale and your disk image, making sure to keep the same dimensions. I’m going to cheat a little.

Custom screenshots folder for en@piglatin, with a custom (colour-shifted) welcome.jpg

One problem with the slideshow is that the screenshots are always in English. Now, you can include translated screenshots. As with anything else here, these go on top of the other files. An Ubuntu install in your language will show the translated screenshots, while an install in English (or any other language) will show the original English screenshots. There isn’t an actual Pig Latin Ubuntu for some reason, so I will translate a screenshot into French for an example.

Some rules of thumb for taking screenshots, if you don’t want to (or it isn’t practical to) do the same thing as the original:

  • Do something practical, like search for a photo or read a web page: don’t open every menu in an application to demonstrate its features.
  • You probably don’t want to have all your own stuff there for the world to see, but if you have some time try to invent a person. It’s just a little more fun than showing a blank photo manager with “Guest” at the top right of the screen.
  • Use the defaults! Create a fresh user account and make sure you don’t touch the font size or the background picture.
  • Make it look good, and if following the above rules makes it bad then please disregard them :)

I went ahead and created a translated screenshot to replace usc.jpg. I took a screenshot of the entire screen, then I opened it in GIMP and scaled to 60% with interpolation set to “Sinc (Lanczos3)”. (You are welcome to choose whatever interpolation works well for the job at hand, of course). After scaling, I used the crop tool, setting a fixed size of 448×304 pixels and leaving a 10×10 border around the application’s main window. (Just line up the box from the crop tool with the top left of the window, then add “-10” to each of the Position number fields).

Cropping a screenshot of Ubuntu Software Centre in French. Yes, it’s also easy to subtract 10 in your head, but OMG cool GIMP feature!

I saved the picture as usc.jpg and added it to the screenshots folder, right beside welcome.jpg. You may add screenshots for as many languages as you want, as long as they fit on the CD, but please keep in mind there is a high maintenance cost.

Now we can try this thing! Open a terminal: we’re going to build a defaults package and see what happens.

cd ubuntuway-defaults
debuild

A package file should appear in the parent directory, called ubuntuway-defaults_0.1_all.deb. Just double click the file and choose Install. To see this in action, you also need to install ubiquity-slideshow-ubuntu and download the slideshow testing tool (Slideshow.py). Sorry, just a little more terminal stuff ahead.

Open a terminal pointed to wherever you downloaded Slideshow.py.

chmod +x Slideshow.py 
./Slideshow.py --path=/usr/share/ubiquity-slideshow --locale='en@piglatin'
Psychedelic!
Translated!

Now, about that text…

We’re going to add some information that we know will be useful for people who receive this disk image, and we will translate some stuff into Pig Latin as an example. The files we need to replace are called welcome.html and gethelp.html, and they’re html files, so if you know HTML it will be a piece of cake.

Again, the easiest thing is to find the files we need from the base ubiquity-slideshow and copy them. In this case, we’ll copy and paste the files and then edit them with our changes. But before you do that, remember that the slideshow is very likely to be available in your language. Before you copy slides from /usr/share/ubiquity-slideshow/slides, look for them under your locale at /usr/share/ubiquity-slideshow/slides/l10n.

With this example, there is (of course) no Pig Latin translation to base on, so we will just add some new stuff on top of the English slideshow. In theory, we could actually translate all the slides to Pig Latin here, but that would make a very poor example because it shouldn’t be necessary: the best practice for this part is to do translations in the ubiquity-slideshow project, and then just tweak those translations in the defaults package.

welcome.html and gethelp.html copied from the base locale at /usr/share/ubiquity-slideshow/slides

Open welcome.html in your favourite text editor, look for the content, and do what you want. Be careful, though: a syntax error can cause all sorts of problems.

Writing some helpful targeted content for gethelp.html

To see the changes we need to build the package again.

cd ubuntuway-defaults
debuild
sudo dpkg -i ../ubuntuway-defaults_0.1_all.deb
./Slideshow.py --path=/usr/share/ubiquity-slideshow --locale='en@piglatin'
The installer slideshow with our localized gethelp.html

Now that we have the defaults package figured out, we can go one step further: build a disk image!

mkdir image-builder-tmp
cd image-builder-tmp
ubuntu-defaults-image --package=../ubuntuway-defaults_0.1_all.deb

This will take a while. When it’s done, it will spit out a file called livecd.ubuntu.iso. Just load that in VirtualBox or something of the sort. When you install, it should look exactly like the preview. (Assuming you use a language that can be selected in Ubiquity, and that language has been selected).

I really upset the live CD trying to make the Pig Latin example show up, so that’s why there are no window titles, but it works, right?

That should get you started. I hope you like it! Go ahead and download the ubuntuway-defaults source package to explore, and please share how it goes.

Adventure and excitement at UDS-Q!

I’m home from UDS-Q in lovely Oakland, California! Okay, I actually got home about four days ago, but I’m slow with these things. It was an interesting week with lots of interesting people. I’ll try to cram it into this post…

The Ubuntu Ecosystem (with a bonus secret session)

I attended a strange session on Monday, called The Ubuntu Ecosystem Part 1 (of 2). We don’t know who scheduled it, or why, or what we were meant to talk about, but it had an alluring title and the room was packed with people. Eventually, Jono stepped in to lead the session and it actually turned into a really nice chat about how Ubuntu is doing as a platform for developers outside the project itself. The session was removed from the schedule, but I think the session’s notes are interesting anyway. It was a good lead-in: app developers were a big topic for the whole week.

There was another session on Tuesday morning: App Developer external outreach. This one was quite interesting. Many different points of view emerged. When we talk about app developers, we tend to be preaching to the choir: we’re often grabbing people who already happen to be making, or interested in making, Linux desktop apps in general. I think Stuart Langridge said it best here: if there is a Linux desktop app that is incompatible with Ubuntu, it’s because they don’t like us. He also challenged the people in the room to name a single Linux app that doesn’t run on Ubuntu already. (I think Stuart is brilliant, by the way). I know there are a few, but what’s in the way is more packaging mumbo jumbo than the software itself. There are many branches to this. Getting things compatible, and getting things packaged and in Software Centre, are two or three different discussions.

Another bit of strangeness is we really don’t seem to be sure what we mean by “app developer.” Do web apps count? Games? For this discussion, at least, “app developer” seemed to be anyone who makes software. I wonder if we should be careful using “apps” as a catch-all without defining it.

So, most of the session was brainstorming where we can go to talk to developers who haven’t thought about building for Ubuntu, and how to get them on our side. There is a lot of software (especially games) that is almost platform independent, and in some cases making that software run on Ubuntu is as simple as the developer choosing to support Ubuntu. There were some interesting ideas about writing articles for magazines and media sites, defining target audiences (demographics?), and making a solid business case for people who like those kinds of things.

At one of the app developer sessions, somebody mentioned it would be nice if there was an Ubuntu presence at places like game development conferences, and there wasn’t much interest. The big one is that Canonical doesn’t really have the resources for it. I wish there were. In terms of third party developers making and supporting stuff for Ubuntu, that weren’t doing it before, I think there is one place where we can say this is really going well at the moment: indie games. No, this doesn’t involve using Qt or GTK or lenses or any of our fancy notification APIs, or even packaging (most of the time). But there are thousands and thousands of people who have paid for, downloaded and played Linux versions of games from sources like the Humble Bundle, likely using Ubuntu, and many indie developers are seriously looking to build their games for Linux so they can reach that audience. That is a heck of an opportunity and I think it needs to be appreciated and nurtured or it will go away as quickly as it came.

Making life easy for app developers

So, continuing along that trend, we also had a productive session called Upstream App Developer Documentation. This is something I have been really glad to see over at developer.ubuntu.com, and it was nice to learn we are on the same page here: great progress, but there is still work to be done!

One thing the site needs is content, so if you’re lovely and talented, think about submitting a tutorial for the Resources section. It would definitely be appreciated.

We talked about making it a little clearer that the site is dynamic, so people might be encouraged to approach it a little differently. We also discussed shiny ways to present API docs. The site has a good start there — particularly with its selection of a target Ubuntu version — but it is so far a little limited in scope and not as shiny as it could be. Alberto Ruiz is working on a neat Django-powered documentation site called GDN, so ideally we can fit these things together.

Installer slideshows

I missed a chunk of Monday — including my own session — because I was trying to sleep off a headache. So, we did the ubiquity-slideshow session, Installer Slideshow checkup and planning, on Tuesday! There weren’t a lot of people there, but I think we had an interesting chat, and it continued nicely in the hallway session later on. It was nice to meet some people who work on installer slideshows for different Ubuntu flavours, and we arrived on a few things to work on for Ubuntu 12.10.

I’m going to make a real start allowing translators to localize screenshots. This is a tough problem because of the tight schedule close to release, so it won’t be finished for 12.10, but I want to have something. Actually creating translated screenshots is part of the puzzle, and that will be similar to what the documentation team does but with some little decorations. Chances are I will be writing about it in the future, and it will involve some magical scripts by Jeremy Bicha as well as some funny manual work to make things look pretty. That bit is kind of entertaining anyway.

The next puzzle is actually getting those localized screenshots to users. They can’t live on the install CD because there just is not enough space (and even if there was, we’d be adding a lot of files that don’t do anything). In the future they might live on a web service, but for 12.10 I’m looking at localized ISOs. There was a great session about that: Localized ISO community growth, Quantal plans. This is an amazing project where loco teams build their own Ubuntu install CDs (like the Italian CD), completely ready to go with a translation of their choice, with extra goodies like custom Firefox bookmarks and Rhythmbox radio stations. I will be playing with slideshow customization in the ISO builder so translated screenshots can be added to language-specific install CDs.

Updates, sound themes and stuff

On Monday morning, there was a session titled Sound Theme, which was mostly a quick update about Ubuntu’s new, upcoming sound theme. I think that project is in good hands. It wasn’t finished last cycle because it’s actually a pretty big piece of work: this isn’t just the startup sound. Having seen how much work is going into this, I really look forward to where it goes.

Take a look at pkgme! I don’t remember the session at this point, but I was told the goal is to make packaging an application as simple as one command in many cases. (This way, an app developer doesn’t have to read the packaging guide and wonder why he needs a patch system for his own project). It sounds quite beautiful.

This cycle, I will be working a little bit with Michael Terry to implement the Software Updates specification. My goal is to get updates presented in a tree list where packages are collapsed beneath their respective applications. So, an update for Firefox will be presented with a single list item that says “Firefox Web Browser” along with a pretty icon, instead of a list of scary package names. This is going to be exciting! I’ll keep you posted, probably with the odd Twitter update.

As always, I have decided to actually use IRC. I do this every UDS. Hold me to it, okay? I actually set Empathy to connect to Freenode along with my other accounts this time, so it’s progress! (Yeah, I know, Empathy isn’t very good at IRC, but this way it will blend nicely with the background).

Oh, while I’m talking about random things, I’d just like to add a thanks for the t-shirts at this UDS! I love the design, and they’re super comfortable, too. So, whoever was in charge of that, you’re brilliant.

Fun!

 

And my favourite plenaries

Dave Walker’s MAAS demo gave me a powerful urge to buy eight computers and a lot of network cables just to play with that stuff, even though I have no use for it at all. It’s an impressive demo.

The lightning talks! I don’t know where or if a video is posted, but there was a fun lightning talk full of trivia about the infrastructure behind UDS. Metres of network cable, distance travelled by our heroic technicians, amount of data downloaded and uploaded, the wifi setup — that sort of thing. I hope it gets posted somewhere.

Ubuntu at Google with Thomas Bushnell. All about how Google is doing with Ubuntu deployed as a desktop operating system for thousands of employees. It’s always interesting to learn about Ubuntu desktops being used in big operations, especially when the resulting software can still be recognized as Ubuntu — which, in this case, it sounds like it is.

Electronic Arts with Richard Hillman, EA’s chief creative director. I think some people are disappointed that EA hasn’t gone and announced SimCity 5 for Ubuntu or something (okay, that would be pretty great), but I was perfectly happy to learn it’s a cautious, ongoing investigation instead of someone deciding to throw an expensive product at Ubuntu and hoping it sticks. This way, it could really work well in the long term — and there is room to sort out any problems along the way. After all, Ubuntu isn’t perfect yet. The Q&A after this talk is really informative, and it connects with the sessions we had about app developer outreach. It’s worth a watch!

Uh…

Thanks for the fantastic event, everyone! As usual I learned a lot, met some cool people, felt very humbled, and am writing way too much stuff in one blog post. And this isn’t even half of it. As usual, I should have made daily posts like all the cool people do. Oh well. See you next year! Quantal Quetzal (that’s pronounced ketzal) will be glorious.

Space Bees Attack! – My Ludum Dare 23 entry

My friend Andrew convinced me to participate in Ludum Dare 23 — a fun and informal 48-hour (or 72-hour) game making competition. He created an insane, brilliant little game called Baby Farm. I went ahead with a poorly thought out idea: “I’ll make a game where you have to build a structure to defend your world where evil things are falling from the sky! With physics!

The “with physics” part is where it went wrong. Words for the wise: physics are evil.

It sort of evolved in an interesting way, though. I decided I would commit to some specific technologies and make them work, rather than worrying constantly about my choice of tools like I usually do. I decided on Python and I grabbed a very nice hardware accelerated 2d graphics library called pyglet (which I had been meaning to learn) as well as pymunk for 2d physics, and I planned to do any graphics as quickly as possible with Inkscape. I started off just making it possible for the player to dump objects in the scene. Originally, the defensive structure was going to be assembled from different materials like bricks and pieces of wood. The results were kind of uninspiring.

After some fiddling, I erased a lot of code and switched to Tetris blocks: they fall from the sky and you place them to form sturdy towers. This is one thing that turned out really well. I have played a few Tetris + physics games, and they tend to be unbelievably frustrating. I wanted something that felt a lot like normal Tetris, so the blocks are easy to fit together and the one you control is fluid and reliable. Of course, this is where I finally explored a very important point first hand: video game physics are not real physics. To keep everything stable, blocks gain a whole bunch of extra mass when they are released, and some custom collision handlers ensure that blocks are as steady as possible while they are being controlled. The block being controlled by the player is pulled along by a set of constraints: one handles rotation (in 90° increments) and another forces the block to snap to a specific point on the X axis.

I then spent a bizarre number of hours trying to rotate a group of sprites around an arbitrary point. Because, of course, it’s great that you can have a physics library that tells you how to position and rotate everything, but if you can’t draw those objects the way it wants, you aren’t getting anywhere. Suffice it to say, I was getting nowhere for a very long time. I think part of this may have been a misunderstanding about coordinate systems, but in the end I solved it by borrowing the Sprite class from Cocos2d, so it could deal with that stuff for me. Yes, I suck at math.

My big regret here is that I completely ignored the point of the game — the Space Bees — until something like six hours before the deadline. This meant very little testing, so I ended up sticking to the first terrible idea that landed in my head: ‘they’ll fly like real bees, and just be really heavy blocks so the physics stuff will take care of itself.’ No, it doesn’t happen that way: I spent five minutes making them fly, and five hours tuning them, fruitlessly. In retrospect, I should have gone for something more like falling rocks, and I should have removed them from the physics simulation. Having these falling bees inside the simulation provided no gameplay benefits and created some rather horrible visual issues. Most importantly, I think it would have helped me if I had focused on establishing some kind of (preferably amusing) challenge as early as possible. As it turns out, that really is a big priority in making a game.

At any rate, this is my first Ludum Dare entry! It is possibly the first time I have built a game with any kind of playability, however brief the amusement may be. It was especially fun seeing the comments people left for my game. I was terrified at first, but they were all lovely, constructive and encouraging comments. The Ludum Dare community is amazing and I definitely look forward to doing this again, armed with the lessons and the confidence I gained this time.

You can download the game for Linux and Windows. If you want to play with the source code, you’re welcome to it. It’s all there in the Linux download.

Rescuing your data from Formstack’s “excess form submission” email

I seem to be accumulating all sorts of little scripts to solve odd little problems sensible people don’t encounter (and therefore don’t have scripts for). Normally I keep these to myself because they’re really ugly, but I’m going to start sharing the more useful ones on my blog. Maybe some day I’ll save someone a bit of time. That would be fun!

Using a Formstack form? Went over your form submission limit and now your email is flooded with thousands of useless error messages?

Yeah, that happened to us, too. (If it didn’t happen to you, just ignore me and carry on not being miserable).

Usually, Formstack stores submissions on its end (and they do a decent enough job of that), but after a point that stops being allowed and the service sends an email for each submission, instead. Thousands of really poorly formatted emails. I think they’re trying to sell something…

So, here is a Python script to turn all that Formstack email back into a single CSV file that you can actually use. The end result looks just like a CSV you would normally download from them, so you can get everything back to normal.

#!/usr/bin/python

# Formstack email scraper, by Dylan McCall, https://dylanmc.ca//-

import argparse

import codecs
import sys
import csv

# Fields that have arbitrary, user-entered text as values, so the parser knows
# to ignore reserved characters like "="
FREE_INPUT_FIELD_NAMES = [
	"Other",
	"Please specify",
	"Do you have any other comments about toxic ingredients in home cleaning products?"
]

def main():
	parser = argparse.ArgumentParser(description="Convert a CSV file full of Formstack overflow emails back to a useful format.")

	parser.add_argument('input', metavar='mail',
		help="The CSV file with Formstack email text in its \"Body\" column")
	parser.add_argument('output', metavar='output',
		help="A file to store the results in")

	parser.add_argument('--input-delimiter', default=',',
		help="Value separator used for input file. Escape sequences like \\t are allowed.")
	parser.add_argument('--headers', default='headers.csv',
		help="A CSV file from Formstack's Export Submissions feature. Only the top part (column names) will be used.")
	parser.add_argument('--unique-id-start', metavar="N", default=99900000, type=int,
		help="Starting point for unique submission IDs")

	args = parser.parse_args()
	args.input_delimiter = args.input_delimiter.decode('string_escape')

	try:
		headerFile = open(args.headers, 'r')
	except IOError as error:
		print "I couldn't open %s." % args.headers
		print "Please use Formstack's Export Submissions feature to download form"
		print "submissions as a CSV file. Save the file to the same place as this"
		print "script, and name it headers.csv."
		print "The file doesn't need any real submissions. I only need it for the"
		print "row of column names at the top."
		sys.exit()

	inFile = open(args.input, 'r')
	outFile = open(args.output, 'wb')

	submissionData = []

	# This bit of code really hinges on how your mail format works.
	# Right now, it's written for Microsoft Outlook's "Export messages as CSV"
	# feature, which has a Body column for message text
	mailReader = csv.DictReader(inFile, delimiter=args.input_delimiter, quotechar='"')
	for row in mailReader:
		mailBody = row['Body']
		data = getDataFromMail(mailBody)
		# Some messages may not have interesting data, so we ignore those
		if len(data) > 0:
			submissionData.append(data)

	# Field names can repeat, so we need to figure out the order they appear
	columns = []
	headerReader = csv.reader(headerFile, delimiter=',', quotechar='"')
	columns.extend(headerReader.next())

	dataWriter = csv.writer(outFile)
	dataWriter.writerow(columns) # Print column names at top of CSV file

	for index, data in enumerate(submissionData):
		# Populate the Unique ID field, which isn't in the message body
		data['Unique ID'] = args.unique_id_start + index

		outputRow = list(['']) * len(columns)
		for fieldID, fieldValue in data.iteritems():
			fieldParts = fieldID.split('###', 2)
			fieldName = fieldParts[0]
			fieldNumber = int(fieldParts[1]) if len(fieldParts) > 1 else 0

			# Find the [fieldNumber]th repetition of the column called fieldName
			columnRepeats = 0
			colNumber = findNth(columns, fieldName, fieldNumber)
			if colNumber >= 0:
				outputRow[colNumber] = fieldValue

		dataWriter.writerow(outputRow)

def findInList(lst, needle, start=0):
	try:
		position = lst.index(needle, start)
	except:
		position = -1
	return position

def findNth(lst, needle, n):
	needleRepeats = 0
	foundIndex = findInList(lst, needle, 0)
	while needleRepeats < n and foundIndex >= 0:
		needleRepeats += 1
		foundIndex = findInList(lst, needle, foundIndex+1)
	return foundIndex

'''
Extract all of the fields listed in the given message body.
@return dictionary of raw field data, mapping a field name+++number to its value
'''
def getDataFromMail(text):
	fields = {}

	IGNORE_COLUMNS = frozenset([
		"WARNING" #Formstack's warning message preceding information dump
	])

	lines = text.split('\n')

	currentKey = None
	currentValue = None

	for line in lines:
		if currentKey == None:
			# Looking for the start of a new field

			if line.find(': ') > -1:
				# Start of a new field.
				# This field can span many lines. Store currentKey so the next lines will be appended.
				# Record field name and first value (after ": ") if applicable
				key, val = line.split(': ', 1)
				key = key.strip()
				val = val.strip()

				if not key in IGNORE_COLUMNS:
					currentKey = key
					currentValue = list()
					if val:
						currentValue.append(val)

		elif currentKey and line.strip() == '':
			# Blank line always means the end of the current field.

			if len(currentValue) == 0:
				currentValue.append('')

			processedKeyValues = processField( (currentKey, currentValue) )

			for finalKey, finalValue in processedKeyValues:
				duplicates = 0
				uniqueFinalKey = "%s###%d" % (finalKey, duplicates)
				while uniqueFinalKey in fields:
					duplicates += 1
					uniqueFinalKey = "%s###%d" % (finalKey, duplicates)

				fields[uniqueFinalKey] = finalValue

			currentKey = None
			currentValue = None

		elif currentKey:
			currentValue.append(line.strip())

	return fields

def processField(field):
	fieldKey = field[0]
	fieldValue = field[1]

	orderedSubFieldKeys = list()
	subFields = dict()

	data = list()

	for value in fieldValue:
		# This behaves poorly for values that, themselves, contain " = "

		# Ugly hack for cases with user input
		if fieldKey in FREE_INPUT_FIELD_NAMES or value.startswith("Other:"):
			splitValue = [value]
		else:
			splitValue = value.split(' = ', 1)

		subFieldKey = ''
		subFieldValue = ''

		if len(splitValue) == 2:
			subFieldKey = splitValue[0]
			subFieldValue = splitValue[1]
		elif len(splitValue) == 1:
			subFieldValue = splitValue[0]

		if not subFieldKey in orderedSubFieldKeys:
			orderedSubFieldKeys.append(subFieldKey)
			subFields[subFieldKey] = list()
		subFields[subFieldKey].append(subFieldValue)

	for subFieldKey in orderedSubFieldKeys:
		if subFieldKey == '':
			completeHeader = fieldKey
		else:
			completeHeader = "%s (%s)" % (fieldKey, subFieldKey)

		subFieldValueList = subFields[subFieldKey]
		completeData = '|'.join(subFieldValueList)

		#Final output!
		data.append( (completeHeader, completeData) )

	return data

if __name__ == "__main__":
	main()

Using this is a little involved. But that’s why you’re still here, right?

First, put that script somewhere. Name it extract-data.py. If you haven’t guessed, you’ll need Python. I’m assuming you’re using Ubuntu, MacOS X or something of the sort, but Windows should work as well.

Now, you need two more files.

Go to the Form Submissions page for your form on Formstack. Click the button near the bottom, to export some submissions as a CSV file. I don’t really care how many submissions you export: this output is just being used to get a proper set of column headers. Save that CSV file as headers.csv and put it in the same place as our Python script.

Next, we need your email, ideally in something like a CSV file. In this case, I was working with a big CSV file from Microsoft Outlook. (We actually used tab-separated values, because Outlook was doing something nightmarish with character encodings. Somewhere between exporting as a TSV and opening the thing differently in LibreOffice, something started to cooperate). Mozilla Thunderbird has an extension to export a folder as a CSV file, too, and it probably won’t be trying to murder kittens.

Now, because there’s no real standard way to do a CSV-formatted heap of messages, you might need to edit the script to get it to read your messages. There’s a little bit of code at the top — basically everything involving mailReader — which you can edit. It just loops through each message in the CSV file and passes that message’s body text to the message scraper. (If you’re feeling creative, you could use an mbox file with Python’s mailbox library).

While you’re at it, you need to edit another variable. FREE_INPUT_FIELD_NAMES is a list of form fields where people can enter whatever they want. (Text fields, for example). You need to add every field that is like this on your form. Otherwise, the scraper will trip over peoples’ inputs.

…Okay, now we’re ready to go!

Open a terminal, go to the place with your script (and mail.csv, and headers.csv), and do this:

python extract-data.py mail.csv output.csv --headers=headers.csv

Boom! Magical, formatted data.

The resulting file, output.csv, should look something like this:

Of course, please be gentle and don’t take this as an excuse to misuse Formstack’s service. They probably wouldn’t be happy with that and they’re being very kind by processing submissions above the maximum at all. Especially, in our case, 1600 over the maximum :)

We used this to recover our data and, barring the hours of cursing and hair-pulling as I tried to make this scraper work, it was quick and painless. I hope it helps someone else out there, too.

Why isn’t Ubuntu Forums in the installer slideshow?

Observe as Dylan ernestly makes a mountain out of a molehill, then desperately backpedals in an effort to not sound like a pretentious bozo.

There is a prominent feature request / bug report for Ubuntu’s installer slideshow. It’s titled “Precise Pangolin – Please include ubuntuforums.org in the slideshow.” Right now there are 84 people who agree that should happen, and many of them have been pointed there from a couple of forum threads. The forums are very good at that.

I’m just going to go and say it: I know it’s important to people, but that isn’t going to happen for Ubuntu 12.04. I’d love to talk about it in the future, and it might happen, as things change, in another release. Here is why it isn’t happening today:

The install slideshow appears for about fifteen minutes, for every Ubuntu user who does a full install from the desktop CD. And then it never appears again. It is fundamentally incapable of being a reference guide. And that’s good, because we already have many excellent reference guides and support sites. The slideshow is not trying to be definitive or in any way, shape or form a distinct source of information.

What is it, then? I want it to be a simple source of ideas, inspiration, and – most importantly – confidence as people dive in to Ubuntu. And I want it to be those three things without sounding any more important than it is (which is not at all). This is a very little thing. My path is rather erratic, but there are a few points I’m quite sure about.

When you help someone think “I can do amazing stuff with Ubuntu that I couldn’t do before,” you’re doing it right (and you deserve some kind of trophy). The “I can” is important, because we do have a large number of “they can” situations, and those aren’t the same. It’s good if people feel powerful right out of the gate.

Two thoughts that do not inspire confidence: “I’d better memorize this” and “OMG so much stuff!”. These thoughts come as soon as you present new information that isn’t a natural part of old information. Ideally, to avoid them, you’ll present “new” information that was already on the user’s mind, and keep the stakes low: if anyone is worried what might happen if they “forget” something, it isn’t working. I want people to know what’s there, but they don’t need to know any details yet. Ubuntu is well designed, anyway: everything will just work as long as people trust it to work.

There is a particular quota for things we should introduce, and that should vary depending on how important each of those things is. We can’t put six things on the screen at once and expect any of those things to be noticed. Right now, I think it’s all maxed out. I don’t think there is room for any more slides’-worth of information, so the ideal thing is refinement (and maybe even removal) of what’s already there. If anything is being added, something else needs to be removed, or at least changed.

We need to be sensitive to the fact that people are currently installing a new operating system, perhaps for the first time, so there is already a lot of new stuff going on.

I know it isn’t doing this yet, but I think a really good installer slideshow will do lot of showing, and much less telling. When we just tell about things, people compare what they’re being told against their existing assumptions (“you probably have to do that with some terminal command,” “I bet they’re all neck-beards”). Quite often, it ends up with the assumptions winning.

“So, that’s a lot of babbling, Dylan, but why Ask Ubuntu and not the forums?”

This one is a mix of things.

The install slideshow ends with a slide that is about getting help. Right now it’s pretty heavy on tech support, but the idea is “don’t worry, you are not alone!”

Yes, I know there's an excess of Planet Ubuntu on the right. Don't worry, just teething problems.

It has a new Twitter thing which I think plays an interesting role, but its main copy points at Ask Ubuntu, and it ends with a link to ubuntu.com/support. Now, I didn’t actually write the copy here, but I’d like to explain what I like about it.

I like that it links to Ask Ubuntu as something to look at right now, because I think it’s possible to show instead of just tell about Ubuntu’s community support. The second link is pretty well just ubuntu.com, so I’m hoping the viewer will think “I already know that,” instead of actually worrying about writing it down, or even looking at it. If we just throw in a third thing, like ubuntuforums.org, I think we’re wandering above the (very unscientific) new things quota, as we start having these two brand new items competing for the same kind of attention, both for the same reason. It’s not unlike having two web browsers installed by default.

Why Ask Ubuntu and not the forums, then? I’m not terribly invested either way, but let’s just compare the two landing pages.

With Ubuntu Forums, the information available at a glance is that you really need to REGISTER, there are a lot of categories, there’s a category for absolute beginners, and (if you’re especially observant) people post quite frequently. The last two pieces of information are quite compelling, but I’m put off by how far the viewer of the slideshow is from actually seeing people get help.

I think Ask Ubuntu just does a better job for the quick glance – “hey, this place looks helpful” – kind of thing. First, there is no obligation to register, and I like that for making things a little more relaxed. Second, the site is all about Q and A and the landing page has a prominent list of questions and their answers. The site also has an “Ask a question” button whether you’re signed in or not, so I think it’s a little more inviting. I think someone can see Ask Ubuntu and understand it in three seconds.

Now, I love the forums, but I think the slideshow needs to be streamlined, and I think Ask Ubuntu gives a more streamlined demo than Ubuntu Forums. Actually, I think we’d be doing the forums a disservice here. There’s a risk that someone’s first taste of the forums will be “whoa, too much new stuff, not now!” (in some fashion), because that’s just the nature of the forums: the forums are big, and complex, and there are a lot of fascinating people who do a lot of different stuff. It’s a huge piece of the Ubuntu community, and it doesn’t fit well in the small snippet of time the slideshow aims to consume, much as the slideshow isn’t linking straight to the Wiki or the IRC page.

I think there’s lots of room to talk about the Ubuntu community in the slideshow, but we need to be gentle. Let’s talk about the forums somewhere that a person can discover the site when they’d like, and be able to give it a full fifteen minutes. There are some great places to bring up UF. There’s the default browser bookmarks, there’s the (kind of neglected) Get More Help section in the Ubuntu Desktop Guide, and there’s ubuntu.com/support. The forums can appear in lots of useful and interesting places, and I’d love to see that happen.

But the slideshow is not something that you can just dump something on. It’s not a big truck. It’s a series of slides!

My new website uses WordPress and Smarty

Spurred on by the rather dull situation of my web dev portfolio being a single long page with pictures and writing on it (which simply will not do!), I spent the last week moving my blog (and my portfolio) to my own webspace with WordPress and a custom theme. This entailed re-learning PHP (ugh), learning more about WordPress, and pulling a lot of hair. And the whole thing is still a work in progress, but it’s here, and it’s better than my old blog. Yay!

WordPress has worked really well for me, so far. It has a really good post editor that doesn’t spew broken HTML all over my posts when I look at them wrong, the SQLite plugin works (for when I was testing locally), installing it was simple, and it has reasonably detailed documentation, most of the time.

But there is one place where WordPress and I simply cannot agree, and that is writing themes. My new site has a fancy header that picks up rows as you go from page to page. My attempt to create reasonable semantics for that under a typical WordPress theme was completely fruitless. The convention in WordPress is to write a theme in straight PHP, using stuff like the_content() to print the selected post’s content directly. Coming from Django, I was not a fan.

What I wanted was a good template engine, like Django’s. I wanted it to support inheritance, so I could have a base page and override specific parts of that base page in each subpage. I wanted the ability to use the same template for different back-ends without too much turmoil. I wanted to stick variables in page content without generating incomprehensible giberish, and I would prefer to separate logic from presentation at least a little. There is just a thing for PHP: Smarty! Now, some searching for Smarty support in WordPress led me to a lot of requests and no real implementations (or attempts), so that’s why I’m here. This site passes through Smarty for just about everything, and I probably didn’t do it right but at least for a small site this method seems to work. (And I’ll keep this post updated if it doesn’t). I’m going to write (or at least brain dump) some tips for those who are looking to do something similar. I am assuming, if you’re interested, that you have some idea how Smarty works and some idea how WordPress works, but not much of an idea is necessary.

This probably isn’t terribly efficient, though it’s worth noting that Smarty is normal PHP code once each template is run for the first time, so at least to me it intuitively felt more expensive than it ended up.

First, you need a custom WordPress theme that acts as an adapter to Smarty, because Smarty can’t talk to WordPress directly. (It can’t call functions). Unfortunately my WordPress theme was made in a big hurry so it isn’t reusable. (A clever person could do better – maybe something like the Sandbox theme). Coming from Django, I decided I would think of WordPress’s template files like Django’s views; the things that go between an HTTP request and the (Smarty) template system. Look through WordPress’s Template Files List. In each of those, instead of displaying something directly we’re going to gather data and send it over to Smarty.

You’ll need to add some stuff to functions.php. First, load the Smarty library (require('includes/Smarty.class.php')). Then we have a Smarty subclass that can be used for the project:

class Smarty_Base extends Smarty {
	function __construct() {
		parent::__construct();

		$basedir = get_theme_root().'/'.get_template();

		$this->setTemplateDir($basedir.'/smarty/templates');
		$this->setCompileDir($basedir.'/smarty/templates_c');
		$this->setCacheDir($basedir.'/smarty/cache');
		$this->setConfigDir($basedir.'/smarty/configs');

		//$this->caching = Smarty::CACHING_LIFETIME_CURRENT;

		//** un-comment the following line to show the debug console
		//$this->debugging = true;
	}
}

I added another subclass that would guarantee some variables be sent to every template. The most important ones here are wp_head and wp_footer, which are popular points for plugins to attach themselves.

class Smarty_Wordpress extends Smarty_Base {
	function __construct() {
		parent::__construct();

		$wptitle = wp_title(' | ', false, 'right');
		if ($wptitle) {
			$this->assign('page_title', sprintf("%s %s", $wptitle, get_bloginfo('name')));
		} else {
			$this->assign('page_title', sprintf("%s", get_bloginfo('name')));
		}

		$this->assign('resource', get_stylesheet_directory_uri().'/resource');

		/* get wp_head */
		ob_start();
		wp_head();
		$this->assign('wp_head', ob_get_contents());
		ob_end_clean();

		/* get wp_footer */
		ob_start();
		wp_footer();
		$this->assign('wp_footer', ob_get_contents());
		ob_end_clean();
	}
}

So, WordPress (bless its heart) doesn’t have a get_wp_head function; it will only print the head directly. We can work around that using PHP’s output buffering functions, which are those blocks of code with ob_start and ob_get_contents. Again, not pretty, but this solves our problem: now the template can choose when and where to stick wp_head and wp_footer. You’ll be using this a lot, actually.

From here, getting Smarty to render my pages has been as simple as making Smarty templates and running them (with the right values) from the appropriate WordPress template files. Because Smarty templates inherit from each other, don’t be afraid to add helper functions since you might have all of your pages sending the same variables to templates because they inherit from the same base.

For example, I added a function that is called for any template inside my blog section:

function prepare_smarty_for_blog($smarty) {
	$smarty->assign('page_number', get_query_var('paged'));
	$smarty->assign('next_posts_link', get_next_posts_link("Older posts"));
	$smarty->assign('previous_posts_link', get_previous_posts_link("Newer posts"));
	$smarty->assign('blog_sidebar_html', get_sidebar_html('blog-sidebar'));
}

By the way, get_sidebar_html is to do any collection of widgets. Just like wp_head and wp_footer, we need to use output buffering because widgets output their contents directly when asked. Otherwise, it’s a pretty ordinary sidebar that we can freely customize from the admin page:

function register_sidebars() {
	register_sidebar(array(
		'id' => 'blog-sidebar',
		'name' => 'Blog sidebar',
	));
}
add_action( 'widgets_init', 'register_sidebars' );
function get_sidebar_html($index) {
	$sidebar_html;
	/* get sidebar contents */
	ob_start();
	if ( dynamic_sidebar($index) ) {
		$sidebar_html = ob_get_contents();
	}
	ob_end_clean();
	return $sidebar_html;
}

In the end, the WordPress template for a post page (single.php) looks something like this:

<?php
/**
 * The Template for displaying all single posts.
 */

$smarty = new Smarty_Wordpress();

$post_id = get_the_id();
$post_full = new PostContent($post_id);
$smarty->assign('post', $post_full);

prepare_smarty_for_blog($smarty);

$smarty->display('blog-post.tpl');

?>

And here is the Smarty template blog-post.tpl:

{extends file="base-blog.tpl"}

{block name=main}
	<div id="post-{$post->id}" {$post->class_html}>
		{if $post->blog_import_url}
		<div class="post-message">This post was imported from my old blog at {$post->blog_import_url}</div>
		{/if}

		<div class="post-main writing expanded">
			{$post->content_html}
		</div>

		<div class="comments-wrapper">
			{$post->comments_html}
		</div>
	</div>
{/block}

Isn’t that tidy?

PostContent is just a little object that grabs bits of information for the selected post and stores it all in readily accessible variables intended for Smarty. Alternatively, you can just fill up an associative array or a dictionary and send that to Smarty. The object was useful for me because I like to needlessly add objects to things, and because I could reuse code between lists of posts and actual posts. It’s pretty simple (and pretty site-specific), but one thing that may jump at you using this approach is lots of WordPress functions know what “the_post” is based on a global variable — $post. Since we are doing so much stuff outside the ordinary flow of a WordPress theme, changing the global variable can cause some problems. So, I tend to do something like this:

global $post;
$old_post = $post;
$post = get_post($post_id);

/* do stuff with the post, like get_the_excerpt() */

$post = $old_post;

Whether you find yourself in that position seems to depend on who you are. I found my code reading post IDs and wanting to immediately delegate processing said IDs to something else. Somebody else might run get_post from a higher point, in which case that global variable silliness is completely unnecessary.

Another more common thing to keep in mind is that WordPress likes to generate strings with HTML for us right away, and sometimes we can’t change them very easily. (And usually that’s a good thing. It’s pretty cool how it will generate a comment form or a post password form and magically process it for us, and it generally just saves time and makes your theme more flexible). To make this a little tidier, I like to add _html to the end of variable names that point to HTML content, instead of fighting a losing battle and making WordPress less cool :)

And that’s pretty well it. There is still lots of boilerplate, but I find it much more readable than a lot of WordPress themes out there. Smarty’s template inheritance is really the main thing I wanted and it works beautifully. In short, you can use Smarty with WordPress, barring some limitations. If you (like me) have been convinced that dedicated template languages are the solution to everything, it could even be worthwhile!

June, 2013

I have been iterating on my theme, recently. I added some responsive design bits and pieces, I changed the layout for sidebars and footers and things, I set a maximum width for the content, and I made comments look a little prettier. This involved a lot of CSS, but it also involved a lot of changes to markup in order to support that CSS. I noticed something interesting: the last modified date for all of my PHP files except comments.php and functions.php is April 2012. I had to modify comments.php because I still haven’t gathered up the courage to sort out that mess and move that template to Smarty, and the only recent change for functions.php was registering a new sidebar. So, as a person who hates PHP, I think this was pretty cool.