In the past few months, we’ve launched What Would Bill Say and a GOP Candidate Tracker, while also relaunching our Patriots Report Card. All three of these projects rely heavily on Javascript and are among the first projects I’ve written in the language. I’m a Django guy, so this front end stuff is scary. The design of these projects were done by my coworkers — I focused on the backend.

Patriots Report Card

This is week 14 of our Patriots Report Card. This was a tool previous developers at the Boston Globe worked on. It runs in Flash and updates a MySQL database. Six weeks ago, we switched it from the previous version. The new version reloads the page when you vote, and so we kept your voting information in the URL instead of saving it within the Flash program.

My coworker, the great Tom Giratikanon, worked on the Flash part. I write the JS that stores your vote in the URL and returns it so the Flash bits can use it, with an assist from Tom.

First a regex I found by Googling splits the various parts of the URL. Then a few variables are given the values from the URL and the SWF file. Lastly, it loads your new votes into a function called reportCardReload, so when you click the next button it reloads the page and keeps your vote information. I know, we don’t have to reload the page. But it’s what I was asked to do and I think it works nicely.

Not exactly the hardest thing in the world, but it was one of my first JS projects I did mostly on my own.

GOP Candidate Tracker

A few weeks ago, we launched our GOP Candidate Tracker. This project was quite difficult, as it uses a lot of brainier Javascript/jQuery and I am still a beginner. Thankfully, the aforementioned Tom and a few internet pals helped walk me through some of the concepts.

First, we have two interns in the newsroom who enter information into Google Docs using a Google Form. Then I download the latest events, paste them into a .csv and run a python script that turns it into JSON. I test it in a fake version of the live page that can only be seen inside the Globe — just to make sure no weird characters got in — and then I paste it into the live JSON file and upload that to cache.

I will be building a Django-based backend for this, so folks will enter the information in there instead of Google Docs in the future. Then the Django app will spit out a JSON file, similar in structure to what we’re already using.

It wasn’t built in Django first because we didn’t have an area to build Django projects at the time. Now we do, thanks to the hard work of our Ops team. So we will be switching that over in the future.

Anywho, this project had to do a few things: Parse a JSON file, sort it in various ways and allow you to click a button and show only a certain candidate (and make that appear as a hash in the URL so you could share the individual person’s dates). I used jQuery, underscore.js, date.js and jQuery-BBQ, the latter built by a former Boston Globe staffer.

What Would Bill Say

As many folks may know, Bill Belichick keeps his quotes pretty bland and quite simplistic. Our sports team asked if we could make something that showed one of his random quotes. I countered with, “What if we allowed you to ask him a question and it answered with a real quote of his?”

This app works like this: You ask a question, it appends the question into the URL, generates a random number and appends that to the URL, too. The random number picks which photo/question to show from a list within the Javascript. It also looks to see if a question has been asked (aka shows up in the URL) and spit out the appropriate page information.  I used jQuery-BBQ again to help me parse the URL and jQuery Form Defaults to help with a few form issues.

My pal Dan Nguyen looked at an early version and found it had some cross-site scripting vulnerabilities because I wasn’t sanitizing my fields. Underscore.js helped me with this.

We wanted to keep everything in the URL so the original question and answer would be easily shareable on social media.

Lessons Learned

These projects mostly do everything inside of the browser, loading flat files (except the Patriots Report Card). While the Candidate Tracker has a wannabe-database backend in a Google doc, it ends up pulling in a flat JSON file. This means I don’t have to maintain an application server, a database server or a caching server. Me gusta. A year ago I probably would’ve set up a full Django stack for each of these projects.

Also, I am no longer scared of Javascript. So that’s a good thing. In the future, I’m going to try to make as many projects with just flat files. I’d also like to start learning more about using Raphael for graphics.

My only resolution last year was to get on stage and attempt to entertain people. I accomplished that goal in these last few months. So I thought expanding my resolutions to three would be doable. A few of these were inspired by the internet.

1. Spend a week eating like a vegan

2. Travel more

3. Buy the majority of my media — books, magazines, movies and television shows — in an electronic format

I once tried to eat vegetarian for a week but forgot on my second day while rushing from one assignment to another.

As for traveling, I don’t leave wherever I live that much. When I do it’s usually to visit my family in Nebraska. I want to take a vacation someday and go see the Pacific Northwest. Hopefully. I hear my beard can be used as currency up there.

And by looking around in my apartment, I own too many books, many of which I won’t read a second time. Same goes for movies. I’ve been going to the library more and I’ve almost finished the latest Steve Jobs biography on my iPad/iPhone, which has been a pleasant reading experience so far. But it’s a bit hefty and I want a product only for reading.

So, I ask you, dear reader: What’s the best device to buy that’s only for reading? I won’t use it to surf the web, check email or do anything else. I just want something that I can buy books with and read them. Let me know in the comments or yell at me at @andymboyle.

We’ve got some people entering in some data in a Google Doc spreadsheet for a project. After fiddling around with attempting to make Google Docs API spit out not-ugly JSON, I said screw it and came up with a new plan: I was going to write a cron job that downloads the CSV from the spreadsheet and then parses it into JSON.

And of course, as a guy who likes Django, I turned to python for the parsing.

First, you will need to remove the first line in the CSV if it had any field names. Put it into a folder somewhere, perhaps. Then make an empty file and name it parsed.json and place it in the same file.

If you’re on a Mac/Unix-y thing, use your terminal and cd into the folder your parsed.json file is. Enter

pwd

in the terminal to see the path to your directory. If you’re on a Windows box, then it’s probably C:\wherever\your\stuff\is\

Now create a new file called parser.py. Enter in the code below, changing /path/to/filename part on line 5 to the bit that was spit out after you typed pwd (or where you inherently found it if you’re in Windows). On line 7, change fieldname1 and whatnot to the name you want each row to have in your JSON file.

import csv
import json

# Open the CSV
f = open( '/path/to/filename.csv', 'rU' )
# Change each fieldname to the appropriate field name. I know, so difficult.
reader = csv.DictReader( f, fieldnames = ( "fieldname0","fieldname1","fieldname2","fieldname3" ))
# Parse the CSV into JSON
out = json.dumps( [ row for row in reader ] )
print "JSON parsed!"
# Save the JSON
f = open( '/path/to/parsed.json', 'w')
f.write(out)
print "JSON saved!"

In your terminal, while you’re in the proper directory, type:

 python parser.py 

BOOM! It should spit out “JSON parsed!” and “JSON saved!” If you wanted to spit out the JSON in the terminal, you could add a line at the bottom:

 print out 

Voila. Now you can turn your CSV files into JSON.

The basic idea is this: Use the data of what people click on with Google AdWords/Microsoft’s adCenter to help you come up with coverage. First, you set up advertisements for a certain area of coverage, say an upcoming convention in your town, or maybe you’re trying to allocate resources between two not-as-often-covered sports teams and not sure which has a greater audience. Then you send those ads to a landing page or article you’ve already written about said topic.

You can then see how many people clicked through the ads, what the bounce rate was, how long they stayed on the article/landing page (using your own embedded analytics, of course) and whether or not they explored more of your website. Then you can deduce whether one subject or another received more page views than the other, all for maybe $100 in advertisement costs (plus the time it took to set up the pages).

Now, say you were planning on hiring a beat reporter to cover one subject over another, I would say that $100 is a cheap price to make sure you’ve made a good decision for your audience.

This may smack in the idea of what I’ve heard some people freak out about, which is catering too much to your audience. Giving them their chocolate cake instead of the peas and carrots they should (says we) be eating. Well, with dwindling resources at many media organizations, I can’t see why it’s a bad idea to gauge your audience before starting a new venture.

Our industry could use more analysis in making decisions, I believe, and thankfully we have a lot of tools at our disposal.

And if anyone is already doing this, I’d love to hear how it’s faired in the comments.

***edit at 10:51 a.m.***

Ted Han has pointed me to this post about how Homicide Watch uses analytics for coverage, which is totally rad. Laura Amico used her site’s analytics as a reporting tool. Awesome.

One of them asked, “Isn’t Siri still in beta?” Yes, yes it is.

I responded that I hate things being called “beta.” The rest of this post will be, hopefully, a more erudite explanation. With less swears. Maybe.

Saying something is still “in beta” is a goddamn cop out. It’s like you’re releasing something and winking, saying, “This could still suck so please be gentle and don’t be mean!” That’s a crock of horse shit. You released a product live, showing it to the world. It is no longer in “beta” mode. It is now in production. Otherwise, why did you release it?

Oh, you say you’re going to add features and fix a few things here and there? Cool. That’s more or less how everything is always done. I don’t think most folks refer to Twitter or Facebook as being in “beta,” as they’re adding features and updates all the time. Most web projects are continually in flux and are constantly being updated. To quote The Social Network, it’s like fashion — it’s never finished.

And to use my journalistic background, this is akin to calling the newspaper the “beta” version of stories, as people can go in and update the online version of articles whenever they want. No, we call it the damn newspaper. Get ‘em while they’re hot.

The phrase has a use, of course: internal testing. Once you release it to the world, though, stop calling it beta. It’s live, goddammit. Be proud of your accomplishments. Building that thing and launching it was hard.

Don’t hide behind a four-word veil of I-hope-this-doesn’t-suck-too-much-itude.

Also, if you think I’m wrong, please explain your arguments below. I’d love to hear a good argument for using beta.

As I normally attempt to save this blog for journalism matters, I’m going to break form a little bit and write about myself. Or rather, I’m going to write about what I want to accomplish in the next year. So, here’s a list, as I hear the internet likes lists:

1. I want to get healthier. More importantly, I want to be able to run three miles at a time on a regular basis without any problem.

2. I want to go to more shows.

3. I want to open source a project from work.

4. I want to perform in front of people somehow. Perhaps on a regular basis. Whether it be musically, funnily, or whatever. I just want to do this. I miss it, and I’m realizing how much it used to make me happy.

5. I want to do more writing. Whether that means on my blog, tumblr, articles, whatever. I just need to do this.

So that’s that. I’m 26. These are my goals before I turn 27. I think I can do ‘em, right? Right. Feel free to offer advice and tips. I’m excited for this next year of my life. Things are quite swell, for the most part. I’m a lucky guy.

I’m trying to keep track of the number of words per post as well as whether or not did any interviews for their piece.

If you see any I’m missing, feel free to post them in the comments.

This led me to thinking about road trips. And how I’ve never really had a full week off since graduating college. And what I would do with such time. And my ever-expanding dreams of evangelizing newsrooms with my general nerdery and web-centricness.

Slowly I came up with an idea of a spending a few weeks during my vacation driving to and from smaller newsrooms around the United States, attempting to teach them the very, very basics about how they could get into the web development game and augment their daily journalism with tools they build themselves.

You know, like Caine in “Kung Fu.” Drive from place to place, meet newsrooms, get into adventures.

I’m not sure how receptive newsrooms would be to this idea, but it’s something I’d like to do. I’m sure other people would love to do something similar. So what I’m proposing is maybe finding 40-50 newsrooms in different parts of the country. Section them off by general area — the Midwest, the South, New England, the Southwest, the Pacific Northwest, etc. And then find someone willing to take a week of their vacation time and spend a day at each organization, attempting to get them up to speed on a few things.

This is incredibly daunting, I know. You can’t build an entire culture in a day, and most newsrooms would probably be quite busy with the standard operations of a daily news organization. But I think you could at least help a newsroom get set up with cloud hosting on Amazon Web Services or Rackspace, maybe set up a server or two for them (using Fabric or something similar, of course). Maybe even have a handful of tools — think the jQuery before-and-after slider — you could set them up with.

A lot of ground to cover in a day, I know. Or maybe one would spend two days at an organization. I don’t know. This is just an idea, but I’m wondering if others would be interested in doing it? Of course, it’d be nice if the news organizations would provide food and lodging, maybe a per diem to help pay for gas and your time. But I see this more as a nice service to spread the ideas of making your news organization more autonomous.

What do you think? Share your thoughts below.

If you’re a person who has these skill sets, you should really think about applying. Where else can you get to build awesome projects that will be most likely seen by hundreds of thousands of people (at some of these news organizations)? Where else will you build news applications that can do some good for the world? Where else can you jump in and try to help revive an entire industry?

Not many places. If you don’t have journalism experience, that’s cool. Most journalists didn’t have it either when they decided to become journalists, so I’m sure most places won’t mind. Odds are if you’ve worked in any sort of a web development environment that required you to have the skills necessary to do great work for a news company. Can you communicate well? Do you do agile development? Can someone tell you “we want these features” and you can do them? Yes?

Then you’ve got most of the skill sets needed to succeed at newspapers. Sure, you may have to deal with some bureaucracy just like any large company. And you may need to fight a bit to start using some newer technologies. But as many organizations have shown, you can succeed. You can make great changes. You can kick ass and take names.

We need you, folks. So start applying.

The big problem with learning how to do this stuff is you need to know a little about a lot just to get a basic project off the ground. And not every tutorial tells you that you need to know X, Y and Z. Therefore, I’ll try to mention all the things you should get some basic reading up on.

But before I start babbling, you need to know that I come at this with a bias toward the programming framework called Django. Just because I like Django does not mean you have to like it.  When possible, I will point out other frameworks/languages you can and should check out. The important thing to remember is that you can get the job done using many different tools. It’s your job to find the one that’s best suited for you.

[edit] Oh, and here’s the tl;dr from Brian Boyer of the skills to learn:

Server ops, SQL, a scripting language (Python, etc.), MVC + templating (Django, etc.) HTML, CSS, JavaScript + libraries (JQuery, etc.)

Just FYI.[/edit]

Now that that’s out of the way, let’s begin!

SO THE INTERNET SORT OF REALLY IS A SERIES OF TUBES

Did I just blow your mind? Okay, well, in order to build anything on the internet, you need to know how the damn thing works. This is a simple tutorial that can explain it to you. You can also scroll down to “How The Internet Works” in this post for an even more layman’s version.

The basics are this: You’ve got servers. On them are databases. A website is basically a database that your computer is asking to talk to. Your browser renders the code that makes the website look pretty.

So the basic areas you need to understand in order to get a project live are this: Servers, databases and website code.

Let’s start with the code on the website. When you go to a website — like this one — something makes it look pretty. You may have heard of this. The basic building blocks are HTML and CSS. If you right click on the page and hit “view source,” the stuff that pops up is the HTML, or hypertext markup language.

After viewing the source, on line 11 you’ll see a link to this stuff. That’s the CSS. It basically makes it quick and easier to style stuff. So instead of writing “make the background of this box red, make the font of any text in it size 14 and make this box 300 pixels wide,” you just call something you previously defined.

This follows one of the basic rules of programming: Don’t Repeat Yourself. You could program something without CSS, which is called using inline styles, such as on this amazing website, but you’d be repeating things over and over. Hence the use of CSS, which stands for cascading style sheets.

So these are some basic building blocks you should learn. You don’t have to be a master at it — I’m certainly not. But you need to know the rudimentary basics so you can chat with a You could try this tutorial, or countless others on the internet.

DATABASES AND STUFF

So knowing HTML/CSS will show you how to make flat websites, basically every individual page has to be hand-coded. Another way websites work is by hitting databases, asking for information and spitting it out on a page. This blog — most blogs, really — work this way.

Think of it like this: Each part of a blog post, or a story on a newspaper website, is made up of different parts. A headline, byline, text field, related links, etc. All of that is stored in a big spreadsheet, basically. And the website is programmed in such a way that when it sees www.thiswebsitewhatever/2011/04/22/best-cat-photos it knows to pull information for that post or story.

So you need to learn the basics of something called SQL, which is the language most databases talk. It stands for “structured query language,” just so you know. A basic line looks like:

 SELECT * FROM awesome_pet_names WHERE pet="cat"

That may be rusty. But basically, you talk to a table named awesome_pet_names, asking for every thing in it where the field “pet” has the word “cat” in it. That’s a basic SQL query, and that’s more or less how the entire internet runs. Especially the part about cats.

SQL is also used a lot in database-oriented journalism, so it doesn’t hurt to know. Here’s a good place to start learning some of the basics. My pal Derek Willis suggests one read this book, The Art of SQL by Stephane Faroult, which is more advanced.

OKAY SO WEBSITES ARE MORE THAN HTML, RIGHT?

OMG you’re learning so fast! Yes, they are more than that. Instead of writing our websites in SQL queries, we write them in what’s called a server-side scripting language. These are languages that talk to the server, hit the database and spit out data onto websites. Some examples include Python (some don’t agree), Perl, Ruby and PHP. Python is what powers Django, Ruby powers Ruby on Rails and PHP can be used on its own. This blog is written in PHP, for instance.

Also, don’t learn Perl. Just trust me.

Before diving into a framework, I would highly suggest learning the basics of PHP and how it interacts with MySQL, which is database software, among the most common on the internet. A good book is Head First PHP and MySQL. It teaches you the basics and it doesn’t get too boring. You can also find some good tutorials online.

Once you get the basics of PHP down — and I suggest learning how to install WordPress and setting up your own blog on your own server. Which we shall now sort of discuss.

HOW DO YOU MAKE A WEBSITE DO?

I’ve always maintained that the hardest part about making a website is setting it to run on a server. So if you make cats_rule.php that lists your favorite cats, you want it to live somewhere that people can see. Just so you’re aware, if you’re lucky enough to work at a place that has server operations people — server ops, ops, whatever they get called — you may not have to deal with this stuff as much. But it certainly doesn’t hurt to learn it.

So, a server runs on an operating system, just like your computer. One that I prefer is Ubuntu 10.04 LTS, which is one of many variations of Linux, an open source operating system that people work on out of the goodness of their hearts. Or you could use Windows Server, which was made by people who want you to have to pay lots of money to use it. Not that making money is bad or anything, but I tend to go with free stuff made by people who want to help the world.

The server, which has that operating system I mentioned, has to actually live somewhere. For many, many years, the only way you could really run a server was to either build your own box or rent space somewhere. Or have awesome ops people who set up stuff in a server room.

You may have heard of this “cloud” thing. Basically, big companies like Amazon have so much server space that they had a bunch left over that they can turn into virtual machines. Instead of making an individual physical server, they just partition their hardware to make multiple versions.

This means you can set up stuff really easy. This also means you can store media really cheap, too. I would suggest using either Amazon Web Services or Rackspace. They are both relatively cheap, with their cheapest server space costing about $10ish a month. Rackspace may be a bit easier because you don’t have to deal with ssh keys, so perhaps start out with that.

You can also use webfaction, which is about $10 a month. What’s awesome is they do a lot of the server setup for you. And you won’t really have to do much server setup, which I explain in a few grafs. So, webfaction is an option, so are many others.

My Django tutorials walk through some of the software you need to install if you’re making a Django-y project, but you basically need server software for your server, despite what you’re coding is. Apache is the standard one many people use. It’s open source and nice. I’m a bit fancier, so I use nginx. Whatever you prefer, it’s just another tool to get the job done. It works alright with PHP/MySQL.

BUT HOW DO I CONNECT TO MY SERVER?

This is where you learn about FTP, SFTP and SSHing into your server. FTP and SFTP are two ways of uploading/deleting/changing files on your live server. SFTP is much more secure than FTP, and other news developers will make fun of you if you use FTP. It’s not necessarily wrong, it’s just not as safe. And if you’re learning, no reasons to not use FTP for now.

But before that, you need to know how to ssh into your server. This involves using the terminal, which is scary for some folks. It’s all text, with no shiny buttons to click. Yet this is the most powerful part of your computer. If you’re using Windows, download PuTTY and scroll down in this link. If you’re using Apple products, then search for Terminal and open it then follow the instructions sort of here. If you’re using Ubuntu, why are you reading this?

Before you can FTP, you need to set up software on the server to let you FTP in. Here’s a quick walk-through of installing vsftpd, which is FTP software. Boom. Now you can FTP into shit. You’re welcome.

SO WHAT ABOUT THESE FRAMEWORKS?

Frameworks are sweet. They make it easier to quickly develop the web. I use Django, like I previously said. If you want to get the intense walk-through of setting up Django — INCLUDING LEARNING ABOUT SERVERS! — you can go through my almost-finished tutorial starting here. You can also go through the creator of Djangos’ own tutorials here.

If you want to learn more about Ruby (but not making websites), the wonderful Dan Nguyen has some great walk-throughs here. This could be a good Rails walk-through. Who knows.

So that’s some basic info on frameworks. If you want more, use The Googles.

BUYING DOMAINS AND STUFF

If you’re going to set up a website, you need to learn how to buy a domain. My pal Emily Ingram walks you through how to use GoDaddy.com to buy them at this page. That’s what I use, but I would definitely suggest you do NOT use them for hosting. Do not do not do not. You can use other service providers to buy .coms, but hey, it’s quick and easy.

Just remember to set your .com’s DNS settings to point toward your server, which is stuff you get to learn on your own, because I am getting tired from explaining the entire internet to you, my dearest friend.

MORE FRONT-END STUFF LIKE CSS/HTML

Now, you may have learned that HTML/CSS stuff. But guess what? There’s more! Javascript and Ajax are neat.

Just so you’re aware, lots of places are looking for people to be kick-ass at Javascript and CSS/HTML. They call this front-end development, because it mostly deals with stuff that happens on the client side, aka in your browser. My focus, and all that stuff about PHP and Django and servers and whatnot, is backend development.

Javascript, and its widely used library jQuery, are what lots of the cool internets are built out of. When you click on something on a webpage, and it moves, or changes color, or slides around, usually that’s Javascript/jQuery. These are things totally worth learning. Basic Javascript tutorials can be found here. Here’s the page for jQuery, which also includes tutorials. Here’s even more in-depth stuff for jQuery. A good Javascript book I’ve been going through is Head First Javascript.

Then there’s Ajax, which basically allows you to reload stuff on the page by hitting the server without having to actually load a new page. Think about how when you’re on Facebook and it pops up, saying your Aunt Glady Crapplebottom just liked that photo of those new shoes you bought. That’s Ajax. Learn about some of it here.

These are skills I need to get better at, as they allow you to make awesome web graphics. People who are good at this stuff make me envious, because it’s like they can somehow do magic. Wouldn’t you like to know magic? Yes you would. So figure it out, folks.

SO NOW YOU OBVIOUSLY KNOW EVERYTHING

Okay. This was nowhere near an exhaustive list. I apologize. I didn’t realize what a monumental task this would be. I know I’ve forgotten stuff, so hopefully nice/mean people will write in the comments about other sites, and I’ll update this post with their wonderful suggestions.

If you’re going into this, it’s also good to keep up to speed on the goings-on in the tech world. You can do that by checking Hacker News every once in awhile. It’s a pretty good site. Another good place to look for answers to questions is Stack Overflow.

Get an account at github and start looking at other people’s code. That’ll be the best way to learn, honestly. And odds are if you’re going to make a project, someone else has already made something similar and they’ve put up free code for you to use. So search for it, figure out how someone else made the similar project and incorporate it into your own.

Join the NICAR listserv. This is the National Institute for Computer-Assisted Reporting. Lots of news developers lurk on the NICAR-L, as we call it. And you get to join the Investigative Reporters and Editors, or IRE, which is also a great organization that you should already be a member of. Why aren’t you? JOIN TODAY.

Also, the most important thing you need to learn is that someone has probably had the same error or problem you’re having while building something. So either you can read the official docs — something I am known to not do and then Derek Willis yells at me — or just Google the error. Sometimes people post similar questions/answers on various message boards or Stack Overflow. If all else fails, tweet at some of us news developers on Twitter.

Most of us learned what we learned through the helpfulness of others, so we’re glad to help others learn more.

HOLY CRAP YOU MADE IT THIS FAR? NOW WHAT?

First off, crack open a beer, because you’re probably thinking OMGWTFLOL THAT IS SO MUCH CRAP. Well, yeah. Making the internet is kind of hard. Hell, it is hard. But it’s a little less emotionally draining than most traditional journalism jobs. And the first time you get something to work, I guarantee you will throw your arms up and shout, “YES!”

Every time I make something work, I feel as though I have reinvented fire. It’s a great feeling, and it’s especially great when you can show someone else the tools of the trade. So if you decide to go down this path, please be vocal about what you’re working on. Write about it. Explain how you did stuff. Be social.

So here’s what you do. Come up with a project. It can be as simple as making a website that tracks the movies or books you own. (FYI, I already made that. See here.) You can even go more advanced, perhaps making something that scrapes all the legislative votes in your state. Hell, go even super easy: Just learn how to use Google Fusion Tables to make something like this.

The point is, go out and make something. Then let us know how you do it. If you get stuck, ask us for help. We’d be glad to see you kick some ass.