Earlier I wrote a piece about using algorithms to help with your journalism, spurred on by this piece in Wired. Now I want to cover more the area that the piece mentions: Using algorithms to actually write stories.

I’ve actually written something that does this. It’s hosted on Github and I’ve got a tutorial for setting it up. I call it FireTracker.

This isn’t a new idea, and I’ve written about it before. Yet only a handful of news organizations have anything like this set up.

The Los Angeles Times is probably one of the better known organizations that does this. The Homicide Report pulls in a spreadsheet, writes a basic story and posts it. It’s like a mad lib, or so Ben Welsh describes in this awesome video.

Crime L.A. pulls in data and runs automatic analyses of every neighborhood. These were big data projects that news organizations used to do maybe once a year that can now be done as often as you want, as long as you’re receiving the data.

Clearly these projects are worthwhile — they are good journalism, they save journalist’s time and they give our audience more information and context. So why aren’t we doing them?

I would say the main problem is our newsrooms don’t have the bodies or the technological know-how. But in rebuttal, I bet 150 years ago we didn’t have many photographers or dark rooms at our offices, either. That didn’t stop places from setting up photo staffs, did it?

Journalistic web development is still young, but we’ve been putting our content online for almost 15 years. We need to catch up and start developing a product online that matches the times we’re in. This is not going to happen at our schools — although they will help somewhat — but we will have to teach ourselves.

If you’re a hiring manager and you have a few slots open for reporters, maybe you should look for people who are in the civic coder realm instead, like Code for America. While I’m sure everyone wishes they had a few more reporters or editors, I would contend it’s a better business decision to invest in more web developers.

I view most of journalism as solving problems. You need to identify the guy was who was murdered, you need to know how much in donations that candidate received this quarter, you need to find an eye-witness to some awesome event. A web developer is just another person who can help you solve problems. Wouldn’t you want more around?

This morning a story from Wired made the homepage of Hacker News. It was about a company that can take in data, run it through a program and spit out a basic news story. The company has been written about before, and when it first bounced around journalism blog-o-circle-sphericals I had meant to write something about it.

This company is doing what we should be doing more of: Automation. Journalists tend to do a lot of work on daily tasks that could be automated. I’m going to give you an example from my younger days as a reporter, something I’m sure many journalists can relate to.

It was early 2007, a year before that photo up there was taken. I was a cops reporter at the Daily Nebraskan, my college newspaper. Three or four times a week I would head to the city/county building complex for crime information. First you would look through a a stack of overnight reports from the sheriff’s office and you’d note any interesting cases. Then the Lincoln/Lancaster County morning media briefing would begin.

Officers would hand you a print-off with everyone arrested overnight and a brief summary (less characters than a tweet) of the crime committed. The assembled reporters — from television, radio and print — would comb through the thick stack and ask questions about interesting crimes. The public information officer would look things up, read off arrest reports and give us information (They often had to do this for legal reasons, if I recall, because open investigative documents aren’t public. Hence the need for someone to read it to you). Sometimes the police chief would appear and tip us off to stories or give context. Someone from the sheriff’s office would wander in and do the same.

My job was to do normal stories that affected campus and students. But I also had to scan the list of everyone arrested and check for anyone who was 18 to 22, prime college ages. Mondays briefing was huge because it included information from Friday night through Sunday, prime collegiate boozing time. I would head back to the newspaper and start typing these names into the unl.edu student directory and Facebook, seeing if I got a match. If I did, I’d circle them on my list.

Then I would sometimes hand the list to the sports department and have them scan the names for “important people,” namely athletes. If there were any, I’d note the name.  I would then call the public information officer and ask about each case. The officer usually had listed whether they were a student or not, as that would be listed as their occupation. This was a nice way to double-check these were indeed students and thus worthy of our coverage.

I’d take the information and either write up individual full articles or throw them into a cops brief column, attempting to get comment from the students. I did this daily. So did/does someone at the Lincoln Journal Star. So did/do probably thousands of journalists across the country.

And it’s incredibly inefficient.

Almost everything I did to identify people on the list was something a computer could have done. Instead of an hour of my time, a computer could do it in 10 seconds. It could give me a list of everyone whose name and perhaps date of birth information matched and other information we entered in about them. Then I could still do my reporting and writing, some of which can’t be replaced by an algorithm, but I’d have taken a bit of the hunting and scanning aspect out of it.

These are the steps to follow at your news organization in order to do this:

• Daily data from the police department in a format I could ingest
• Enter the “important names” into a list
• Writing a program to match the names and also find people college-aged

I think the first part would be the hardest. But maybe not in Lincoln, as their police chief was one of the data-savviest people I’ve ever encountered in my life. Tom Casady, now Lincoln’s public safety director, has blogged for years about data and other musings. He once gave me about 10 years of geocoded crime data for free. You won’t find many heads of departments like him, I gather.

Convincing your public safety officials to get you this data will take some time and some tact. You could present it as a cost-saving measure — less printed pages means less cost to their department (unless they’re requiring your news organization to foot the bill, of course). But odds are they are already emailing themselves this sort of data. They just need to add your news organization to the list.

Finding the important names may be something you will have to undertake yourself. From past experience in trying to set up something similar at a previous job in 2009 (wherein I failed to get the project going), you should just take initiative and build a list yourself. Add everyone you can think of, from city/county council members, important business persons, head of local institutions, sports teams, etc.

Lastly, you will need to write the program. This I can’t help you with. Thankfully, at least one news organization already does this and has brains you can pick. Ben Welsh from the Los Angeles Times already did just this. He talks about that and lots of other awesome ideas in this amazing video.

Seriously. Watch that video if you haven’t. Do it now. I’ll be writing a second post on this topic that he also touches on in the video.

What does a project like this do? It takes a bit of the daily grind off your back, allowing you to do other reporting. If I had done this project while I was still in college, I probably would’ve still attended the morning briefings. But even if it saved me an hour a day, that’s five hours a week, or 12.5 percent of your time. That’s a lot. In one year, five hours a week adds up to 6.5 weeks a year.

Wouldn’t you rather have that time to do other reporting? I sure would.

Have you tried this? Would you be willing to try it? What problems do you see that I don’t? Let me know what you think in the comments.

Photo by Jay Carlson, via Facebook.

A little history first: In the early days of News Organizations On The Internet, your content went into a content management system. It was probably clunky (probably still is), hard to customize and maybe even harder to rebrand different sections. Your stories went into this CMS, sometimes through a pagination system, then you added a photo, maybe a video. You clicked a button and poof, story on the internet. But what if you wanted to set up a special corner, call it “Mary the Margarita Maven” and have someone review the town’s margaritas? Not so easy to do with your current setup.

Enter blogs. They were their own content management systems: Moveable Type, TypePad, WordPress, whatever they were called, they made it easy for news organization to quickly throw up different content that maybe didn’t fit the traditional “news” role. Maybe it was opinionated, maybe it was about inside decision-making processes, maybe it was cute cat photos. Or margarita reviews, whatever. These content management systems were, on the whole, like the ones your news organizations used originally, just programmed with different backends and usually easier-to-use. And sometimes this content didn’t go into the newspaper/broadcast — it was “online-only” and meant to entice people to your web product.

Sadly, blogs brought along a stigma that people still use  – which is wrong — that they’re done by people in their pajamas in a basement somewhere. Blogs are not the same as regular news content, some media folks thought, because they weren’t in your “main” CMS. They had a wall between them and they are different. They may even be branded differently, with a different header and logo. They weren’t the same as regular content because they were in a different system! Right?

Wrong.

It’s time to stop bifurcating your content as blogs and news because they run on separate systems. It is all content, so why not call it that? Even if you have outside people writing posts on your website that are unmoderated by your staff — that’s still content that’s part of your media outlet’s website. I don’t have any research proving this, but in my short journalism career many media outlets just slapped the name “blog” on something because it lived in a different CMS. We should stop this. Please.

You have a crime blog? A doggy blog? Maybe a sports blog? Wonderful! I bet you gave it a cool name, like Crime Watch, Fido’s Place and Sports Gal. All you have to do is stop calling it the Crime Watch blog. And the Fido’s Place blog. And the Sports Gal blog. Just say “Check out Crime Watch for more crime coverage.” Or whatever.

If it’s opinion content, call it that. If it’s news content, great! That’s what it is. Start thinking of it all as content as opposed to “this is a blog post” and “this is a news story.” If you copied a news story and pasted it into a blog post, DOES IT SOMEHOW CHANGE? No. It does not.

I know calling things “blogs” has some cultural cachet, as if a blog is somehow more up-to-date than your normal news content. But again, that’s probably because your main content management system is clunky and slow and the new “blog” is super duper fast. That’s a silly reason to distinguish between the content. Did you credit photos taken with digital cameras instead of film in a different way? Of course you didn’t (I hope). You can change people’s thinking on this — why distinguish one part of your content as fast and the other as slow? Stop it.

Let’s work on making our systems more integrated and faster. Help your readers come to realize that all of your content is fast and awesome. It’ll take some rebranding work, sure, but I bet our industry is capable of it.

So embrace your content, people. Give it a hug and a pat on the head and maybe a slice of pizza. Welcome it into your news community. I apologize for any rebranding you may have to do, but trust me, it’s going to be worth it in the long run.

UPDATE: Hey new visitors! Feel free to yell at me on Twitter (@andymboyle) or comment below. I’d love to hear your thoughts. 

In the past few months, we’ve launched What Would Bill Say and a GOP Candidate Tracker, while also relaunching our Patriots Report Card. All three of these projects rely heavily on Javascript and are among the first projects I’ve written in the language. I’m a Django guy, so this front end stuff is scary. The design of these projects were done by my coworkers — I focused on the backend.

Patriots Report Card

This is week 14 of our Patriots Report Card. This was a tool previous developers at the Boston Globe worked on. It runs in Flash and updates a MySQL database. Six weeks ago, we switched it from the previous version. The new version reloads the page when you vote, and so we kept your voting information in the URL instead of saving it within the Flash program.

My coworker, the great Tom Giratikanon, worked on the Flash part. I write the JS that stores your vote in the URL and returns it so the Flash bits can use it, with an assist from Tom.

First a regex I found by Googling splits the various parts of the URL. Then a few variables are given the values from the URL and the SWF file. Lastly, it loads your new votes into a function called reportCardReload, so when you click the next button it reloads the page and keeps your vote information. I know, we don’t have to reload the page. But it’s what I was asked to do and I think it works nicely.

Not exactly the hardest thing in the world, but it was one of my first JS projects I did mostly on my own.

GOP Candidate Tracker

A few weeks ago, we launched our GOP Candidate Tracker. This project was quite difficult, as it uses a lot of brainier Javascript/jQuery and I am still a beginner. Thankfully, the aforementioned Tom and a few internet pals helped walk me through some of the concepts.

First, we have two interns in the newsroom who enter information into Google Docs using a Google Form. Then I download the latest events, paste them into a .csv and run a python script that turns it into JSON. I test it in a fake version of the live page that can only be seen inside the Globe — just to make sure no weird characters got in — and then I paste it into the live JSON file and upload that to cache.

I will be building a Django-based backend for this, so folks will enter the information in there instead of Google Docs in the future. Then the Django app will spit out a JSON file, similar in structure to what we’re already using.

It wasn’t built in Django first because we didn’t have an area to build Django projects at the time. Now we do, thanks to the hard work of our Ops team. So we will be switching that over in the future.

Anywho, this project had to do a few things: Parse a JSON file, sort it in various ways and allow you to click a button and show only a certain candidate (and make that appear as a hash in the URL so you could share the individual person’s dates). I used jQuery, underscore.js, date.js and jQuery-BBQ, the latter built by a former Boston Globe staffer.

What Would Bill Say

As many folks may know, Bill Belichick keeps his quotes pretty bland and quite simplistic. Our sports team asked if we could make something that showed one of his random quotes. I countered with, “What if we allowed you to ask him a question and it answered with a real quote of his?”

This app works like this: You ask a question, it appends the question into the URL, generates a random number and appends that to the URL, too. The random number picks which photo/question to show from a list within the Javascript. It also looks to see if a question has been asked (aka shows up in the URL) and spit out the appropriate page information.  I used jQuery-BBQ again to help me parse the URL and jQuery Form Defaults to help with a few form issues.

My pal Dan Nguyen looked at an early version and found it had some cross-site scripting vulnerabilities because I wasn’t sanitizing my fields. Underscore.js helped me with this.

We wanted to keep everything in the URL so the original question and answer would be easily shareable on social media.

Lessons Learned

These projects mostly do everything inside of the browser, loading flat files (except the Patriots Report Card). While the Candidate Tracker has a wannabe-database backend in a Google doc, it ends up pulling in a flat JSON file. This means I don’t have to maintain an application server, a database server or a caching server. Me gusta. A year ago I probably would’ve set up a full Django stack for each of these projects.

Also, I am no longer scared of Javascript. So that’s a good thing. In the future, I’m going to try to make as many projects with just flat files. I’d also like to start learning more about using Raphael for graphics.

My only resolution last year was to get on stage and attempt to entertain people. I accomplished that goal in these last few months. So I thought expanding my resolutions to three would be doable. A few of these were inspired by the internet.

1. Spend a week eating like a vegan

2. Travel more

3. Buy the majority of my media — books, magazines, movies and television shows — in an electronic format

I once tried to eat vegetarian for a week but forgot on my second day while rushing from one assignment to another.

As for traveling, I don’t leave wherever I live that much. When I do it’s usually to visit my family in Nebraska. I want to take a vacation someday and go see the Pacific Northwest. Hopefully. I hear my beard can be used as currency up there.

And by looking around in my apartment, I own too many books, many of which I won’t read a second time. Same goes for movies. I’ve been going to the library more and I’ve almost finished the latest Steve Jobs biography on my iPad/iPhone, which has been a pleasant reading experience so far. But it’s a bit hefty and I want a product only for reading.

So, I ask you, dear reader: What’s the best device to buy that’s only for reading? I won’t use it to surf the web, check email or do anything else. I just want something that I can buy books with and read them. Let me know in the comments or yell at me at @andymboyle.

We’ve got some people entering in some data in a Google Doc spreadsheet for a project. After fiddling around with attempting to make Google Docs API spit out not-ugly JSON, I said screw it and came up with a new plan: I was going to write a cron job that downloads the CSV from the spreadsheet and then parses it into JSON.

And of course, as a guy who likes Django, I turned to python for the parsing.

First, you will need to remove the first line in the CSV if it had any field names. Put it into a folder somewhere, perhaps. Then make an empty file and name it parsed.json and place it in the same file.

If you’re on a Mac/Unix-y thing, use your terminal and cd into the folder your parsed.json file is. Enter

pwd

in the terminal to see the path to your directory. If you’re on a Windows box, then it’s probably C:\wherever\your\stuff\is\

Now create a new file called parser.py. Enter in the code below, changing /path/to/filename part on line 5 to the bit that was spit out after you typed pwd (or where you inherently found it if you’re in Windows). On line 7, change fieldname1 and whatnot to the name you want each row to have in your JSON file.

import csv
import json

# Open the CSV
f = open( '/path/to/filename.csv', 'rU' )
# Change each fieldname to the appropriate field name. I know, so difficult.
reader = csv.DictReader( f, fieldnames = ( "fieldname0","fieldname1","fieldname2","fieldname3" ))
# Parse the CSV into JSON
out = json.dumps( [ row for row in reader ] )
print "JSON parsed!"
# Save the JSON
f = open( '/path/to/parsed.json', 'w')
f.write(out)
print "JSON saved!"

In your terminal, while you’re in the proper directory, type:

 python parser.py 

BOOM! It should spit out “JSON parsed!” and “JSON saved!” If you wanted to spit out the JSON in the terminal, you could add a line at the bottom:

 print out 

Voila. Now you can turn your CSV files into JSON.

The basic idea is this: Use the data of what people click on with Google AdWords/Microsoft’s adCenter to help you come up with coverage. First, you set up advertisements for a certain area of coverage, say an upcoming convention in your town, or maybe you’re trying to allocate resources between two not-as-often-covered sports teams and not sure which has a greater audience. Then you send those ads to a landing page or article you’ve already written about said topic.

You can then see how many people clicked through the ads, what the bounce rate was, how long they stayed on the article/landing page (using your own embedded analytics, of course) and whether or not they explored more of your website. Then you can deduce whether one subject or another received more page views than the other, all for maybe $100 in advertisement costs (plus the time it took to set up the pages).

Now, say you were planning on hiring a beat reporter to cover one subject over another, I would say that $100 is a cheap price to make sure you’ve made a good decision for your audience.

This may smack in the idea of what I’ve heard some people freak out about, which is catering too much to your audience. Giving them their chocolate cake instead of the peas and carrots they should (says we) be eating. Well, with dwindling resources at many media organizations, I can’t see why it’s a bad idea to gauge your audience before starting a new venture.

Our industry could use more analysis in making decisions, I believe, and thankfully we have a lot of tools at our disposal.

And if anyone is already doing this, I’d love to hear how it’s faired in the comments.

***edit at 10:51 a.m.***

Ted Han has pointed me to this post about how Homicide Watch uses analytics for coverage, which is totally rad. Laura Amico used her site’s analytics as a reporting tool. Awesome.

One of them asked, “Isn’t Siri still in beta?” Yes, yes it is.

I responded that I hate things being called “beta.” The rest of this post will be, hopefully, a more erudite explanation. With less swears. Maybe.

Saying something is still “in beta” is a goddamn cop out. It’s like you’re releasing something and winking, saying, “This could still suck so please be gentle and don’t be mean!” That’s a crock of horse shit. You released a product live, showing it to the world. It is no longer in “beta” mode. It is now in production. Otherwise, why did you release it?

Oh, you say you’re going to add features and fix a few things here and there? Cool. That’s more or less how everything is always done. I don’t think most folks refer to Twitter or Facebook as being in “beta,” as they’re adding features and updates all the time. Most web projects are continually in flux and are constantly being updated. To quote The Social Network, it’s like fashion — it’s never finished.

And to use my journalistic background, this is akin to calling the newspaper the “beta” version of stories, as people can go in and update the online version of articles whenever they want. No, we call it the damn newspaper. Get ‘em while they’re hot.

The phrase has a use, of course: internal testing. Once you release it to the world, though, stop calling it beta. It’s live, goddammit. Be proud of your accomplishments. Building that thing and launching it was hard.

Don’t hide behind a four-word veil of I-hope-this-doesn’t-suck-too-much-itude.

Also, if you think I’m wrong, please explain your arguments below. I’d love to hear a good argument for using beta.