Publishing a Music Feed from iTunes

When I was creating this site, one of my goals was to expose things that reflect my personality. One way to do so was to include items that reveal, to an appropriate degree, the course of my daily activities. To that end, I've included links to my Flickr photos, Twitter tweets, and to what I refer to as my iTunes music feed. This article will explain how I implemented that last item: getting my play history from iTunes to this website.

The evolution of an idea

Software development is an evolutionary discipline. Only rarely does the first approach completely meet the requirements of the problem space, and, more importantly, the "standard of elegance" expected by a good engineer. This is especially true when the project requirements are ill-defined, or the project is ad-hoc, such as this was.

When I began the project I had a rough idea of what I wanted to accomplish, but I hadn't considered how to implement it. My project requirements were simply to get the recently played tracks from iTunes into a format I could use to expose that data on my website.

At the outset, I knew iTunes kept a version of its library files available for use by third-party use. This is, on the Mac, found at ~/Music/iTunes/iTunes Music Library.xml. I peeked at the format of that file, and thought to myself, "this will be easy." All of the data I wanted to access was there, in Apple's XML plist format. This is my journey from idea to workable implementation.

Attempt #1: Try available software

My first thought was that someone out on the interwebs would have already cracked this problem. (Software engineers are lazy. Why "reinvent the wheel?") So I did a quick search and found a module called pyItunes. I installed it and whipped up a quick script, based on the sample code.

from pyItunes import *
from datetime import *
import time

pl = XMLLibraryParser("/Users/travis/Music/iTunes/iTunes Music Library.xml")
l = Library(pl.dictionary)

now = datetime.today()
one_day_ago = now - timedelta(days=1)
for song in l.songs:
    if song.last_played is not None:
        date = datetime.fromtimestamp(time.mktime(song.last_played))
        if (date >= one_day_ago):
            print song.name

I was rather proud of my quick progress. And then I actually ran the script, and realized it wasn't going to work. At all. It was very slow.

Attempt #2: Parsing the library as a .plist

Then I thought I'd try to parse the plist myself. I looked for a nice, friendly method to parse that data, and found Python's plist library. In short order I whipped up a script to read the data I wanted.

from plistlib import *
from datetime import *

pl = readPlist("/Users/travis/Music/iTunes/iTunes Music Library.xml")
songs = pl['Tracks'].values()
songs.sort(key=lambda s: s['Play Date UTC'] if 'Play Date UTC' in s else datetime.min)
recently_played = songs[0:10];
writePlist(recently_played, '/Users/travis/Desktop/recently_played.xml')

Again, it was easy to write the script, but it was, again, far too slow to be practical.

You software engineers can probably look at the above code snippet and realize why without much trouble. For the rest of you, let me explain. The script is quite straightforward. It simply parses the iTunes library to get a list of the songs, sorts them by the play date, and writes he most recently played 10 songs to another XML plist. The keyword in the previous sentence is simply. The script worked, but it was S-L-O-W, taking about 90 seconds to run. Yikes!

Attempt #3: Use XPath to reduce the problem set

On seeing the failure of my first two attempts, my instinct was to reduce the size of the problem. I turned to XPath, an XML-processing technology that allows access to just the parts of an XML document that are wanted. It turns out that Apple's XML plist format is pretty unfriendly for a lot of standard XML technologies. It's just, well, "special." I eventually arrived at a workable XPath query to get just the song elements from the library, thinking it would be faster to just deal with those, rather than parse out all of the playlists, etc. as well.

Enter Amara, a very capable, flexible Python library for working with XML.

from amara import bindery, xml_print
from amara.bindery.model import *

library_file = '/Users/travis/Music/iTunes/iTunes Music Library.xml'
xpath = "//preceding-sibling::string='Tracks'/dict/dict"

for song in bindery.pushbind(xpath, source=library_file):
    print frag

That worked to get the songs, but it wasn't significantly faster. And it hadn't even sorted the songs by date yet. Sigh.

Eureka! Using the right tool for the job

Although each of my previous solutions worked, none provided the level of performance I wanted. To be fair, this was not the fault of Python or of the libraries I had used. It turns out that my iTunes library file is pretty large, and my scripts approached the task in a "brute force" method that did not scale well. Reading the entire (large) library file in order to search out the recently played songs was untenable.

I considered other approaches, such as using a SAX parser, which would only parse one XML node at a time, reducing the memory footprint considerably. I had even started work on a script using a SAX parser when I made some important realizations.

The first moment I realized I was using the completely wrong approach occurred while examining the XML output of one of the previous scripts to explore further ideas. I noticed that the list of the latest songs was not accurate. I looked into the issue and discovered that XML version of the iTunes Library is not constantly synched. It only saves either periodically(?) or on quitting iTunes. Doh! Do your homework before starting to write your software implementations, kids!

Despite this frustration (and embarrassment) of this discovery, it freed me from pursuing that particular avenue of exploration and forced me to "think outside of the box" I had made for myself. I soon realized that iTunes itself held the solution: smart playlists.

I created a smart playlist, titled "Recently Played", modeled after the "Recently Added" playlist that is included with iTunes.

Recently Played smart playlist rules

I then wrote an AppleScript to read that playlist and export the tracks to a file, in JSON format. (I'm definitely not an AppleScript expert, so don't judge too harshly.)

#!/usr/bin/osascript

property NewLine : "
"

-- JSON format string constants
property JSONArrayStart : "[" & NewLine
property JSONArrayEnd : "]" & NewLine

property JSONDictStart : "{ "
property JSONDictEnd : " }"
property JSONRecordSep : "," & NewLine

-- The playlist whose tracks will be exported.
property PlaylistToExport : "Recently Played"

-- The output file.  Duh.
property OutputFile : "tmp:RecentTracks.json"

on run

    if appIsRunning("iTunes") then

        tell application "iTunes"

            set the_tracks to tracks of playlist named PlaylistToExport
            set the_tracks_ref to a reference to the_tracks
            set num_entries to the count of the_tracks

            -- Assemble the JSON dict for each track.

            set output to JSONArrayStart

            repeat with i from 1 to num_entries
                set aTrack to item i of the_tracks_ref

                local tname
                local tartist
                local talbum
                local trating
                local trackRepr

                set tname to the name of aTrack
                set tartist to the artist of aTrack
                set talbum to the album of aTrack
                set trating to the rating of aTrack

                set trackRepr to "  " & JSONDictStart & "\"name\": \"" & my replace_chars(tname, "\"", "\\\"") & "\", \"artist\": \"" & my replace_chars(tartist, "\"", "\\\"") & "\", \"album\": \"" & my replace_chars(talbum, "\"", "\\\"") & "\", \"rating\": " & trating & JSONDictEnd

                if i < num_entries then set trackRepr to trackRepr & JSONRecordSep

                set output to output & trackRepr
            end repeat

            set output to output & NewLine & JSONArrayEnd

            -- Write the output to a file.      
            try

                set f to (open for access file OutputFile with write permission)
                set eof of f to 0
                write output to f as «class utf8»
                close access f
            on error
                try
                    close access f
                end try
            end try

        end tell

    end if -- iTunes is running

end run

-- Replaces the srch string in the provided txt with the repl string.
on replace_chars(txt, srch, repl)
    set AppleScript's text item delimiters to the srch
    set the item_list to every text item of txt
    set AppleScript's text item delimiters to the repl
    set txt to the item_list as string
    set AppleScript's text item delimiters to ""
    return txt
end replace_chars

-- Reports if the application of the provided appName is currently running.
on appIsRunning(appName)
    tell application "System Events" to (name of processes) contains appName
end appIsRunning

This approach works! It's fast (a second or so), accurate, and relatively efficient.

The last step was to set up a cron job to periodically run the AppleScript and send the output file to my web server. With those pieces in place, the track listing on my site is accurate within about 20 minutes (the delay is due to the cron job interval and web content caching)--Good Enough™ for a personal web site.

Feel free to use any of the scripts in this article or to send me feedback if you come up with a better approach.

Comments

  1. At Jan. 15, 2010 @ 7:37 a.m. Travis Lehman said:

    I am quite jealous of your domain name, my friend!

  2. At March 18, 2010 @ 4:43 a.m. Aviator said:

    Hi Tavis,
    Nice post!
    I am trying to parse itunes music library using SAX parser.
    Here is my code . Not really able to achieve at printing album names.
    public class SAXParserExample extends DefaultHandler{

    List<Song> myTracks;

    private String tempVal;

    //to maintain context
    private Song tempTrack;


    public SAXParserExample(){
    myTracks = new ArrayList<Song>();
    }

    public void runExample() {
    parseDocument();
    printData();
    }

    private void parseDocument() {

    //get a factory
    SAXParserFactory spf = SAXParserFactory.newInstance();
    try {

    //get a new instance of parser
    SAXParser sp = spf.newSAXParser();

    //parse the file and also register this class for call backs
    sp.parse("C:\\iTunes Music Library.xml", this);

    }catch(SAXException se) {
    se.printStackTrace();
    }catch(ParserConfigurationException pce) {
    pce.printStackTrace();
    }catch (IOException ie) {
    ie.printStackTrace();
    }
    }

    /**
    * Iterate through the list and print
    * the contents
    */
    private void printData(){

    System.out.println("No of Tracks '" + myTracks.size() + "'.");

    Iterator<Song> it = myTracks.iterator();

    while(it.hasNext()) {
    System.out.println(it.next().getAlbum());
    }
    }

    //Event Handlers
    public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
    //reset
    tempVal = "";

    if(qName.equalsIgnoreCase("dict")) {
    //create a new instance of employee
    tempTrack = new Song();

    }
    }
    public void characters(char[] ch, int start, int length) throws SAXException {
    tempVal = new String(ch,start,length);
    }
    public void endElement(String uri, String localName, String qName) throws SAXException {

    if(qName.equalsIgnoreCase("dict")) {
    //add it to the list

    myTracks.add(tempTrack);

    }
    else if (qName.equalsIgnoreCase("key") && tempVal.equalsIgnoreCase("Name"))
    {
    if(qName.equals("string"))
    tempTrack.setName(tempVal);
    }
    else if (qName.equalsIgnoreCase("key") && tempVal.equalsIgnoreCase("Artist"))
    {
    if(qName.equals("string"))
    tempTrack.setName(tempVal);
    }
    else if (qName.equalsIgnoreCase("key") && tempVal.equalsIgnoreCase("Album"))
    {
    if(qName.equals("string"))
    tempTrack.setName(tempVal);
    }
    else if (qName.equalsIgnoreCase("key") && tempVal.equalsIgnoreCase("Play Count"))
    {
    if(qName.equals("integer"))
    tempTrack.setName(tempVal);
    }

    }

    public static void main(String[] args){
    SAXParserExample spe = new SAXParserExample();
    spe.runExample();
    }

    }
    Could you provide any help?

Have any thoughts about this post? Add your comment.