Jonesieblog

In this series of posts, I’m going to show how I am customising the excellent “Planet Reader” application to produce a stand-alone program that updates data about a collection of blogs stored in a Ruby on Rails application - the feeds table looks like this:

create_table “feeds” do |b|
b.column “link”, :string
b.column “title”, :string
b.column “description”, :string
b.column “pubDate”, :datetime
b.column “language”, :string
b.column “error_tag”, :integer
b.column “site_url”, :string
end

Here’s how Planet Reader works now:

1. get “planet” information and a list of feeds from a config.ini file
2. create a “planet” object based on the config.ini data
3. visit each of the feeds using the Universal Feed Parser
4. If the feed has changed since the last visit, update the information stored in a cache file
5. “subscribe” the “planet” object to this feed
6. churn out static html files based on all the data.

For my purposes, the “planet” object is irrelevant, but I’ll leave it alone, as it helps the whole thing to hang together! My strategy for customising Planet Reader is as follows:

1. get database information from database.yml
2. connect to database and grab a list of all the feeds
3. export this list to a config.ini file
4. get “planet” information and a list of feeds from that config.ini file
5. create a “planet” object based on the config.ini data defaults
6. visit each of the feeds using the Universal Feed Parser
7. If the feed has changed since the last visit, update the information stored in a cache file and also update the database
8. “subscribe” the “planet” object to this feed
9. churn out static html files based on all the data.

Simple eh? Unfortunately I’ve never done any serious Python programming, so I’m learning as I go! To begin with, I’ll be editing the “planet.py” file, which is the file that gets executed to run Plant Reader. You can download the files here if you want to play along, and will, of course, need to have installed Python! I’ve been using the Eric IDE, which is a simple “apt-get install” on Debian.

Getting database information from database.yml

A bit of googling turned up PyYaml, which does the job. Once I’ve installed that, I just need to add:

import yaml
environment=”development”
dbconfig=open(”../config/database.yml”,’r') # we are in lib directory
config_data=yaml.load(dbconfig.read())
host=config_data[environment]['host']
username=config_data[environment]['username']
passwd=config_data[environment]['password'] or “”
database=config_data[environment]['database']

Getting a list of feeds from the database

Now I just need to connect to the database and pull out a list of feeds. First, install the Python-MySQL library. Planet Reader identifies feeds by their URLs, but of course my database will have an id field for each feed, so we’ll pull that out too:

Con= MySQLdb.connect(host,username,passwd,database)
Cur=Con.cursor()
Cur.execute(”select id, link from feeds order by ‘id’ “)
Results=Cur.fetchall()

Putting the feed data into a config file

This turns out to be pretty easy. The ConfigParser library has already been imported, and it does writing as well as reading. The variable config_file has already been set to default at “config.ini”, so:

config=ConfigParser()
for f in Results:
  config.add_section(f[1])
  config.set(f[1],”id”,f[0])
  output=open(config_file,’w')
  config.write(output)

does the job!

I wrap all this stuff into an “update_config_from_database” method, comment out the bit of code that insists upon some “planet” information in “config.ini” and I end up with Planet.py

Running this will produce a set of cache files and static html pages based on all the feeds in my database.

This is pretty cool, but I haven’t written anything back to the database yet. That’s what “Part 2″ will be about!

Stumble it!

Apologies to those of my readers who are not web developers - what follows may well make no sense to you whatsoever!

I am so please with myself! For nigh on a year now, I have been aware of a potential problem with FreeMIS (the Ruby-on-Rails based  management information system for schools that I have developed) and today I solved that problem.

The problem is that the printing of a full set of reports for all the pupils in a year group is a huge task, which can take Rails anything up to 130 seconds to complete. By the time it has done this, the browser and or web server have got bored and given up waiting. I have played around with the configuration of Apache and FastCGI, and tried using mongrel instead, but none managed to fix the problem 100%.

In PHP, it would be easily solved by purging the data produced out to the browser repeatedly through the task, but Rails does not have this option.

Last night as I was drifting off to sleep I realised that I could solve the problem by doing the reports one pupil at a time!

Obvious, you might think, but the system needs a “one-click” solution for printing all the reports for a register class - it would not be acceptable to require user input to print out each individual pupil’s reports.

What I realised last night was that if I did the printing of reports via Ajax and RJS, then I get exactly what I wanted.

I start by submitting a form via Ajax to print the reports for a register class.

The action renders the reports for the first pupil in the register class at the bottom of the relevant div, then puts a hidden variable into the form, with the id of the next pupil in the register class. It then tells the browser to “click” on the submit button of the form. All the reports for the register class then come out, pupil by pupil, and no individual browser<->server cycle lasts more than a couple of seconds, so no chance of timing out. I can even put a percentage progress bar at the top of the page. Happy happy joy joy

Stumble it!

I finally got ’round to buying the new edition of this classic manual for Ruby on Rails web development.  It covers everything in Rails 1.2, so there’s the new <% form_for %> blocks, integration testing and much much more.  If you are thinking about getting into Rails, buy it now!

Stumble it!

The Scotedublogs wiki has become the de facto place to find Scottish edubloggers or to announce your own blog’s arrival on the scene. It has done a great job, but is beginning to outgrow its current wiki format.

One key reason why our blogging community has flourished where so many previous attempts to develop an on-line community of teachers have failed is this: we built and continue to build the community ourselves.

I am therefore delighted to be able to say that we are continuing this tradition by building our own exciting new hub for our community: I am working with Peter, John and others from the Scottish edublogging community to produce www.scotedublogs.org.uk

This site already provides the following:

• A directory of Scottish edublogs, viewable by authority or by tags
• The facility to add new blogs
• Colour-coding of blogs based on recency of last post (red=hot!)
• A quick view of recent posts for each blog
• The facility to add tags to blogs
• The facility to export an OPML file of any of the lists of blogs you view - this file can be imported into a feed reader so that you instantly add all the chosen blogs to your reader.
• An overall view of what’s new via a dynamically updated list of the latest posts from any of the blogs in the system.

In the very near future you will be able to add blogs and [it does this now] create your own custom feeds in the site for use in your feed reader. For example, you could create a feed of all the posts from maths blogs, or the feeds from all Glasgow pupil blogs.

The site will also produce statistics to analyse what the community is talking about, how much we are talking and how often we are posting.

The current site is still very “beta”, but is already providing a useful resource. More importantly, it is demonstrating the power of the Open Source development model by utilizing the combined talents of many members of our community to produce a tool that will help the community to flourish and grow.

If you would like to help with this project, please join the discussion group. The development process is completely open, and we are keen to get input from as many members of the community as possible. Check out the site, and let us know what impresses you, what you would change and how you can help.

If you would like to see the code that makes it all happen, you can visit the repository. In fact, if you so desired you could grab a copy of the code, set up your own version of scotedublogs and change it to suit yourself. This is what Open Source means - it’s much, much more than just being zero-cost, or sharing your data via feeds.

Stumble it!

Every now and then I see an article about an alternative implementation of some aspect of Ruby on Rails in another language. The latest is about Propel, an ORM layer in PHP. Let’s compare and contrast:
Propel:

$dish=new Dish();
$dish->setName(”Beans on Toast);
$dish->setDescription(”Yummy”);
$dish->save();

Ruby on Rails:

dish=Dish.create(:name=>”Beans on Toast”,
:description=>”Yummy”)

Propel:

$c = new Criteria();
$c->add(AuthorPeer::FIRST_NAME, “Leo”);
$results = AuthorPeer::doSelect($c);
foreach($results as $author) {
print “Author: ” . $author->getLastName() . “,
” . $author->getFirstName() . “\n”;
}

Ruby on Rails:

Author.find_by_first_name(”Leo”).each do |author|
puts “Author: ” + author.lastname +”,” + author.firstname
end

In the last example, Ruby on Rails would of course delegate the outputting of author details to a view - I put the “puts….” line in to illustrate how much more intuitive the ActiveRecord syntax is. No disrespect to Propel - it is clearly a major piece of work, has features that are not present in ActiveRecord and could make a huge difference to those obliged to code in PHP. But I know which I’d rather be using! The problem is that PHP is just plain ugly compared to Ruby. I used to use PHP, but I can’t really understand why so many people are still using it. Inertia I guess. Tell me if I’m wrong!

Stumble it!