Latest Updates: Afrigator RSS

• 

  Pubsubhubbub on Google appspot

  Stii 7:36 am on March 11, 2010 | Comments: 2 Permalink | Reply
  Tags: Afrigator, google (8), PHP (7), pubsubhubbub (3), pushpress (2), Python (13), Wordpress (13)

  It is the weirdest thing. I was having issues subscribing to blogs that is using pubsubhubbub.appspot.com using a Python script. It returned a 500 Internal Server error without fail. So I rewrote the process in PHP and like magic, it is gone… I used PHP and CURL to subscribe to the appspot service. In Python I used the urllib2 library. Somewhere in the back of my mind I vaguely remember appspot not liking urllib2, but I haven’t really checked due to time constraints. I’ll have a look soon, but if you’re going to use Python to subscribe to feeds, I would suggest trying to do so with CURL.

  The good news is that all is good on Afrigator now. All blogs that are using some form of Pubsubhubbub service will be realtime in no time.

  If you’re running on Wordpress (not Wordpress.com) and you have not yet installed PuSHPress, please do so soon!

  Welcome back! You should subscribe to my RSS feed here.
  You should follow me on Twitter here
  You should follow me on Gatorpeeps here.
   

  Reply Click here to cancel reply.

  Name (required)

  Email (required)

  Web Site

   
• 

  Realtime feed updates with Pubsubhubbub on Afrigator

  Stii 8:34 am on March 10, 2010 | Comments: 7 Permalink | Reply
  Tags: Afrigator, pubsubhubbub (3), Wordpress (13)

  Okay, I’ve implemented this beast. Feeds that support Pubsubhubbub functionality will now be able to distribute their posts as they are published to Afrigator. Well, all but Blogger or Blogspot feeds as I’m having a weird issue with pubsubhubbub.appspot.com which is the hub for Blogger/Blogspot blogs.

  If you are using Wordpress.com (i.e. yourname.wordpress.com) your good to go. If you run your own installation of Wordpress, I strongly suggest that you install the PuSHPress plugin. Why? Here is a good couple of reasons:

  1. It get you more exposure. Your latest post would be published immediately. This results in your content being distributed as they’re published instead of hours later.

  2. It saves you bandwidth. We don’t need to check your entire feed every hour. We sit back and wait until you publish a blog post.

  3. We’re not the only ones doing it. Other services and aggregators also uses Pubsubhubbub. Services like Friendfeed, etc. Which means you’ll save even MORE bandwidth and your content would be even more realtime.

  Okay, I’m still having issues with Blogspot blogs. They use a Google Appspot application to enable the realtime publishing of their feeds. However, when I try to subscribe at pubsubhubbub.appspot.com it always returns a 500 Internal Server Error and I’m not sure why. I tried setting various headers as I thought that maybe it is because I’m using a Python script to subscribe, but to no avail. I checked using the service Hurl and that works every time without fail. I’m a bit frustrated and lost. If anyone have any ideas, I’d welcome it.
   
• 

  Dare you to say Pubsubhubbub when drunk

  Stii 4:41 pm on March 9, 2010 | Comments: 1 Permalink | Reply
  Tags: Afrigator, pubsubhubbub (3), pushpress (2), Wordpress (13)

  What a crazy name! Anyway, the idea is much cooler than what the name is. Basically, what Pubsubhubbub does is it enable your blog to broadcast your new blog posts as you publish them. Instead of aggregators like Afrigator periodically fetching and processing your feed, you tell Afrigator “HEY! Listen up! I’ve written something new and here it is.”

  A lot of services uses it nowadays and it is gaining traction. If you’re on blogger/blogspot or wordpress.com you needn’t worry about it as they have already enabled it for those platforms. If you are on a self hosted wordpress blog, I suggest you check out the awesome wordpress plugin called PuSHPress. Install it now! It would be worth it ;)

  Oh and here is a video explaining it in a cool way.

  Note: I’m going to churn quite a bit of posts to test our Afrigator implementation of Pubsubhubbub.
   
• 

  The future of newspapers online

  Stii 7:44 am on October 29, 2009 | Comments: 3 Permalink | Reply
  Tags: Afrigator, news (2), newspapers, online news, the hub

  Newspapers love to hate the internet. It sees it as its nemesis and rightly so, since it has such a free and sharing culture. No news are safe. It can easily be copied and redistributed by other users making it very difficult for newspapers online to monetize the news. They can try and go the legal route, but that has a more negative result.

  

  Photo credit: http://mcfarlaneusa.wordpress.com/

  In my opinion, there are two ways newspapers can monetize their online offering.

  The first obvious way is advertising. Yes, it may not rake in the cash the same way that print ads did in the past, but that does not matter as it would bring in a little money and it would greatly influence the second choice.

  The second option is through a subscription model. Before you get your knickers in a knot and shout “dumb idea, would never work!”, hear me out.

  Gerd Leonhard often says:

  Stop trying to sell the copy. The copy is worthless. Sell the things around the copy.

  This is why a subscription model might just work, but news sites would have to get very, very clever. They would have to build services around news that would make people pay for the service. Sorry, but the news on it’s own is simply not enough. How would they do it? I have a couple of ideas. I’ll keep them to myself for now, since, well, some of it will be implemented in Afrigator in due time, so be patient! One feature would be to remove all the flashy ads if a user subscribes. Making the first option (ads) complimentary to a subscription model, won’t you say?

  I love The Hub and the concept behind it. It is a step in the right direction and it is a very real experience that could possibly enhance your news consumption. It would be possible monetize at some stage.

  Maybe if the papers do it clever enough, they could release full feeds instead of partial feeds so we could try and build onto the value add… Just a thought. What do you think?
   
• 

  RJ van Spaandonk, common sense of a garden gnome?

  Stii 10:11 am on October 6, 2009 | Comments: 5 Permalink | Reply
  Tags: Afrigator, Apple (2), core group, rj van spaandonk

  Tech Central recently did an interview or “personality profile” of RJ van Spaandok, Executive Director of the Core Group. The Core Group has the monopoly on Apple and Nintendo products in South Africa, just in case you did not know. I don’t think people are refuting the fact that RJ is a learned business man. He has a MBA from my esteemed ex-employer Wharton School of Business. Like he says:

  “There are people who have an MBA and there are people who have an MBA. It’s something people in SA don’t want to hear but it’s very different studying at a local school and studying at one of the world’s top universities. If you want to do an MBA, go to one of the well-established institutions, otherwise you’re wasting your time and money.”

  

  (Photo credit: Tech Central)

  It is just a pity that they did not have a class in Common Sense. After the Twitter PR disaster and more recently the article on Tech Central in which he states that he “prefers fine wine to bloggers”, I thought I’d lay out a couple of facts.

  Let me start by stating that my decision to acquire an Apple Mac can directly be contributed to 3 prominent bloggers. Mike Stopforth, Dave Duarte and Justin Hartman. Today I own 2 Macbooks, 1 Mac mini, 1 iPod Touch and 2 iPhones. If I’m not mistaken it was Mike, Justin and myself that also tipped Mark Forrester’s decision to buy an Apple. (Mark, am I correct in saying this?) I can think of a number of bloggers that has invested in Apple Macs and I think I’d be safe to say their decision was greatly influenced by their peers [read: bloggers].

  Thats all good and well, but let’s face it, these couple of sales won’t buy RJ an year’s supply of fine wine. Hardly. So I turned to Afrigator and extracted a couple of statistics regarding bloggers and Apple:

  “Apple Mac” 3372 blog posts written by 754 bloggers.
  “iPhone” 7552 blog posts written by 949 bloggers.
  “iPod” 4344 blog posts written by 1242 bloggers.
  “OS X” 1272 blog posts written by 520 bloggers.

  Not quite so insignificant. Consider that the average Apple user tends to vigorously defend the Apple brand whenever anything is said or whenever Microsoft Windows gets a bit too much exposure. I wrote two bashing Windows 7 posts just last week. RJ, money can’t buy this. Just to put it into perspective:

  “Windows 7” 845 blog posts written by 207 bloggers.
  “Windows Mac” 1474 blog posts written by 551 bloggers.

  Consider that Windows 7 is probably the best operating system ever to be released by Microsoft and Snow Leopard the biggest ball of fluff by Apple, I’d say it would be common sense to nurture the relationship with your product’s greatest fans, wouldn’t you? Instead RJ prefers fine wine.

  Sure, one can argue that the internet does not reach much of their target market. If you’re a garden gnome. Tell me, how many articles was written about Apple in the last 12 months in traditional media? This many? I doubt it. What makes it even more significant is that this is unpaid word of mouth marketing. Yes, RJ, unlike you, we don’t get paid to write. When we say something about Apple it is an honest opinion. Thing is, our words carries a lot more weight with some of our readers and friends than your paid writings ever will. EVER. Should I say it again just for extra effect? Okay. EVER!

  RJ, here is the best advice you’ll ever get for free:

  Put on a pointy red hat, a fake beard and register under the name Fizzcrank Fullthrottle for UCT GSB’s Nomadic Marketing course. It may be worth much more than your MBA in the long run.
   
• 

  That Giant Mistake by The Parlotones...

  Stii 4:31 pm on September 17, 2009 | Comments: 4 Permalink | Reply
  Tags: Afrigator, giant mistake, parlotones, Social Media (4), social stream

  What a band. Nice, down to earth bunch of guys. Not only are they extremely friendly, but they tend to leave a lasting impression. They did for me. Great guys making great music and now making great wine. It is a fantastic wine called Giant Mistake. Smooth dry red wine blend that is easy to drink. Too easy if I have to go by my swollen head today! Well done to Hands on Wine, The Parlotones and Rohan Strydom from A Company Called it.

  

  A lot have been said about the parties involved, but I think I should just highlight what a clever guy Rohan is. A year ago he was a programmer at MIH SWAT and he left to start his own consulting business. He overheard that Kahn from The Parlotones always wanted to have his own wine and that got the gears spinning… He brought them together with Steve Straker from Hands on Wine and 12 months later a Giant Mistake was made. (I love the name! It plays so well.) In other words, we have an IT guy who made a dream come true. Who said we were boring?! Rohan, dude, stay as you are and remember, this is only the start…

  

  Rohan also got Afrigator involved in the project and we did a Social Stream for them to track all the activity happening on selected Social Media channels. Have a look over here.

  Well done to everyone involved in this project and thank you to the Parlotones for a great party!

  

  My giant mistake? Havin way too much of the Parlotones’ Giant Mistake!

  

  Check out Justin and Chris’s overview of the evening:
  http://justinhartman.com/2009/09/17/the-parlotones-make-a-giant-mistake/
  http://imod.co.za/index.php/2009/09/16/private-performance-by-the-parlotones-at-cape-grace-cape-town/
   
• 

  HTTP conditional GET with Python urllib2

  Stii 10:47 am on July 2, 2009 | Comments: 1 Permalink | Reply
  Tags: Afrigator, conditional get, ETag, http, Last-Modified-Date, Python (13), rss (2), urllib2

  When aggregating or reading crap loads of RSS feeds, it makes little or no sense to read every feed every time you check, when most feeds is updated only once a day. To give you and idea, at Afrigator the size of the feeds are half a gig (500 MB), so if you do that every hour you consume 12 giga bytes of data in 24 hours. This simply to get about 2000 new blog posts per day.

  To alleviate load off the system and data transfers, you can do a HTTP conditional GET which basically check the RSS feed’s HTTP headers to see whether or not the feed was updated since the last time you checked and if it was, you’ll process the feed, else just ignore it. It does this by checking the ETag and Last-Modified-Date HTTP header attributes. It also only fetches the headers and not the entire feed, so only a fraction of the data is retrieved.

  ...
  req = urllib2.Request(url)
  
  req.add_header("If-None-Match", etag)
  req.add_header("If-Modified-Since", lastmodified)
  
  opener = urllib2.build_opener(NotModifiedHandler())
  url_handle = opener.open(req)
  
  if hasattr(url_handle, 'code') and url_handle.code == 304:
      return
  else:
      headers = url_handle.info()
      new_etag = headers.getheader("ETag")
      new_last_modified = headers.getheader("Last-Modified")
  
      if new_etag != None and new_last_modified != None:
          store_new_etag(new_etag, new_last_modified, self.id)
  
      #get the content and write to file
      content = url_handle.read()
  ...
  

  If you’re interested to know the more technical aspects of what happens, see this brilliant post. If you plan to build a feed reader at all, you need to use this function. You will not only kill your bandwidth, but everybody else’s if you don’t use it. If you built your own blogging platform, you need to make sure that you add the necessary ETag and Last-Modified-Date headers to your RSS feed. Will tell you next time how to do that. If you are on Wordpress, Blogger or Movable Type it should be fine.
   
• 

  More reasons for Gatorpeeps

  Stii 1:54 pm on May 11, 2009 | Comments: 3 Permalink | Reply
  Tags: Afrigator, gatorpeeps (3), kohanaphp (5), microblogging

  Finally we can talk about Gatorpeeps officially! We’ve been rather quiet about it, primarily since we needed to finish a few things and stress test it before we punted it.

  

  We had our fair share of “Blegh, another Twitter clone…” and we expected it. The thing is, we love twitter! Always have, always will.

  So why did we clone it then? Simple. Some of the reasons Justin already covered over here, but I’d like to add that Afrigator was in need of something social networky. We needed to add a communication dimension and as everybody knows, there is no better way to do that other than Twitter. Yes, we could have leveraged off the Twitter API. That is true, but the Twitter API is not without it’s limitations.

  Besides that, we also have a few other ideas, such as Twitter recently killed SMS functionality in Africa, and it left a huge gaping gap in that market.

  We did not leverage off some Open Source app like Laconi.ca or Jaiku. We wanted to initially, but due to the fact that we want to seamlessly integrate Gatorpeeps with Afrigator and add a bunch of unique features, we thought it best to rather develop our own flavor into the mix. For those interested, it is backed up by the brilliant Kohana PHP framework :). Was it the best idea? The right thing to do? Time will tell.

  Now the real work starts. Now we need to do all these promised features and integration… Fun times!
   
• 

  Afrigator ranking gymnastics

  Stii 3:07 pm on May 6, 2009 | Comments: 13 Permalink | Reply
  Tags: Afrigator, blogrank (3), blogstats (4)

  I had to smile when I read a comment left on Diary of a gay Kenyan:

  Afrigator seems to be quite fond of gymnastics in their rating of blogs.

  Basically, The Gay Kenyan complained that one night he noticed that his ranking dropped suddenly into the fiery pits of Beelzebub. He went from 48th in Kenya to 271. He was deeply shocked by this. He is not the first or only person who complains about this. On the Blogs24 network, we’ve seen countless complaints.

  Better I explain this, but before I do, let me say that they are not wrong. We’re not necessarily doing it right. In fact, I’ll go as far as saying there is not really a wrong or right. It is just the way we are doing it at the moment.

  We’ve got an Afrigator ranking script that we run every 4 hours. (To find out exactly what the Afrigator ranking script does, see this post: Whats in an Afrigator ranking) Thus, every 4 hours, we calculate a new “score” or rank for each blog in our system. Effectively this means that the ranking can change every 4 hours!

  In addition to that, we reset the Unique Visits and Page Views ranking every Sunday night round about midnight-ish. So, if you had a fantastic amount of visitors this week, next week they don’t count anymore as we start afresh.

  In order to maintain a good ranking position, you need to constantly (at least on a weekly basis) have a steady flow of visitors to your site. If you have any suggestions regarding these processes, please feel free to tell us. We’re always open to suggestions! ;)
   
• 

  Switched to Python Fabric

  Stii 11:18 am on March 4, 2009 | Comments: 12 Permalink | Reply
  Tags: Afrigator, capistrano (3), deployment, fabric (3), Python (13), svn (4)

  We’ve been using Capistrano to deploy Afrigator to our various servers for a while now. I can seriously not complain or say anything bad about Capistrano. Thing is, I’m more familiar with Python than I am with Ruby, thus for me it just makes more sense for me to use Fabric.

  Here is a small example of how to write a typical deploy script with Python Fabric:

  First, define your various servers which you need to deploy to. Ideally, you’ll have a test, staging and live server. Thus you’ll set them up as follows.

  def test():
      config.fab_user = 'test_user_name'
      config.fab_hosts = ['test.yourserver.com']
  
  def staging():
      config.fab_user = 'staging_user_name'
      config.fab_hosts = ['staging.yourserver.com']
  
  def live():
      config.fab_user = 'live_user_name'
      config.fab_hosts = ['www1.yourserver.com', \
  'www2.yourserver.com', 'www3.yourserver.com']
  

  This allows you to deploy your code to the various servers. Please note, you need to setup automatic login for your different servers. See here how to do automatic logins.

  Next, write the steps you would take to deploy your site manually:

  def deploy():
      "Deploy code to servers"
      msg = "deploying"
      require('fab_hosts', provided_by = [test,staging,live])
      local('svn ci -m "$(msg)"')
      run('svn export repos /path/to/repository/export/')
      run('cp -R /path/to/repository/export/* /path/to/your/site/')
  

  To deploy to the staging server you can run the following command:

  $ fab staging let:msg="Reason for check in" deploy
  

  To deploy to the 3 live servers, all you do is:

  $ fab live let:msg="Reason for check in" deploy
  

  The command works as follows: fab is the command. live/stating/test are the environments you would like to load. If you said staging, it will do the commands in deploy for the staging server. The last part is the command you want to run. In this instance, deploy.

  The let:msg=”Reason for check in” is the coolest bit! It basically allow you to override Fabric variables. If you look closely, in my deploy script I’ve set a variable msg to just say “deploy”. Now that is a stupid SVN message for a commit. In order to commit with meaningful messages, I override the msg variable with my own message. Simple, yet very effective!

  You may want to write a number of different functions in a single fabfile. If you have 100 commands and you’re not one hundred percent sure, just do a:

  $ fab list
  

  and all the commands available to you will be printed with their description. Fabric rocks! Seriously.