Here are some statistics of desktop aggregators that hit diveintomark.org on November 18, 2003. I measure unique IP addresses (multiple hits only count once); it’s an imperfect measure, but it’s all you’re going to get. I have no idea if these numbers are representative of anything beyond my own site, but they are representative of the traffic on my site. All the recent dates I tested (including weekends) show a similar curve.
| Aggregator | Unique IPs |
|---|---|
| Total | 2063 |
| NetNewsWire | 936 |
| SharpReader | 318 |
| FeedDemon | 177 |
| NewsGator | 157 |
| Radio Userland | 96 |
| nntp//rss | 83 |
| Straw | 46 |
| Amphetadesk | 43 |
| RSS Bandit | 36 |
| NewsMonster | 36 |
| FeedOnFeeds | 30 |
| Syndirella | 29 |
| Shrook | 19 |
| NewsDesk | 18 |
| FeedReader | 14 |
| Awasu | 10 |
| Aggie | 6 |
| EFFNews | 6 |
| Hep | 3 |
I had an inkling of what was going on, but I didn’t really see it until I threw the numbers into Excel and plotted them on a logarithmic curve:
Yep, that’s a power law. R2 = 0.9472, for those who care about that sort of thing. It would be better, except NetNewsWire is actually more popular than the power law predicts the front-runner would be.
Let’s do that thing again, the one where we all bicker in predictable ways. Where people claim, variously, that my numbers are suspect, my methods are bogus, my chart-making skills suck, power laws don’t exist, weblogs are the great social equalizer, Shirky is a dick, and the entire endeavor is hopelessly sexist because all these programs were written by men.
Go.
§
LOL–well, I only was wondering how much weight your recommendation gives the two front runners.
If you started recommending two other aggregators, for, say, three months, would we see the stats shift?
“Those that resemble flies from a distance”
God I love this category. Have I mentioned I love this category?
— Dave S. 
Ken, I’d forgotten those recommendations were there. I doubt they make much difference though; I don’t prominently link to that page anymore and I don’t think I have that much influence on the software people choose in any case. I recommended Amphetadesk for months (there and elsewhere) and it quickly got swmaped by three-pane aggregators when they came out (stats here: http://diveintomark.org/archives/2003/01/02/the_year_in_review )
I also expect these stats will shift rapidly once FeedDemon goes final. It’s just light years ahead of everything else.
— Mark
…my numbers are suspect…
only because this only represents all visitors to diveintomark.org, which are probably very different than the rest of the aggregating population. (a guess).
…my methods are bogus…
looks okay to me.
…my chart-making skills suck…
not going there.
…power laws don’t exist…
feh. what does?
…weblogs are the great social equalizer…
http://www.google.com/search?q=%22*+is+the+great+social+equalizer%22+-education
…Shirky is a dick…
yup.
…and the entire endeavor is hopelessly sexist because all these programs were written by men.
*sigh*.
— kami
You might also want to mention that you have 456 subscribers that view your site through Bloglines. The Bloglines crawler includes the number of subscribers to a feed with the User Agent information, so this info should also be appearing in your web logs.
Good work on the counts. It would be interesting to see what the numbers look like if dynamic IPs were somehow taken into account. Dynamic IPs make getting accurate subscriber numbers for the various desktop aggregators very difficult.
Why is http://diveintomark.org/about/stats/bybrowserdetail.html (and the rest of /stats/) now 401?
— TRS-80
I use NewzCrawler, but I didn’t see it on the list. Why not?
I’ve been prevaricating over whether or not to try out FeedDemon given that Bloglines meets all of my aggregator needs (or at least those I’m aware of). But, having used both HomeSite and TopStyle for extended periods, I admit that it’s impossible for Nick Bradbury to write a less than outstanding software application. So I’ve just downloaded FeedDemon — thanks for the recommendation.
When your stats “shift rapidly once FeedDemon goes final”, will that finally prove that power laws don’t exist?
Hey Mark, I just ran some numbers and they came out way different, but I suspect that I’m missing some obvious things. Could you provide a couple words on methodology? Nothing detailed, just a 50,000-foot view, i.e. do you look in any particular logfile fields or just match against a set of strings or what? I assume you start by restricting to fetches of the feeds…
— Tim Bray
Hmm, disregard the previous, a brute force aggregation of the user-agent field tallied against unique IPs and no error-checking gives me a list that starts like this. Which is unsurprisingly remarkably like Mark’s. Bloglines claims to have 277 subscribers. And BTW don’t underestimate Bloglines, I set my Mom up with it and it does her just fine.
963 NetNewsWire
150 SharpReader
79 FeedDemon
75 Mozilla
75 NewsGator
57 Radio
34 RssBandit
29 Feedreader
24 Straw
22 AmphetaDesk
19 -
17 Python-urllib
14 nntp
14 Shrook
13 Syndirella
11 Java
11 FeedOnFeeds
11 UltraLiberalFeedParser
9 libwww-perl
9 Awasu
7 ActiveRefresh
6 rawdog
6 RssReader
— Anonymous
Out of curiousity, what program did you use to plot the graph and fit the power law? It’s refreshing to see something other than Excel (blech).
— Micah
It’s Excel! You like it, don’t you :)
— Firas
I find it amazing that a Mac client is the most popular. Could this be because your readers are a non-representative sample?
Mark, does Bloglines users count on this? I started to use Bloglines instead compiled feedreaders, Web feedreaders are far better for my job.
— mini-d
Mark,
I think you would be surprised how many people may find that page and then may just go and download it. And then the first think the do is add your feed.
Maybe track all links going out to them that way you can track it and see what the real numbers are.
— Donny
(Lets do that again, the one where you all bicker in predictable ways) OK!
Where’s your error? That’s joining the dots, not a curve! Reverse the x-axis for impact, or switch them around, we (humans) love strong data decentered to the right. Why not try a double-y plot with browsers too? Come on, snap outta of it!
And by the way, when you speak of power-laws and tout numbers (>2000 aggregators), it might have been a good idea to divide your raw data by 10 to minimise penis-envy and follow-up calls of sexism — if any.
— Gummi
Well, the stats are nice, but I have to agree with Gummi: you cannot just connect the dots. Statisticians hate that. ;-)
I wonder if there is a large difference between your ‘web audience’ and your ‘feed audience’? Especially for Mac vs. Windows users? And how much do both overlap?
— Roel
I took out the connect-the-dots and added a real trend line.
— Mark
EffNews Represent!
*Ahem* Who are the other 5? :)
Hey, some of those million NNW users might like to Google MacOS X RSS and see what comes up.
[I have no shame]
— Graham
Does NewsMonster use an agent that cannot be distinguished from common browser agents? There should be at least one NewsMonster hit.
— Claude
I’ve added NewsMonster.
— Mark
Hmm. How about some sort of error estimate on those points. Something like ~ root N for a count of N.
— jgraham
Hmm. I wonder why bloglines (www.bloglines.com) isn’t showing up? I use it, I’m certainly not the only one.
— Larry Wright
“I also expect these stats will shift rapidly once FeedDemon goes final. It’s just light years ahead of everything else.”
I use FeedDemon (RC4a), and I’ve just noticed that your site renders badly when I use FeedDemon to view it. It cropped on the left side. Odd…
— Sérgio Nunes
See? You’ve affected the curve again. Just downloaded FeedDemon after not knowing poop about it. Face it, Sparky… you’re not reporting trends, you’re creating them :).
The way I understand power laws, they are based on freedom of choice. There is no true freedom of choice between all aggregators in your list though as Windows users cannot choose NetNewsWire, and Mac users cannot choose SharpReader, FeedDemon, NewsGator and many others from your list.
And while there is of course also a freedom of choice regarding the system someone uses, this decision is made independent of the aggregator they choose. Nobody would buy a Mac just to run NNW, or a PC just to run SharpReader.
The windows aggregator market is fragmented with many available aggregators competing for market-share. For Mac-users, there aren’t nearly as many options and NNW seems to have the vast majority of this market.
I think this is why NNW falls of the power law scale – it shouldn’t be on the same scale as Windows aggregators in the first place.
This data distribution is an example of ZIPF’S LAW (see http://www.few.eur.nl/few/people/vanmarrewijk/geography/zipf/ or http://linkage.rockefeller.edu/wli/zipf/ for examples) which is named after the Harvard linguistic professor George Kingsley Zipf (1902-1950). This “law” is the observation that frequency of occurrence of some event, as a function of the rank when the rank is determined by the above frequency of occurrence, is a power-law function.
You will find that this distribution occurs in lots of things from the population of cities to word frequencies within text to names of children to web hits. It’s amazing to see how often this shows up. See http://cuwu.editthispage.com/1999/12/15 for an example of how another web site saw their hits correspond to this “law.”
— Tim Rand
“I’ve been prevaricating over whether or not to try out FeedDemon”
That’s odd, Jonathan, you always struck me as very honest.
Mark: why does http://diveintomark.org/about/stats/bybrowserdetail.html now need authentication? I’ve always found it rather interesting to watch the aggregator trends there…
My speculation for the reason that NetNewsWire is at the top of the list is that Mac users tend to be out and active on the net while the majority of PC users tend to use it for e-mail and web browsing and haven’t yet found the RSS scene. For a long time, there have been studies that Mac users tend to use more applications than PC users. The Mac community said that it was because of the common User Interface Guidelines that are followed so that a casual user can pick up a new application with relative ease. My other speculation is that NNW is just SUCH A GREAT APPLICATION!
— Tim Rand
Quick and dirty (no coffee yet):
mnot:~> zgrep /blog/index.rdf /var/log/httpd/mnot-access_log.1.gz | cut -d ” ” -f 1,12- | sort | uniq | cut -d ” ” -f 2- | cut -d “/” -f 1 | sort |uniq -c | sort -gr | head -25
148 “SharpReader
90 “NetNewsWire
85 “Mozilla
35 “FeedDemon
25 “Syndirella
22 “NewsGator
20 “Shrook
13 “Feedreader”
11 “wTicker
8 “AmphetaDesk
7 “RssBandit
7 “Googlebot
6 “-”
5 “RssReader
5 “NetNewsWire Lite
5 “Java
5 “Aggie
4 “Snownews
4 “libwww-perl
4 “Bloglines
3 “UltraLiberalFeedParser
3 “Radio UserLand
3 “QuepasaCreep v0.9.14″
3 “Jakarta Commons-HttpClient
2 “Wget
Luke: you’re right about platform being an independent variable. Without NetNewsWire skewing the graph, the results are an even better power law (and you’re on top at the moment).
As for my stats, I made some server changes last week that had the unintended side effect of borking my stats for a while (that’s why I’m using two-week-old data here). It’s all better now, but now I want to work out a better presentation and make some other scripting changes, so they’re offline for a while.
— Mark
Sérgio (#25), the same happens with my very own site, I presume it is due to the fact that we’ve centered the page via CSS.
This isn’t a FD specific problem, as you can find out by narrowing your browser window.
Chris is partially correct. It is not FD-specific, and it is due to how we’re centering our content, but there are other ways to center using CSS that do not exhibit this behavior. I was looking into this very issue last night and I may have a solution later this week.
— Mark
I’m just curious as to how you can tell the Feed On Feeds users, since it’s a browser-based, server-side application…is it just that all we FOF users are too boring, and all name our folders /fof/ ? ;)
>>”I’ve been prevaricating over whether or not to try out FeedDemon”
>”That’s odd, Jonathan, you always struck me as very honest.”
Adam, I’m glad you perceive me as honest. But as someone who (usually) takes care to choose his words carefully, I’m embarrassed that I used “prevaricate” (speak or act in an evasive way) instead of “vacillate” (waver between different opinions or actions; be indecisive).
I should simply have said I was in two minds about downloading FeedDemon. Though not any longer — it’s just as spiffy as Mark suggested.
Stats from last month (November):
grep ‘\.rss ‘ boston.conman.org | escanlog -agent | sort | uniq -c | sort -rn
1118 NetNewsWire/1.0.6 (Mac OS X; http://ranchero.com/netnewswire/)
748 -
623 Bloglines/1.0 (http://www.bloglines.com; 3 subscribers)
562 Localfeeds: Geographic Syndication, http://www.localfeeds.com using UltraLiberalFeedParser/2.5.3 +http://diveintomark.org/projects/feed_parser/
237 centericq/4.9.8
194 nntp//rss v0.3 (Linux 2.4.18-386 i386; http://www.methodize.org/nntprss/)
155 MagpieRSS/0.51 (+http://magpierss.sf.net)
123 YahooRssBot/1.0
96 Bloglines/1.0 (http://www.bloglines.com; 2 subscribers)
93 SharpReader/0.9.3.2 (.NET CLR 1.1.4322.573; WinNT 5.1.2600.0)
86 centericq/4.9.9
73 Feedster Crawler/1.0; Feedster, LLC.
69 SharpReader/0.9.2.1 (.NET CLR 1.0.3705.288; WinNT 5.1.2600.0)
54 Syndic8/1.0 (http://www.syndic8.com/)
52 RPT-HTTPClient/0.3-3
44 YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5;)
34 SharpReader/0.9.3.1 (.NET CLR 1.1.4322.573; WinNT 5.1.2600.0)
30 Fresh Search :: Terrar (http://www.terrar.com/)
25 Java/1.4.1_05
24 Jakarta Commons-HttpClient/2.0rc1
23 Wget/1.8.2
22 nntp//rss v0.3 (Linux 2.4.18-686 i386; http://www.methodize.org/nntprss/)
13 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)
12 libwww-perl/5.65
8 Mozilla/5.0 Galeon/1.2.6 (X11; Linux i686; U;) Gecko/20020913 Debian/1.2.6-2
8 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
6 Wildgrape NewsDesk/1.1
5 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
5 Java/1.4.1_01
4 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)
3 Wget/1.9
2 larbin_2.6.2 larbin2.6.2@unspecified.mail
2 RSS Voyager (http://www.searchguild.com/)
2 Python-urllib/2.0a1
2 Mozilla/4.06 [en] (X11; U; Linux 2.0.27 i586)
2 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
2 MagpieRSS/0.5 (+http://magpierss.sf.net)
1 larbin_2.6.3 webmaster@netzhoehle.de
1 agricola/0.9 (mod:RSS; http://www.conoze.com/agricola)
1 Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030827
1 Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
1 Mozilla/5.0 (Macintosh; U; PPC;) Gecko DEVONtech
1 Mozilla/4.7 [en]C-CCK-MCD (WinNT; U)
1 Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
1 Mozilla/4.0 (compatible;)
1 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)
1 Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)
1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 95)
1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; DigExt)
1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
1 Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
1 Mozilla/3.04 (X11; I; Linux 2.0.35 i686)
1 Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
1 Microsoft URL Control – 6.00.8862
1 Microsoft Internet Explorer/4.40.426 (Windows 95)
1 Java1.3.1
1 FP based on UltraLiberalFeedParser/2.5.3 +http://machinespeak.net/
escanlog is a program that pulls out requested fields from the log file, in case anyone is interested.
You should have at least one Mozilla Firebird RSS Reader Pannel in there too ! :) thought i am not sure how it would appear in hte logs
I find that it is more than adequate as an RSS reader, and does not use java (== slow), and does not use IE as a rendering engine (need I say more)
Elaine: Even though FoF is a server-side aggregator, it still sends a User Agent when grabbing the RSS file.
“FeedOnFeeds/0.1 (+http ://minutillo.com/steve/feedonfeeds/)”
Thanks for the info, Steve. One always wonders vaguely about how one’s tools actually work, when one doesn’t have time to play with them…although your answer makes sense, if I’d actually thought about it for half a second.
Not to get all meta, but I’m noticing the relative civility — nay, disinterestedness — in this discussion vs. the last power-law-blog foofurah.
Perhaps because Mark sliced off all the contentiousness right at the front: “Let?s do that thing again, the one where we all bicker in predictable ways.” (That line cracks me up; can I use it at family get-togethers?)
Perhaps because we’re talking about software, not people, and there isn’t (yet!) a religious war in agreggators. (tho one wonders whether the wars over aggregation formats, operating systems, or programming languages will carry over.)
Instead, I see polite questions about methodology, chart-making, comparisons of data….
How lovely. ;)
— Elaine
Ah, nope. Power-laws exhibit straight lines on log-log charts; not log charts. Much more severe than expodential.
One of my favorite examples:
http://enthusiasm.cozy.org/archives/000012.html#000012
One of the posters above opined that they arise when users have choice. That’s not entirely accurate. They to arise when user choices mimic those of the existing users. This may or not lead to users making the optimal choice for themselves.
Power-law curves tend to arise either the absense of rich information to guide the choice (i.e. herd instinct); or, when there is a concrete reason to make the same choice as others (i.e. standardization of exchange).
That this curve isn’t a power-law, yet, suggests that people are picking their news reader based on preferences other than herding or closed standards.
The first news reader vendor that figures out a proprietary way to make blog authors customize their feeds for him (as IE managed to do), or creates a significant amount of trading between his users (as Microsoft Office did) and the market will consolidate. Since at that point there will be more than just the herd instinct (“everybody else is doing how bad a choice can it be”) driving up the slope of the power-law.
— Ben Hyde
Speaking of feeds; can someone recommend any _good_ software for publishing feeds peoples feeds on a website? I’ve looked around abit but havn’t really found something really nice yet… :(
— Anonymous
I am no longer accepting public comments on this post, but you can use this form to contact me privately. (Your message will not be published.)
§
© 2001–9 Mark Pilgrim