Anyone else think that Godwin’s Law should be amended to include Rails?
Just a thought.
SUMMARY: I spent two years trying to make Rails do something it wasn’t meant to do, then realized my old abandoned language (PHP, in my case) would do just fine if approached with my new Rails-gained wisdom.
INTRO / BACKGROUND:
Back in January 2005, I announced on the O’Reilly blog that I was going to completely scrap over 100,000 lines of messy PHP code in my existing CD Baby (cdbaby.com) website, and rewrite the entire thing in Rails, from scratch.
I hired one of the best Rails programmers in the world (Jeremy Kemper aka bitsweat), and we set off on this huge task with intensity. The first few months showed good progress, and Jeremy could not have been more amazing, twisting the deep inner guts of Rails to make it do things it was never intended to do.
But at every step, it seemed our needs clashed with Rails’ preferences. (Like trying to turn a train into a boat. It’s do-able with a lot of glue. But it’s damn hard. And certainly makes you ask why you’re really doing this.)
Two years (!) later, after various setbacks, we were less than halfway done.* (To be fair to Jeremy’s mad skillz: many setbacks were because of tech emergencies that pulled our attention to other internal projects that were not the rewrite itself.) The entire music distribution world had changed, and we were still working on the same goddamn rewrite. I said fuckit, and we abandoned the Rails rewrite. Jeremy took a job with 37 Signals, and that was that.
I didn’t abandon the rewrite IDEA, though. I just asked myself one important question:
“Is there anything Rails can do, that PHP CAN’T do?”
The answer is no.
I threw away 2 years of Rails code, and opened a new empty Subversion respository.
Then in a mere TWO MONTHS, by myself, not even telling anyone I was doing this, using nothing but vi, and no frameworks, I rewrote CD Baby from scratch in PHP. Done! Launched! And it works amazingly well.
It’s the most beautiful PHP I’ve ever written, all wonderfully MVC and DRY, and and I owe it all to Rails.
Inspired by Rails:
*- all logic is coming from the models, one per database table, like Martin Fowler’s Active Record pattern.
*- no requires or includes needed, thanks to __autoload.
*- real MVC separation: controllers have no HTML or business-logic, and only use REST-approved HTTP. (GET is only get. Any destructive actions require POST.)
*- all HTML coming from a cute and powerful templating system I whipped up in 80 lines, all multi-lingual and caching and everything
*- … and much more. In only 12,000 lines of code, including HTML templates. (Down from 90,000, before.)
Though I’m not saying other people should do what I’ve done, I thought I should share my reasons and lessons-learned, here:
SEVEN REASONS I SWITCHED BACK TO PHP AFTER 2 YEARS ON RAILS:
#1 - “IS THERE ANYTHING RAILS/RUBY CAN DO THAT PHP CAN’T DO? … (thinking)… NO.”
For 2 years, I thought Rails is genius, PHP is shit. Rails is powerful, PHP is crap.
I was nearly killing my company in the name of blindly insisting Rails was the answer to all questions, timeframes be damned.
But when I took a real emotionless non-prejudiced look at it, I realized the language didn’t matter that much.
Ruby is prettier. Rails has nice shortcuts. But no big shortcuts I can’t code-up myself in a day if needed.
Looked at from a real practical point of view, I could do anything in PHP, and there were many business reasons to do so.
#2 - OUR ENTIRE COMPANY’S STUFF WAS IN PHP: DON’T UNDERESTIMATE INTEGRATION
By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members’ Login Area, and dozens of cron shell scripts were ALL going to have to change. 85 employees re-trained. All customers and clients calling up furious that One Big Day, with questions about the new system.
Instead, I was able to slowly gut the ugly PHP and replace it with beautiful PHP. Launch in stages. No big re-training.
#3 - DON’T WANT WHAT I DON’T NEED
I admire the hell out of the Rails core gang that actually understand every line inside Rails itself. But I don’t. And I’m sure I will never use 90% of it.
With my little self-made system, every line is only what’s absolutely necessary. That makes me extremely happy and comfortable.
#4 - IT’S SMALL AND FAST
One little 2U LAMP server is serving up a ton of cdbaby.com traffic damn fast with hardly any load.
#5 - IT’S BUILT TO MY TASTES
I don’t need to adapt my ways to Rails. I tell PHP exactly what I want to do, the way I want to do it, and it doesn’t complain.
I was having to hack-up Rails with all kinds of plugins and mods to get it to be the multi-lingual integration to our existing 95-table database.
My new code was made just for me. The most efficient possible code to work with our exact needs.
#6 - I LOVE SQL
Speaking of tastes: tiny but important thing : I love SQL. I dream in queries. I think in tables.
I was always fighting against Rails and its migrations hiding my beloved SQL from me.
#7 - PROGRAMMING LANGUAGES ARE LIKE GIRLFRIENDS: THE NEW ONE IS BETTER BECAUSE *YOU* ARE BETTER
Rails was an amazing teacher. I loved it’s “do exactly as I say” paint-by-numbers framework that taught me some great guidelines.
I love Ruby for making me really understand OOP. God, Ruby is so beautiful. I love you, Ruby.
But the main reason that any programmer learning any new language thinks the new language is SO much better than the old one is because he’s a better programmer now! You look back at your old ugly PHP code, compared to your new beautiful Ruby code, and think, “God that PHP is ugly!” But don’t forget you wrote that PHP years ago and are unfairly discriminating against it now.
It’s not the language (entirely). It’s you, dude. You’re better now. Give yourself some credit.
Ok. All that being said, I’m looking forward to using Rails some day when I start a brand new project from scratch, with Rails in mind from the beginning.
But I hope that this reaches someone somewhere thinking, “God our old code is ugly. If we only threw it all away and did it all over in Rails, it’d be so much easier!”
It’s often been said that Perl’s greatest strength is CPAN, Perl’s vast collection of free libraries contributed by developers from around the world. Recently I started to wonder about RubyForge and how RubyForge stacks up against CPAN in general.1
First, length of service. CPAN has been around for 12 years (October 1995). RubyForge has been in existence for just over 4 years (July, 2003).
Second, the number of users. RubyForge boasts over 20,300 users. CPAN, on the other hand, has far less at just over 6,150 registered users.2 Not every registered user is associated with a project, however. There are 3635 users are associated with a project on RubyForge, while on CPAN there are 3774 users associated with at least one library. Of the 3635 users on RubyForge, 849 are associated with more than one project.3
Third, the number of libraries. CPAN boasts approximately 13,500 separate libraries. RubyForge currently has approximately 5000 separate libraries, organized into about 4800 projects. That means, on average, most projects have one library, but some have multiple libraries per project.4 While RubyForge has far fewer libaries than CPAN, the ratio isn’t nearly as large as I would have thought.
Quick aside. I didn’t do any real analysis against Python, but the home page for the Vaults of Parnassus shows 2025 libraries.
Fourth, library quality and usefulness (more subjective here). There’s a lot of overlap and, well, cruft on CPAN.5 There are over 300 Acme (joke) modules. There are multiple wrappers for the same underlying library, such as many of the “Tiny” and “Simple” modules. There are libraries that should have been bundled together but weren’t, such as the various Chemistry::PointGroup libs. There are also libraries that have similar or identical functionality to other libraries, such as many of the List and/or Array libraries.
On top of that, a healthy chunk of the Perl libraries on CPAN are either unnecessary in Ruby or contain behavior that’s already baked into Ruby itself. Examples include a large collection of OO modules (Class::Accessors and the like), a large number of modules that create various IO, Number, File, Array, String and Hash classes, and related methods, that Ruby has builtin (Array, Array::List, File::chmod, and so on), over 200 “Tie” modules (Ruby doesn’t need ‘tie’), over 90 libraries for interacting with CPAN or the RT backend itself (I only know of one library for RubyForge), and equivalent libraries that are already part of Ruby’s standard library (e.g. libwww).
Last, release frequency. Between August 14th and September 14th there were 1133 releases from 690 distinct libraries on CPAN 6. RubyForge, by contrast, had 612 releases in the same date range (although I wasn’t sure how many of them were from distinct libraries at the time of this writing). So, slightly less than half of CPAN at the moment.
What do all these numbers mean? Good question. I think, at the very least, it means that the Ruby community is doing very well in terms of library development and releases. I give Tom Copeland a huge amount of credit for that, in that I think the very act of creating RubyForge fostered an atmosphere of development (collaborative or otherwise) and inspired programmers new to Ruby to take the step of releasing their own software. I can’t prove it, of course. It’s just a gut feeling I have after watching the Ruby community grow for seven years.
It also means that Perl is still going strong, cruft and all. You can’t really argue too much with their release rate, and some of it is really good stuff, too. I would say that CPAN still has the edge in database interfaces, Apache libraries and wrappers for 3rd party commercial libraries, among a few other things.[7]
But, we’re catching up, and fast. :)
See you next Wednesday.[8]
1 When I say “CPAN”, I’m generally referring to search.cpan.org plus RT.
2 I scraped the Authors pages to get the total. At the time of this writing it was 6152, although a handful of these appear to be generic logins.
3 CPAN isn’t a collaborative development environment, so there may be multiple users actively associated with a given library, but there’s no way to tell without manually digging through files.
4This number does not include a number of libraries listed on the RAA that are not on RubyForge. I’d make a very rough guess of about 200. Hard numbers welcome.
5 I should know. I own some of the cruft.
6 There were anywhere from 1 to 10 releases per library in that period. Some had multiple releases in the same day, a curious trend on CPAN.
7 Port a Perl module today! I should mention that I didn’t do any very long term trending, so I guess it’s possible that the releases per month have dropped, but I somehow doubt it. Feel free to prove me wrong (or right).
8Many thanks go to Tom Copeland for providing me with the RubyForge statistics.
Listening to David's RailsConf Europe 2007 keynote, I've learned that he's going to a release a beta version soonish. He's asked me to do some benchmarking, comparing the performance of edge Rails to the latest stable branches. Here's a chart:
stable11 is the current svn version of branch 1-1-stable, stable12 is current version of 1-2-stable branch and edge is the current version of Rails trunk. Numbers are requests per second (measured on my MacBook during David's talk).
If you're a numbers type, these are the numbers:
page c1 totalc2 total c1 r/sc2 r/s c1 ms/rc2 ms/r c1/c2 empty 0.947091.05049 1055.9951.9 0.951.05 0.90 welcome 1.228781.34536 813.8743.3 1.231.35 0.91 recipes 1.372141.50364 728.8665.1 1.371.50 0.91 my_recipes 1.369581.49912 730.2667.1 1.371.50 0.91 show 3.469444.27918 288.2233.7 3.474.28 0.81 cat 3.726754.58788 268.3218.0 3.734.59 0.81 cat_page5 3.856244.68700 259.3213.4 3.864.69 0.82 letter 3.699704.54140 270.3220.2 3.704.54 0.81 all requests 19.6697123.49406 406.7340.5 2.462.94 0.84 GC statistics c1 totalc2 total c1 #gcc2 #gc c1 gc%c2 #gc% c1/c2 2.276573.09640 21.025.0 11.5713.18 0.74c1: 1.2-stable, c2: edge, r/s: requests per second, ms/r: milliseconds per request
I haven't had time to analyze the code, but it seems that anything loading a large number of active record objects takes a rather large performance hit.
I hope we can improve performance before 2.0 gets finally released. Unfortunately, I have almost no time to work on this. However, if anyone has a performance patch, I can look into the it.
A key property of agile development is figuring out how to make a system go live with a small subset of features. We build software for the business value it offers, the quicker we go live, the faster we get at least some of that business value.
My colleague Dave Leigh-Fellows told me one of my favorite examples of this kind of thinking. It came when we has working for a brokerage firm. They had a new kind of product that they wanted to get into the market. The full software support for this was a web page that the customer filled in that generated the necessary transactions against the back-end system. But Dave came up with a way to get the product into the market faster than that.
The first two versions may not have been the most elegant solutions ever conceived, but they did get the product into the market much more quickly. I've not come across any other examples of iterative development that use roller-skates, but that may be more due to lack of imagination rather than lack of need.
This year is a good year for Ruby on the east coast. We kicked things off with a bang with the Gotham Ruby Conference in April, saw the Ruby Hoedown come and go in Raleigh bringing along with it some great videos of their talks, and of course, will see the Seventh International RubyConf in Charlotte in just a couple months.
However, these aren’t the only games in town. We’ve also got Ruby East 2007, which is a one day, multi-track, Rails friendly regional conference at the Penn State Great Valley Campus. They’ve managed to get an impressive list of speakers together, and even let two of those vagabonds from the Ruby Reports project sneak in to do their ranting.
If you’re local to the area and don’t think you’ll make RubyConf this year, this is a good chance to still get your fix. If you’ve been tinkering with Ruport and want to shake down me or Mike, this is also a chance for that.
From what I’ve seen so far, these regional Ruby conferences are always a lot of fun, especially because each one takes on its own unique flavor. If you want to go to this one, you should probably register soon because the event takes place on Friday, September 28, which is right around the corner.
Hope to see you there!
Like most Brits my age I grew up with a sci-fi Children's program on BBC called Doctor Who. (For those who know, my doctor was Jon Pertwee, although I also saw a good bit of early Tom Baker.) It was actually the longest-running sci-fi TV series in the world, running from 1963-89.
A few years ago it was revived in the UK and has become a big hit - and not just for children. Doctor Who always had themes and scripts that went beyond the children's' audience and the series developed a huge fan base that lived off books and audio series even when the TV series died. The revival follows this with shows that are written to appeal as much to adults as kids. It was really great to sit with a couple my age and their 8 and 10 year old daughters and enjoy the new series. The scripts and acting are good, the only change is that the special effects are also good now (the old special effects made classic star trek look high-tech).
At home we don't watch much telly, the last shows we watched regularly were Buffy and Angel. Cindy, being American, had never seen Doctor Who growing up, but she loves the new series. When we get a new set of DVDs there's usually several nights of "it's late, we're tired, but maybe we can do one more".
If you've never seen Doctor Who the place to start is the opening episode of the revived series: Rose. (Wikipedia has mind-bogglingly comprehensive coverage, but I won't link from here as it's naturally full of spoilers.) Rose not only introduces the set up you'll need for other episodes (who the Doctor is, what the TARDIS is) but also does a really good job of distilling the tone of Doctor Who, capturing the mix of adventure and comedy.
If you like Rose then you can either carry on with the full first season or cherry pick highlights. If you prefer the latter I'll suggest my favorites. My big favorite from the first series was the two part The Empty Child / The Doctor Dances. I rate this as better than most films I've seen, certainly better than most TV. (It won a Hugo award so it's not just me.) It was written by Steven Moffat who is also known for writing the comedy series Coupling. Almost as good is Dalek. It lacks the humor but scores due to a wonderfully intense performance from Christopher Eccleston. I also really like the final two part (Bad Wolf / The Parting of the Ways) but you really need to see the whole series to appreciate it properly. (A tip if you do watch the whole series: don't watch the trailer for Bad Wolf (it appears at the end of Boom Town) as it gives away an important part of the plot of Bad Wolf.)
The second series has a different actor playing the Doctor (they have a nice technobabble rationalization to allow them to change actors easily). It doesn't quite hit the high spot of The Empty Child but is still really good. My suggestions for cherry pickers here would by The Girl in the Fireplace (another Moffat Hugo win) and The Impossible Planet / The Satan Pit.
When I said the second series didn't quite hit the heights of The Empty Child, I'm not being very reasonable because those two episodes are far too good for any TV series to live up to. However the third series (not yet all broadcast in the US - it's good to have friends in the UK) hits those high notes twice. Human Nature / The Family of Blood is a super two-parter that threatens to take away Steven Moffat's crown of writing the best episode. Moffat's response is Blink, which is as good a 45 minutes of TV as you could ever hope for. Not just has it got a great story and some cracking humor, it also achieves Doctor Who's higher purpose. You see Doctor Who is only secondarily about entertainment, it's primary purpose was always to scare the living daylights out of small children. I may be too old now to get behind the sofa, but I do remember how much I enjoyed it.
I was in Boston, about to fly out to our office in Calgary. I look at my calendar to see if I have a meeting. First one is at 10.30am - cool no need to rush out of bed in the morning.
I turn up and am two hours late. What happened was that I was invited to the meeting that begun at 8.30am Calgary time. Lotus Notes saw my computer was set to Boston time, and helpfully converted the time zones for the two hour shift.
You could argue that this was my fault for not paying attention. After all I know this is how Notes works and was being careless when I read my diary. I don't buy this, Donald Norman noted a while ago that we tend to blame ourselves for errors that are due to bad usability - like a door with a handle that you should push.
Time zones are particularly vulnerable to this kind of problem It's most notably a problem in calendaring applications, but you see this issue in enterprise software as well. There is a temptation to try to be clever with handling time zones, but this temptation leads to trouble if the software isn't quite clever enough, which is what happened in this case.
I'd prefer Notes ignore time zones completely. You set the time for the place the meeting occurs in and that's what it should show you. Who cares about the time zones? It's only the time on the ground that usually matters. When I look at my calendar for a day in Calgary, I want to see the Calgary times for that day - wherever I happen to be when I'm looking at it.
The exception, of course, is a phone meeting that spans time zones. But here you should do something exceptional for a phone meeting rather than complicate handling a physical one. It could be something that allows you to set a flag for a phone meeting, and then you get a different time display. Or it could be as simple as just allowing you to put the timezone on the time display, and leaving the conversion to the reader. With phone meetings you think more about time-zones than you do for physical meetings (or at least I do).
The important lesson here is to make the most common case (physical meetings) simple and only do complicated things for less common cases as exceptions (quite possibly manual exceptions). Calendaring time zones get into trouble because they make the common and simple case be more complex. The problem occurs because the designers wanted to use the same data for both the simple and phone cases - but that just gets the simple case in trouble.
Usually when I hand out a prize for worst-user experience Lotus Notes is at the top of the queue. (Indeed I find it embarrassing to admit that ThoughtWorks uses the damn thing.) But the worst time-zone experience award goes to Microsoft Office, although to be fair this was many years ago. I had recently bought a PDA (running Windows CE version 2). I had put some all day meetings into my calendar, flown to Chicago, and to get the PDA to alert me to meetings I changed the timezone of the PDA to one hour earlier.
Suddenly every one of my all day meetings shifted to a day earlier. This was due to a catalog of errors. First off they had stored an all day meeting as a meeting from midnight to midnight - that's the kind of representation error on a TimePoint that often gets people into trouble. Then it was compounded by shifting the times of meeting when I changed time zones - so all day meetings were now 11pm to 11pm. This, of course, is due to putting the time zone into the meeting so it looks like it shifted when I changed time zone on the PDA. Then to cap it off, the software was clever enough to know that an all day meeting should only show the day of the meeting, but the day it chose was the day of the start of the meeting - that was now one day earlier. That's the poor timepoint representation biting back.
I was in the Calgary office last week and had a good chat with John Kordyback, one of our most trusted technical leads. He's worked on, and dug into, a number of travel loyalty software systems (frequent flyer/sleeper etc) and we talked about the nature of these kinds of things and how to think about them in a more fruitful manner.
The core of of a loyalty system is a system to keep track of points (or miles). This should allow customers to see their points and also for the company to manage the unredeemed points. Although it seems that most people don't see it this way, this is essentially an accounting system, just switching points for dollars. John's observation was that repeatedly he runs into what people see as difficult problems that are much easier to deal with once you put accounting spectacles on.
An example of this is dealing with ad-hoc changes. However good your automated rules processing is, there always cases when something odd happens and you have to intervene manually. The result for many systems is and ad hoc change to totals that is error prone and unaudited. With an accounting frame of mind, however, you look at these changes as accounting adjustments and the patterns for this are well understood.
A notable difference between a loyalty program and most accounting systems is that a loyalty program is more about managing liabilities rather than managing assets. Hence there's more focus on things like risk management, as well as common themes like taxes and revenue reporting.
Many loyalty systems have multiple kinds of points, such as such as regular miles and elite qualifying miles. This is a common point of complexity. If you use an accounting viewpoint, however, you can track these easily as multiple currencies.
An interesting twist on this is potential points. If I book a flight for next month, the airline needs to know that there are miles I will earn when I fly next month (potential miles). These potential miles affect their liabilities. However it's only when I fly that they turn into real miles. Again accounting thinking can help here, we can use multiple currencies again, or use an accounts payable notion. The mechanisms are there and well understood, we just have to apply the model to the situation.
We fleshed this out in practice where we also found it really helpful to use TestDrivenDevelopment. A group of people spent a couple of weeks trying to sort out potential miles with planned design, but the core issue was cracked in a couple of days with TDD. The crucial part of this was focusing on examples to make the problem concrete.
The accounting analogy also applies, although partly less directly, to deciding how to award miles for activity. Any program has activity rules that need to be very flexible and need to cope with constant changes to the loyalty program. We can look at this as following the model of domain events triggering accounting entries through using Agreement Dispatchers. This is a pattern John and I have used lots of times and works well to these kinds of changing rules. Essentially we have agreements that represent the overall program rules for a class of participant. Each agreement consists of a set of posting rules keyed by the type of event and a date range. When an domain event occurs (a hotel stay) we look up the agreement dispatcher for the customer, and use the event to look up the right posting rule. We then run the posting rule to create the appropriate accounting entries to represent the miles for the event. The time dating of the events allow us to change posting rules over time but still be able to handle old events and correctly do automated processing of adjustments. (Some day I'll finish writing up these patterns, but what I have on the web is hopefully enough to give you some ideas.)
The second aspect of a loyalty system is tracking the customer experience. Since the accounting requires the system to record the customer's activity, the loyalty system acts as a natural base to learn from the customer's interactions with the company. Much of this is data mining - looking for patterns in customer behavior which can lead to new products and promotions. You can also use this activity history to assess the success of promotions - if you offer a mileage bonus for flying a route what is the response like?
Like me, John is a strong proponent of using ReportingDatabases, and this is a good fit for this kind of problem. The accounting side needs a very different set of data structures and uses regular updates as activities occur. The customer experience analysis is all read only, so you can use less normalized structures with regular, but not necessarily real time, feeds from the accounting side.
Taking it further, it seems reasonable to completely decouple the accounting and customer experience systems. They are both usually lodged together as a single customer loyalty system because they track the same events. Yet since they differ so much on the inside it may make more sense to treat them as two separate systems that feed off the same event stream (the accounting side would probably generate some events for the customer experience side too).
One of the habits of customer experience tracking is frequent changes to the system to support new kinds of analysis. We speculated that we could try an approach that had a single stored event log of customer activity, and plug in relatively independent 'miners' that would transform selected information from the log into more particular data structures to do different kinds of analysis. The miners could be relatively independent of each other and thus easier to build.
As you can see, our discussion did shift from looking at John's experiences to some of our joint speculations about how a system like this could be built in the future. What's clear to us is that there is a lot of room for exploring new ideas in this space that could introduce a new set of abstractions that would lead to systems that can provide better support to this business activity. More and more attention is being paid to this these days, so this seems like a fruitful territory for us to work in.
Is changing the interface of part of the code a refactoring?
The answer to this question is pretty simple - changing an interface is a refactoring providing you change all the callers too. A great example of this is Rename Method, which is an interface changing refactoring present on pretty much all refactoring tools.
The changing of all the callers is an essential part of this refactoring. Just changing an interface declaration will break the system - and thus isn't a behavior preserving change.
Interface changing refactorings do assume that you can get hold of all the callers, which is why you have to be much more careful with PublishedInterfaces. With a published interface, the interface itself is part of the observable behavior of the system.
Dynamic languages can make these changes much more awkward. Static typing really does help here in pinning down exactly which interface is being called at various points. Reflective calls that can also make it harder to find, either by embedding method names in strings or even composing them at run-time. This is another area where tests are essential even in environments that have refactoring tools.
These days, it seems I hardly have the time for doing fun random hacks. So here I’ve started one, and if anyone finds it interesting, please take it from here and let me know how it turns out.
Loosely based off of AIML, kind of, but not really:
class Conversation def initialize(person) @person = person @response_id = 0 end attr_reader :response_id def say(msg) print "#{@person}: " @response_id = Response[@response_id].respond_to(msg) end class Response def self.responses @responses ||= {} end def self.[](id) responses[id] end def initialize(id) @id = id self.class.responses[id] = self @matchers = [] @messages = [] end attr_reader :id,:matchers def when(pattern,id) @matchers