Happily retiring TabTerm

2007 October 26

With the release of Mac OS X Leopard, we are happy to see tabs have been integrated into the terminal app -- as promised. In light of this, we are happy to announce the official retirement of TabTerm.

TabTerm has been the tabbed terminal choice for students, professionals and hackers from Santa Barbara to Shanghai.

Quotes:

"TabTerm, we had a good run. You pulled us through a hard time while waiting for Leopard. It is hard to see you go."

-- Ryan Garver, CTO

Suggestion Box

Now that we're no longer busy keeping TabTerm up and runinng, we're officially opening the doors to your suggestions for other mac apps, especially programming related.

Send your suggestions to labs@elctech.com or leave us a comment here.

Happy Tabbing!

Sliding Session Timeout plugin

By default, sessions in Rails expire at a fixed time from the moment they are created. The Sliding Session Timeout plugin lets you configure your sessions to expire in a sliding window, a fixed time from the last page view.

To use it, place a call in your controller (typically application.rb) with the number of seconds in which to time out the session:

class ApplicationController < ActionController::Base
  sliding_session_timeout 3600
end

You can use Rails' built-in number helpers to make it more readable:

class ApplicationController < ActionController::Base
  sliding_session_timeout 60.minutes
end

You can also pass an optional method to be called upon session timeout:

class ApplicationController < ActionController::Base
  sliding_session_timeout 60.minutes, :on_expiration

  def on_expiration
    # do stuff...
  end
end

Feedback is welcome!

Updates:

Mandaryn asks: "Do you know how to force a redirect to a normal action from ajax call?"

By default, sliding_session_timeout will just call reset_session when your session times out. If you have a filter e.g. login_required to enforce that users are logged in, I've found this is one place you can handle the different types of requests rather than in an :on_expiration callback. The problem with adding render or redirect_to in an :on_expiration callback is that if any of your other filters (such as login_required) perform a render or redirect, you'll get a DoubleRenderError. So until I discover a nice way around that, I handle request formats something like so:

  before_filter :login_required
  sliding_session_timeout 10
  
  def login_required
    respond_to do |format|
      format.html { (redirect_to(login_url) and return false) }
      format.js do
        render :update do |p|
          p << "location.href = '#{login_url}';"
        end and return false
      end
    end unless session[:logged_in]
  end 

Readability Tips

From: fr.ivolo.us

Formatting Dates & Times

Coming from Java and other frameworks, the hardest part about Ruby on Rails can be finding the little things that make coding easier.

Here are some tips relating to dates and times:

>> @user.logged_in_at.strftime("Last login: %m/%d/%y at %I:%M %p")
#   => "Last login: 03/05/07 at 06:30 PM"

This can become:

>> @article.logged_in_at.to_s(:last_login)
#   => "Last login: 03/05/07 at 06:30 PM"

can be accomplished with:

ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(
  :last_login => "Last login: %m/%d/%y at %I:%M %p"
)

A little further:

>> @article.last_login
#   => "Last login: 03/05/07 at 06:30 PM"

with:

class Article < ActiveRecord::Base
  def last_login
    logged_in_at.to_s(:last_login)
  end
end

More with Date and Time

>> 1.year
=> 31557600
>> 1.year.ago
=> Fri Oct 27 18:10:01 -0700 2006
>> 2.years.from_now
=> Tue Oct 27 12:10:15 -0700 2009

And to make working with date ranges easier:

>> t = Time.now
=> Sun Oct 28 00:10:32 -0700 2007
>> (t.year-5..t.year+5).to_a
=> [2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017]

Or the same thing for your views:

select_year(Time.now)

Displays:

Leopard ImageMagick

You've likely found this post if you are rebuilding a Rails environment on Leopard and tried to install ImageMagick via MacPorts. Using the install command:

sudo port install imagemagick

You will see a lot of nice compiling and then failure with the following error:

tiffgt.c:303: warning: unused parameter 'x'
tiffgt.c:303: warning: unused parameter 'y'
tiffgt.c: In function 'raster_special':
tiffgt.c:348: warning: unused parameter 'x'
tiffgt.c:348: warning: unused parameter 'y'
/bin/sh ../libtool --tag=CC --mode=link gcc  -O2 -Wall -W  -L/opt/local/lib -L/opt/local/lib -L/opt/local/lib -o tiffgt  tiffgt-tiffgt.o ../libtiff/libtiff.la ../port/libport.la -L/usr/X11/lib -framework GLUT -lobjc -framework OpenGL -ljpeg -lz -lc 
libtool: link: gcc -O2 -Wall -W -o .libs/tiffgt tiffgt-tiffgt.o  -L/opt/local/lib ../libtiff/.libs/libtiff.dylib ../port/.libs/libport.a -L/usr/X11/lib -framework GLUT -lobjc -framework OpenGL /opt/local/lib/libjpeg.dylib -lz -lc
ld: cycle in dylib re-exports with /usr/X11/lib/libGL.dylib
collect2: ld returned 1 exit status
make[1]: *** [tiffgt] Error 1
make: *** [all-recursive] Error 1

Error: The following dependencies failed to build: tiff
Error: Status 1 encountered during processing.
macbook:~ user$ 

There is already a trac ticket on MacPorts for this issue with a solution, but the quick fix is to edit the tiff package port file:

sudo vi /opt/local/var/macports/sources/rsync.macports.org/release/ports/graphics/tiff/Portfile

Find the section that reads:

platform macosx {
  configure.args-append --with-apple-opengl-framework
}

And modify it to read:

platform macosx {
  configure.args-append --with-apple-opengl-framework
  post-configure { reinplace "s,-L/usr/X11/lib,," ${worksrcpath}/tools/Makefile }
}

Finally, clean any pending build of tiff and reinstall:

sudo port clean tiff
sudo port install tiff

Enjoy!

Transcoding with On2 and Ruby

The FlixEngine by On2 has a great API. The only problem was that there were no Ruby bindings for it, that is, until a few months ago. Matt Bauer of MMMultiworks started a great Ruby implementation off of it's C API. I recommend reading Matt's post about it, as he explains about the engine and its implementation in some good detail:
http://blog.mmmultiworks.com/2007/5/18/on2-flix-engine

On a recent project, I had the opportunity to use his bindings, and ended up implementing some other chunks of the FlixEngine API. The patches is quite big, and I hope to be adding it to the RubyForge project soon. Here is a small example that illustrates how easy it is to interface with FlixEngine though the Ruby API:

   1  <pre>
   2    # load the file
   3    flix_engine = On2::FlixEngine.new(localhost, 0)
   4    flix_engine.source_path = "my_video.mpg"
   5    flix_engine.output_path = "my_video.flv"
   6  
   7    # set video scaling parameters
   8    flix_engine.scale({
   9      :height => 250,
  10      :width  => 250
  11    })      
  12  
  13    # set exported PNG scaling parameters
  14    flix_engine.png({
  15      :height  => 250,
  16      :width  => 250,
  17      :export_time  => "10000",
  18      :prefix  => "my_video",
  19      :suffix  => "video_thumb" 
  20    })      
  21    
  22    flix_engine.state  # queued
  23  
  24    # start encoding
  25    duration = Benchmark.realtime {
  26      flix_engine.encode
  27      while (flix_engine.encoding?)
  28        flix_engine.state  # running
  29        sleep 1 
  30      end     
  31    }
  32  
  33    flix_engine.success?  # boolean status
  34    flix_engine.duration  # the duration of the outputted video
  35    flix_engine.flix_error  # flix engine errors
  36    flix_engine.system_error  # system specific errors
  37  </pre>

There are quite a few other API calls that you can utilize as well. If you use On2's FlixEngine, and have any other additions/questions, please let Matt or I know !

Note that a bunch of the calls in the example are not in trunk at the moment, and if you need them now, let me know.

Happy transcoding !

Safely exposing your app to a ruby Sandbox

Creating wrapper classes for the sandbox

When creating my sandboxed game of Tictactoe (where a user can upload a new algorithm and play tictactoe against it), I wanted to expose only a small part of my application to user uploaded code. In the follow code, for example, I would want to provide user access to only a few methods of the

   1  Board

   1  class Board < ActiveRecord::Base
   2    has_many :moves
   3    belongs_to :algorithm_x, :class_name => "Algorithm", :foreign_key => "algorithm_x_id"
   4    belongs_to :algorithm_o, :class_name => "Algorithm", :foreign_key => "algorithm_o_id"
   5  
   6    def make_move!(x, y)...
   7    def move_matrix...
   8    def log_info(msg)...
   9    def winner...
  10    def game_over...
  11    def make_computer_move!...
  12    def human_turn?...
  13  end

If I want to allow the user's code to access

   1  make_move, moves, move_matrix, log_info

   1  class BoardWrapper
   2    def initialize(board); @board = board; end
   3    def make_move(x,y); @board.make_move(x,y); end
   4    def moves; @board.moves.collect {|m| MoveWrapper.new(m) }; end
   5    def move_matrix; @board.move_matrix; end
   6    def log_info(msg); @board.log_info(msg); end
   7  end

acts_as_wrapped_class

This is pretty cumbersome to build, so I built

   1  acts_as_wrapped_class

• Automatically generate a wrapper class for each class marked as

     1  acts_as_wrapped_class
  

• Dispatch methods that match (or don't match) a safelist or blacklist
• Finds appropriate wrappers for return results (meaning if

     1  Board
  

  returns a

     1  Move
  

  then

     1  BoardWrapper
  

  returns a

     1  MoveWrapper
  

  )
• Wrap the contents of arrays and hashes (same as above, but will work with arrays of

     1  Move
  

  , and Hashes containing

     1  Move
  

  )
• Dispatch

     1  ===, hash, <=>
  

  methods directly to the wrapped objects. Compare two wrappers objects and get the same results as the two wrapped objects.

The above example is much shorter when written with acts_as_wrapped_class:

   1  class Board < ActiveRecord::Base
   2    acts_as_wrapped_class :methods => [:moves, :make_move!, :move_matrix, :log_info]
   3  
   4    def make_move!(x, y)...
   5    def move_matrix...
   6    ...
   7  end
   8  
   9  class Move < ActiveRecord::Base
  10    belongs_to :board
  11      
  12    acts_as_wrapped_class :methods => [:x_pos, :y_pos, :is_x, :created_at]
  13  end

Simple executing acts_as_wrapped_class inside the definition of

   1  Board

Try to access

   1  winner

   1  wrapper._wrapped_class

   1  Board

View the RDOC for acts_as_wrapped_class for more detail.

acts_as_runnable_code

In order to make sandboxing user code even easier, I created another gem: acts_as_runnable_code. This gem helps you with the creation of the sandbox, the referencing of the wrapper classes, and automatic wrapping/unwrapping of data as it flows in and out of the sandbox. It assumes the following about your application

• you have objects that store user uploaded code in them
• you want to use your classes in the sandbox with reduced functionality provided by acts_as_wrapped_class
• you want to evaluate an instance of user uploaded code within the context of some instance of a wrapped class

When writing tictactoe, I created an Algorithm model which stored user uploaded code in a database TEXT field. I also wanted to evaluate that code using the binding of the

   1  Board

   1  class Algorithm < ActiveRecord::Base
   2    acts_as_runnable_code
   3  end
   4  
   5  @board = Board.find(id)
   6  @board.algorithm_x.run_code(@board, :timeout => 1.0)

View the RDOC for acts_as_runnable_code gem.

To see tictactoe in action, create your own algorithm, and test the safety of the sandbox (scary!) visit tictactoe.mapleton.net

Sandboxing in ruby

A few weeks ago, I decided to make a rails-based game. I wanted to bring the strength of ruby's metaprogramming into the game world, so I investigated sandboxing user uploaded code blocks. The only ruby sandbox was written by Why the Lucky Stiff, and you can find complete details on it here:

The Freaky Freaky Sandbox

The sandbox is an amazing hack on ruby's lookup tables to essentially allow a completely separate execution context with its completely own set of classes. The interesting part is how it interfaces with the outside world (the "Jungle"):

• Classes can be copied in from the Jungle using

     1  Sandbox.import
  

  , and exist in both places with separate definitions. The sandbox automatically does this with simple essentially classes like String, Object, Hash, Array, etc.
• Classes can be proxied in from the Jungle using

     1  Sandbox.ref
  

  . In this case, a proxy class is defined in the sandbox with exactly the same name as the outside class, but with only a two methods: const_missing & method_missing. When a method is called on the proxy, the sandbox is disabled and the actual method executes outside the sandbox. The result of the method is Marshalled into the sandbox, and it is enabled again.
• Objects can be copied into the sandbox using

     1  Sandbox.set
  

  as long as they're defined there. This is accomplished by marshalling
• Objects can be returned from the sandbox at the end of a

     1  Sandbox.eval
  

  call. This is accomplished by marshalling

Now you're ready to start writing your own applications using the sandbox. It's a pain to install in ruby 1.8.6, because it requires a small patch, but Why says that it works without patch in ruby 1.9.

Its about the team

A few weeks ago Derek Sivers posted an opinion piece on O'Reilly's Ruby Blog. The article basically raps on Rails for not being flexible enough to meet Derek's needs in his application. As the comment thread grew it quickly became clear that the root of the problem was Derek's lack of knowledge of the Ruby on Rails paradigm. At the end of the day, I don't think it is appropriate for me to lecture on the "right" and "wrong" tools for a job. I have not tried every language or framework; and if Derek is hacking the thing out by himself, then by all means use what you're comfortable with. What I can talk to is one issue that didn't get as much attention as I think it should have in the flame war that ensued. Maintainability.

When building applications where more than one person is involved there is an important question to ask when choosing a tool. Can your team maintain the work product? And, are they happy while they do it? Both of these questions are important to long term project success.

The first question addresses the ease of maintenance. This is the subject of numerous punch-lines and has become the hallmark (not in a good way) of a number of languages. Project maintainability is of particular importance when agile methods are being applied. If a strong up-front design phase is not present then you must be able to make changes efficiently, effectively, and frequently. The agile community would call this refactoring. Clearly this mentality necessitates high maintainability in the language, frameworks, and tools that are being used. Most languages and frameworks historically considered "enterprise grade" (think Java or .Net) leverage refactoring support in the IDE. This is necessary because those languages tend to have significant syntactic bloat. Take a look at "Hello, World!" in Java to see what I mean.

Ruby on Rails has very high information density which allows for concise and clear communication of functionality. The language syntax is very flexible. If you look at ActiveRecord or ActionController you will see constructs that implement complex features in a matter of a few lines. If solving a problem prevents you from writing code that is easily understood by other programmers then the language and libraries are not doing their job. Ruby has the flexibility to support this and Rails provides the libraries to facilitate clear communication of ideas between programmers through code.

The second question speaks to team morale. There is a reason that Ruby on Rails is particularly popular in the startup "Web 2.0" space. If you were a programmer working with your friends in a garage you would choose a language that was fun. But this question goes deeper than "is it fun to program in the language/framework?" Looking at it from the direction of maintainability, can we have an enjoyable experience while maintaining and improving an existing code base? This is impacted by a number of factors beyond language syntax, such as how difficult changes are to make, or how fragile the code is. Ruby is a fun language to program in. Rails is well structured and handles the boring details that never change for you. Projects in Ruby on Rails generally age gracefully. You will rarely see the addition of a new feature in Rails require a massive cascade of changes through out the code base.

Ruby on Rails has succeeded because it can answer both of these question with a resounding yes. Doubly so when the community values are embraced. Convention over configuration; Don't repeat your self; Write tests. I have never felt that PHP facilitates clear communication between programmers. Nor have I ever felt that maintaining PHP was anything close to an enjoyable experience.

RSpec Tutorial

Being a very popular Behaviour Driven Development framework, RSpec is not very well documented. Yes - you can find the answers to all your questions, but not in one place, and that's only for technical questions. Questions like "When should I use mocks?" and "What's the difference between a should_receive and a stub?" are much harder to find answers for.

Using that as a springboard, I've started posting tutorials on the RSpec framework with the intention of build a comprehensive guide to Behaviour Driven Development with RSpec. As of this posting, the existing tutorials need a bit of reorganizing, but there's plenty of helpful stuff on there for now.

More to come. Feedback is welcomed and essential.

Current Tutorials:

• Part 1 - Intro to BDD
• Part 2 - A Simple Test
• Part 3 - The Basics
• Part 4 - Mocking
• Part 5 - Advanced Testing Tips

Amazon S3: Simple(?) Storage Service

Some of my ELC comrades and I are developing a site that lets charities create online storefronts to sell books, CDs, and other media. A big part of the development process thus far has centered around importing huge product datasets from the project's affiliated media distributors.

Being an AWS-oriented company, with most of our projects hosted on EC2 via RightScale, the natural choice for a media hosting solution was Amazon's Simple Storage Service, or S3. So, after a few days' work on the less-than-simple task of migrating millions of product images to our S3 account, I've decided to publish my findings so others might have an easier time of it. The following is a summary of the software, Ruby libraries, and tricks I used to get through it.

Software

JungleDisk

Mac/PC/Linux, $20, http://jungledisk.com/

JungleDisk software mounts your S3 as a virtual drive. Sounds cool but the implementation is actually quite clunky and, as you'll read below, proprietary and flawed..

JungleDisk has a feature that allows for "fast renaming". As renaming or moving objects (files) on S3 is a technical impossibility, this feature leverages a hidden JungleDisk-owned EC2 instance. Because the EC2 server is within the same Amazonian cloud, it can serve as a faster intermediary storage tank for files en route to their "new name" or "new path".

Unfortunately, the JungleDisk software uses a proprietary method of storing files on S3. When a JungleDisk-created bucket is viewed using more conventional methods, the file structure is obscure and incomprehensible. See images below..

The mounted virtual disk looks good..

..but this is what's really in the bucket:

Furthermore, JungleDisk cannot recognize S3 buckets that aren't named according to its arbitrary bucket-naming scheme. Boo. Bottom line: JungleDisk is no good for serious developers, and is even of questionable value to a novice performing simple backups, especially considering that it's no longer a free program.

Transmit

Mac, $29.95, http://panic.com/transmit/

A popular (S)FTP client for Mac that recently (version 3.6) added S3 support. I didn't end up using this much because the software imposes a restrictive 'left pane is your stuff, right pane is their stuff' metaphor that is likely helpful for newbies but ultimately frustrating if you want remote file listings ('their stuff') in both panes, e.g. your EC2 filesystem on the left and S3 on the right. Duh.

S3Fox Firefox Add-on

https://addons.mozilla.org/en-US/firefox/addon/3247

S3Fox provides a dual-pane file manager for S3 within Firefox. It seems to work pretty well for managing smaller sets of objects but is ultimately not reliable for doing larger scale transfers or permissions management. I was initially impressed to see the ACL management options, which are actually more functional than some of the standalone software I tried, but upon further inspection it became clear that S3Fox's ACL management is flawed: Though it's possible to set rights on individual objects (files), it appears that S3Fox doesn't really understand the notion of a "directory" within a bucket (who can blame it?), and therefore does not properly handle ACL configuration on "directories". Furthermore, when changing an entire bucket's ACL and clicking "Apply to subfolders", there's no way to know if it's successfully setting permissions on the bucket's contents without hand-checking them. In my case it wasn't, so it wasn't of much use to me.

One bonus: S3Fox provides a handy 'Copy URL to Clipboard' context menu item that produces the fully qualified web address of an object, ie. s3.amazonaws.com/bucket_xyz/path/to/file.jpg

Forklift

Mac, $29.95, http://binarynights.com/

Forklift is awesome! It's a relatively new dual-pane file manager for Mac that supports lots of protocols, S3 included. Unlike Transmit, it allows you to view whatever you want in either pane. It handles large file listings well and it has the most functional ACL management of any of the software I tried out..

This actually works!

Forklift also boasts a highly fuctional and customizable keyboard shortcut system, allowing for more nimble file management than what can typically be accomplished in a GUI.

My only real complaints about Forklift are that it doesn't provide options for file renaming/moving on S3, and that moving a file from EC2 to S3 or vice-versa means downloading the file to your local machine and re-uploading. It would be great to see a future version of Forklift leveraging a behind-the-scenes EC2 instance for quick file renaming and moving (like JungleDisk), or at least direct transfers between S3 and EC2 filesystems. Forklift is allegedly the first file manager for Mac that supports FXP so who knows, they might do something similar for AWS stuff..

Programmatic Approaches

Ruby Libraries

After playing around with various software options with little luck, I decided it was time to move to a more programmatic approach. There are quite a few Ruby libraries out there for interfacing with S3, but in the end I chose Marcel Molina's AWS::S3. Unlike every other library I tried before finding it, it's current, thorough, well documented, and it actually works!

Other Interesting Options

FuseOverAmazon is a fuse filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files "natively" in S3 (i.e., you should be able to use other programs to access the same files).

s3-bash is a small collection of BASH scripts for PUTting, GETting and DELETEing files. I played with it a bit and it works but it doesn't seem to have any ACL support.

Gotchas / Caveats / Pitfalls

Think Sustainably

Be sure to choose a sustainable directory structure and naming convention for your media before you upload it to S3. Once you've created a bucket you can't rename it. You can't rename files. You can't move files. You can only copy them, so choose wisely.

Consider Access Control before Uploading

By default, your objects (files) will not be readable by anyone but their owner. If you want files to be viewable to the public, consider specifying permissions as you upload the files rather than after, especially if you're dealing with a large set of files. The following is an excerpt from the AWS documentation on Access Control Lists (ACLs):

Bucket and object ACLs are completely independent; an object does not inherit the ACL from its bucket. For example, if you create a bucket and grant write access to another user, you will not be able to access the user’s objects unless the user explicitly grants access. This also applies if you grant anonymous write access to a bucket. Only the user “anonymous” will be able to access objects the user created unless permission is explicitly granted to the bucket owner.

Atomize your Directories

Most computers start to get pissed off when you've got more than a few thousand files in a directory. If you're working with huge numbers of files, creating subfolders with smaller files sets will help keep your directories from getting unwieldy and your software from crashing.

Conclusion

Forklift, S3Fox, and Transmit are easy and functional tools for doing small-scale management of an S3 account, but if you're doing serious development and moving around a lot of data you'll probably need to implement a custom hand-written solution. For more ruby/rails-centric S3 material, check out my S3-related del.icio.us bookmarks or check out all the interesting projects and tools labeled with 's3' on Google Code.

..and good luck!