Avoiding the Tar Pits of Localization with Jeff Casimir

In this Ruby HangoutJeff Casimir of JumpstartLab talks about:

  • The tar pits of localization
  • Writing better code
  • Fixing magic data
  • White labeling your app

Contents

  1. Internationalization
  2. Localization
  3. Locale Selection
  4. Bonus: White labeling
  5. Summary
  6. Q and A


1. Internationalization

Even if you’re only going to support one language, internationalization will help you to build a better application.

We’re about to embark on a dangerous journey ..

Internationalization & Localization, i18n, l10n

As a developer considering internationalization and localization, you’re like this mammoth. You’re out walking and you think: “I’ll walk through this water – how deep could it be?” .. and pretty soon, you’re stuck in the tar pit.

The tar pits of localization

That’s how internationalization is. It starts out easy enough, and eventually you’ll be drinking tar.

i18n & l10n

Internationalization, i18n is the one-time process of preparing an application to support more than one locale. In a Rails application, that typically involves going through your controllers, models and views, pulling out all the content strings and replacing them with references to a look-up file or some other means of finding the translated text.

Localization, l10n is the many-time process of taking an application which has already been internationalized and creating a locale for it – US Spanish or Brazilian Portuguese, for example.

Why businesses care
Localization provides a better user experience. The more users you have, the more customers you have. The more customers you have, the more money you make.

Why developers care
Developers typically try to stay away from localization. The tar pit is scary, and making money is generally somebody else’s problem.

The main reason developers care about internationalization is that it results in better code.

Magic Data
Internationalization takes a scalpel to your code.  I’ve written many controller actions like this – something I call magic data.

Magic data

Magic data leads to several problems, one of which is copy-edit commits. That’s where you change a line of code only to fulfil a marketing or copy change demand.

For example when a submit button changes to save, or a flash message changes from your article was created to your article was saved.

That kind of stuff pollutes your code. It looks like you’re making functionality changes – but you’re just changing a bunch of strings.

When you look at a controller action like this, you’re just looking for the magic data, the strings.

It’s one of the reasons I like to use symbols in Rails whenever possible because it separates functional code from magic data. Whenever you see strings, they should be removed. So how do you remove them?

Establish a look-up dictionary
This is what a YAML-based dictionary in Rails looks like.

YAML based dictionary

With this dictionary established in config/locales in a default Rails application, you can utlise those keys with the t helper.

You can access the YAML hierarchy via a dot notation. So this means a top level key of article having a child key of deleted.

YAML hierarchy

The t helper works in a view or a controller context. If you’re not in either of those contexts, you can speak directly to the i18n class, call t on it, and pass in your key that way.

However, that’s typically a warning sign. If you’re calling the translate helper from the model layer, you’re probably doing a bad thing because translation is about presentation and models shouldn’t be presenting things.

I’m not saying it’s wrong to use I18n.t – but do it with hesitation.

Structuring the keys
The first rule is to keep them short – and you do that by avoiding snake-casing the text.

I’ve often seen the text snippet turned into the key by replacing all the spaces with underscores. This isn’t a maintainable way to go. Instead you should focus on the core meaning of the text.

Previously, you saw a key called article.deleted rather than article_has_been_deleted – because has_been is not core to the meaning.

Instead, just focus on the central idea – because that text is very likely to change, but the core meaning of the key is unlikely to change. Also, forget about reusability.

Look-up dictionaries are not programming. Resist the urge to do fancy things with your keys and translations – interpolation, automatic pluralization and singularisation.

The more complex you make your keys and translations, the further down in the tar pit you’ll find yourself. As far as possible, use simple keys and simple strings.

If you want to do something more complicated, the Draper gem can be helpful in handling complex logic at the view layer.

I was inspired to write this gem from the Microsoft community. They have this idea of view models and Draper helps create models that deal with presentation concerns at the view layer.

Magic data in a controller

So, we remove magic data from the controller and use the t helper to replace them with keys. This is what a nicely internationalized controller action looks like.

Magic data replaced by keys

When you look down at the model layer, even more magic data tends to spring up. These are all spots that could be dealt with using internationalization.

Magic data at the model layer

The numbers here are probably better handled in configuration files because they’re unlikely to change between different locales.

If they’re not going to change across different locales then they belong in a config file.

If they are going to change – like that message in line 3 – those belong in a locale file.

In this case, ActiveRecord can help you out – especially when you’re setting a specific validation message.

When an ActiveRecord model fails validation, ActiveRecord will attempt a sequence of look-ups in the current locale.

ActiveRecord look-up hiererchy

This is a description of that look-up hierarchy – the most useful of which is activerecord.errors.messages

In your YAML file, under the messages child key, you can define keys for each built-in validation and Rails will automatically use those strings.

Rails guide messages

One gotcha is that you have to define the keys by what the Rails guide calls message rather than what it calls validation.

In this example, your key would be blank, not presence

YAML structure for ActiveRecord messages

Putting it all together, when I structure my keys like this, I can delete the message line from my model, leaving just validates_presence_of :title

Removing magic data from models

A word of warning, this is an example of where you’ll be tempted to get fancy. You’ll say “I can get the attribute name and dynamically inject it into the message.”

Don’t do it. You’ll regret it.

Just keep it simple. It works fine in English because title is one word. In other languages, it might be two words, you might also have a capitalisation problem, etc etc.

This leads down a road of pain. Just keep the messages simple and use the string you define in the YAML file.

You can also define the print name of your attributes if you follow this kind of hierarchy:

Automatic attribute naming

So, here I’ve given the article model a string of Title and the comment model a string of Your comment.

If you’re using label tags like you should be, along with any other form helpers built in to Rails, they will make use of these strings automatically – you don’t have to monkey around with your forms.

In this example, if you’re just outputting a label, it will show the string Your comment instead of body on the comment form.

By pulling these strings out of my functional code, I have one less reason to change my functional code, which improves its durability, which makes it better.

Even if you’re only going to support one language, internationalization will help you to build a better application.


2. Localization

Let’s say you have a blogging app, and you want to present the articles in multiple languages. Pretty soon you’re going to run in to the problems of creating multiple translations, managing the import and export of the content and keeping these translations in sync.

This is where you start to enter the nightmare mode of localization because of this scenario:

  • Write a blog post in English
  • Send it off for translation to Japanese
  • Make a couple of revisions in the English text
  • Receive back the Japanese translation of the original English text (minus the revisions)

Now the Japanese translation is a slightly different version than the English one – and we’re only talking about two languages. When you’re supporting 12 or 20 languages, this becomes a tremendous issue.

One solution is CopyCopter, formerly a commercial product by ThoughtBot in Boston. It didn’t really work out for them so they open-sourced it. Now you can run your own server.

The model of CopyCopter is that you keep your keys and strings in a separate app. You run a rake task on your primary app to fetch copy from that remote app. Your translators just interact with the remote app, without needing access to your primary app.

A second option is to run it right in your primary application with a gem called Globalize3, primarily authored by Sven Fuchs who also wrote the i18n gem.

Globalize3 works but it makes you realize what a complicated problem this is, because it is a complicated solution.

You end up creating tables for your fields so each field in your primary table will have its own sub-table where each row is a different language – so that they can all be time-stamped and versioned.  Your database schema will be bananas, you’ll have tables proliferating like bunnies.

This third option is my favourite one. Since I discovered this app, I will not use anything else.

To clarify, I have no affiliation with these people other than thinking that they’re bad-asses!

Locale has a gem that you install in your app which talks to their app and manages your translations. It is amazing, so please go out and use it because these people are doing bad-ass work.

i18n, l10n, Rails internationalization and localization

Honestly, I don’t care how much it costs. Once you try those other options, you will start throwing your money at these people to save you from the tar pit.


3. Locale Selection

How do we determine what locale your user actually wants?  The Locale_Setter gem helps you do this, and there are five broad strategies for deciding which locale to display:

  • Geo-location
  • Browser preferences
  • URL parameters
  • Account preferences
  • Default settings

1. Geolocation - sucks, don’t do it.

Think about when you’re travelling – does your current physical location on the planet change the language you want to read on the web? Geo-location is stupid. Please don’t tie a person’s location to the language they speak.

2. Browser preferences – deep down in your browser preferences there are language settings which you can access. Those preferences are submitted with every request made on the web.

If you go to a Rails view template and use the debug helper to output request.env – you’ll see http_accept_language buried in the output.

One of my favourite debugging or investigation techniques is from a controller you can call render: text and then pass it some text. It’ll spit that raw text back to your browser without any wrapping html.

http_accept_language

Here’s a slightly more complicated result:
en-US,en;q=0.8,es;q=0.2

There are three separate comma-separated parameters in this second example.

It means that the user:

  • Prefers US English
  • Understands 80% of generic English
  • Understands 20% of generic Spanish

Because the user submits this information of every request, you can just parse it out with a regex and pull out the locale names.

Regex to parse http_accept_language

You can probably get away with assuming that the language they submit first is the one they want most.

LocaleSetter will give you back an ordered list of locales.

LocaleSetter produces an ordered list of preferred locales

Once you know what language the user wants, you can query the i18n library to see what locales are available.

Querying i18n for available locales

Then the challenge is to match the user’s preferences to the available locales. You find the locales they want and convert them to symbols and compare them to the supported locales (because they’re stored as an array of symbols), and then find the first match in order of their preference (not yours).

This seems like a simple problem. This matcher module takes in requested as an array of the symbols the user has asked for, and matches that against your available locales.

Matching requested and available locales

The single ampersand operator finds the intersection of two sets – the elements that are common to both – and orders them by their position in the first set.

In this example requested will take order priority over i18n.available_locales.

3. URL parameters  can also be used to determine which locale to serve.

Locale parameters in the URL

What’s good about using url parameters is that it makes it really easy to switch locales for development and debugging.

It’s easy to pull the locale parameter out of the url with a simple:
before_filter :set_locale to get back the params hash and look for the locale key. If it’s nil, we leave the current locale unchanged.

Getting the locale from the url

There is a danger here. You can query the symbol module for all the symbols currently defined on a system – and you’ll get back a very large array of hundreds, maybe thousands of symbols.

If you set the locale to some string that the user passed in:
garbage_from_a_user, and then query for it, you will find that it is now set as a symbol.

Dangers of user submitted strings in urls

The danger is that there’s now one more symbol in the symbol table than there used to be. This opens you up to a denial of service attack because it allows users to spam your urls, generate symbols, fill your symbol table and crash your application.

A safer option is to match more intelligently by pulling out the params locale and sending it to the params module which triggers the matcher.

LocaleSetter matcher

Things get interesting in the matcher where instead of symbolising the input and comparing it to the available locales, we stringify the available locales.

Here in the available method, we map to strings which is safer because strings are garbage collected – and they’re our own strings anyway.

We map those strings and compare them to the requested strings.

Mapping requested strings to available strings

Isn’t it wasteful to create these strings?

Yes, but Rails creates thousands of strings – something like 16,000 objects on a Hello World request. So the strings we’re creating here are statistically insignificant – don’t worry about it.

There’s another problem with these locale-embedded links. You embed the locale in the url, click any link on the page and now your the locale is lost to subsequent requests.

Thankfully, Rails provides an easy way to handle this, you just need the default_url_options method.

default_url_options

This allows you to manipulate any url generated from a path helper. If you are a good citizen of your view templates, and you’re only ever using path helpers and never writing your own a tags, path helpers will automatically embed the locale in every url.

But, you probably have a primary locale which serves the majority of your users – so why bother embedding the locale in this default locale? Here, we don’t embed the locale if we’re using the default locale.

4. User preference  You might store locale in the user table and the locale_setter gem checks whether the current user has a preferred locale.

If they have, the gem will pull it out and attempt to match it against the available locales with the matcher module.

Selecting locale from user preferences

Putting it all together. How do we prioritise these locale selection options? Here’s my recommended look-up chain:

  1. URL parameter – this allows you to override your own user preference, despite being logged in, and manipulate the url to control your locale
  2. User preference – If users are logged-in and they haven’t monkeyed with the url, go ahead and use their stored preference.
  3. Browser settings
  4. Default locale
  5. No geolocation

The most interesting part here is the set_locale method which prioritises the params, then the user, then the http and then the default locale.

LocaleSetter set_locale
LocaleSetter set_locale

4. i18n also makes white boxing (white labeling) easy.

Here’s a strategy you could use to get one application serving two different clients

At the top of your application layout, you’ll have lines like this that specify what javascript and stylesheet to use. Those are just strings and all internationalization and localization do is output strings.

White boxing / labeling an app with i18n

You can use the t helper here and make locales per client and set stylesheet and javascript files per client / locale / subdomain, for example.

White boxing / labeling an app with i18n

White boxing / labeling an app with i18n

5. Summary

  • Step 1: Internationalize – carve up your code, pull out the magic data, pull out the strings and get them into lookup dictionaries.
  • Step 2: Localise your user content. Don’t do this on you’re own, you’ll hate life. Just use Locale, its awesome.
  • Step 3: Determine the locale and try out my gem
  • Step 4: Hack i18n beyond translations – anything that has to do with strings, you can do with i18n


6. Q and A

Q: Is there anything which will test localization files to ensure that all lookups are defined for all languages?

A: No, but by default the i18n gem will output the name of the key if the translation is missing. This is good because the app doesn’t crash but it’s bad because users see garbage. You can overrride that exception handler, and I recommend you do so – particularly at the CI level. Run CI through each of your locales and set your exception handler to raise an exception for missing translations. You will catch issues that happen because of translations.

Q: Is there a way to mount a locale using a url holder pattern like /locale/stuff?

A: Yes, it is possible but it might cause some issues. I’m not too wild about using subdomains either.

Q: How do you decide when to keep something in a model as a constant, as opposed to keeping it as a setting?

A: In my opinion it’s always wrong to keep it in the model. From a code quality standpoint it is always wrong to have numbers in the middle of your functional code – because that number has a domain meaning and it should have a name. It shouldn’t be 6. it should be front_page.articles_limit. The only reason we put that 6 there is because we’re lazy. You write that code once, but you read it and debug it many times. You’re better off typing the long thing. So, the first step is to go through code and move the integers up to constants. Once you see all the constants, you realise that if you wanted to change them you could change the source code or monkey patch it with an initialiser. That could work, but if you’re going to write an initialiser you might as well just write an initialiser. So, take those constants, bring them over to an initialiser and handle configuration data there.

Q: Whether you’re changing a constant in the model or data in the initialiser, you’re still making code changes?

A: Yes, same repository, but different conceptual area. We’re changing the parts of the code that change when configuration changes, but we leave the models alone which reflect business logic.

This entry was posted in LocaleApp and tagged , , , , , , , , . Bookmark the permalink.

9 Responses to Avoiding the Tar Pits of Localization with Jeff Casimir

  1. Paritosh says:

    Awesome, thanks for full explanation. I found it very helpful, even bookmarked it for later reference.

  2. Peter says:

    Using an URL part for specifying locale (like site.com/en/info) has a nice result that search engines now index a version of your site in every language and not just the default one (or worse, some random one determined by geolocation if you use it).

    • admin says:

      Good point. Subdomains also a useful option here too – something like de.domain.com – but Google currently treats subdomains as unique in terms of ranking power so. de.domain.com will have it’s own ranking weight distinct from http://www.domain.com. Interlinking them can pass the Google juice around a bit though.

  3. Jan says:

    Great writeup! It would be fair though, to note somewhere that the author seems to work for locale (at least this is where this post is hosted), and thus a certain bias can be assumed…

    • admin says:

      Hi Jan – nope, Jeff doesn’t work here and isn’t affiliated with us at all. We just asked his permission to write up his presentation and post it here. First we knew about Jeff’s praise for Locale was listening to the Ruby Hangout.

  4. Henrik N says:

    Q: Is there anything which will test localization files to ensure that all lookups are defined for all languages?

    A: Yes, e.g. https://gist.github.com/2994129 (via my “Rails i18n tips” blog post).

  5. Steven Ringo says:

    I don’t see how localeapp can work as a substitute for the Globalize3 gem. The gem works to localize database-backed data and not the data in yml files.The localeapp gem only handles yml file translation.

  6. Balint Erdi says:

    Hey,

    To take out the pain out of manually replacing the strings in your templates with the I18n.t message strings, I’ve created the i15r gem: https://github.com/balinterdi/i15r. It saves you a ton of time, the more templates you have the more.

    Should you decide to try it, please leave a comment or open an issue in the repo.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comment moderation is enabled. Your comment may take some time to appear.