FINN Technology

Haskell at FINN.no

2018-10-18T09:00:00+00:00

Favorites

FINN has this feature where users can mark classified ads as a favorite, making it easy to come back to certain ads later. The feature has been around quite a while, and the systems involved, both frontend and backend were becoming quite hard to maintain. So the AdView-team set out to redesign the whole stack. This blog post will focus on the backend API, which is written in Haskell.

Haskell

Haskell is a purely functional programming language, with a powerful type system. The ability to express intent using types brings correctness, and the composition of a large program as small, independent building blocks makes it easy to reason about the code.

A large ecosystem of production grade libraries are available from Hackage. We make use of Servant (API), Aeson (JSON), Hasql (postgres) and many more. Servant is a type-level API, meaning the “sum” of all our endpoints become a distinct type. This, in turn, means that refactoring our API gives us heavy compiler assistance. More than once during development, we had the need to do major changes in the endpoints (verbs, paths, request body, responses) and found that once every compilation error was resolved, the new API was working correctly.

To manage external packages, build and test code, we use Stack, because of its familiarity to maven, gradle and sbt-users. stack build will compile the project, stack test run all the tests. Stack pulls down Glasgow Haskell Compiler (“GHC”) along with required libraries. It also supports building and running of the final program in Docker.

Now Docker is important, because of the cloud infrastructure in FINN; any technology running in Docker can be used as a new Micro Service.

Experiences

So what are the downsides to using Haskell? Well - there is really only one. We have to have a certain amount of developers, at least in our team, know Haskell. We meet this challenge in three ways. Firstly by doing a Haskell course internally at FINN. 21 of our developers have signed up for an Introductory Haskell Course from the University of Glasgow, secondly we cheated a little by simply recruiting another Haskell developer. And lastly, we will put some effort into the Oslo Haskell Meetup group, hopefully spawning even more Haskellers.

Really? No more downsides? Well, it should be mentioned that our build-times on Travis CI (self-hosted) are not super-awesome-great. We are currently looking at 8-9 minutes for a complete build (including integration tests). The Stack build tool uses Docker for every task, and some times needs to pull a new version of an image. This hurts build times, but we can live with this as it gives us benefits in terms of isolation.

As for performance, the Warp web server is doing an excellent job of spawning lightweight threads and keeping CPU and memory usage low. An indication of memory and CPU usage is given below. Note that the old API still have way more traffic, so a direct comparison is very unfair!

favorite-api is the new API (Haskell).
classified-favorite-management-server is the old API (Java, hence the long name).
You can barely see the 32 MB memory footprint of the new API in the graphs!

Check it out

We are really looking forward to putting more load on our new API, and doing more Haskell in the future. You can check out the redesign of favorites for yourself. And if you would like to learn some Haskell, a great starting point is Learn You A Haskell For Great Good

Preparing your product for machine learning

2018-08-09T08:00:00+00:00

At many companies, there are few data scientists and many projects which may be very interesting for the business to apply machine learning on. For most of these companies, starting greenfield projects with a data scientist on the team might be either a difficult political battle (to get assigned resources) or impossible, if there are no data scientists yet.

But how can a team prepare their project for the day when there will be a data scientist around, and in the process, maybe improve their current product as well?

In Google’s “Rules of ML”, they make some good suggestions for this in their introduction:

Do machine learning like the great engineer you are, not the great machine learning expert you aren’t. Most of the problems you will face, are in fact engineering problems. Most of the gains come from great features, not great machine learning algorithms.

This means, the team already has great potential to improve their product in ways that the data scientist might help them do in a more structured way later. I believe as long as the engineers and programmers know how machine learning algorithms like having the data shaped, they can take advantage of this to move forward confidently.

What kind of data does a machine learning algorithm like?

Computers like things that are quantifiable by numbers, and this is probably obvious to many programmers, but it’s easy to forget this when we have tools like Google that can find and seemingly understand what we want just by typing a couple of words.

According to a Kaggle survey from 2017, “dirty data” is the biggest problem faced by machine learning practitioners. This means data being unstructured in some form, either by being severely skewed in one direction or the other, being full of holes (only subsets of users even have a given feature), or data that has to go through processing before actually becoming a usable feature. Making sure that this friction is minimized is important later for the progress of machine learning functionality, but it also enables teams to make more use of heuristics right away if the data is better structured.

So, what’s important to keep in mind when tracking or saving user information that you plan on using to predict their behavior?

Is the data numerical?

Again, this cannot be understated, computers like numbers. So if at all possible, allow users to input data such as:

Q: “How many years have you studied at university?” A: “4”
Q: “How much do you like cookies?” A: “0-100”

This is easily malleable, and very easy to interpret for ML algorithms.

Is the data categorical?

If you can’t make your data into a number, the second best thing is to make it into a category, which can be almost like a number from a computer’s perspective. Turning our previous numerical example into a category could look something like this

Q: “What is your highest achieved degree from University/College?” A: “Bachelor’s”.

From the ML model’s perspective, we can simply represent each category as a number. For example, if we have “Bachelor’s, Master’s and Ph.D.” as possible category answers here, it would be representable by “0, 1, 2” if we keep these ordered in the same way going forward.

As long as you properly restrict your categories, and provide some nice UI/UX for the user to interact with, this will do just as well as a numerical approach. Just don’t let users type in whatever they like as their categorical answer, such as “Bachelor” “B.A.”, “About f0ur years” etc, as this will doom you to forever clean up data afterwards before using it. (if you think the last option was a joke and didn’t fit the question at all, you’re right, but users will eventually write something similar)

UI - Making categorical/numerical inputs work

There are some very simple, yet powerful things you can do with your UI to facilitate the usage of categorical variables.

Use auto-complete for your inputs, instead of free text inputs. This allows the user to write in a natural way, but only select from the subset you have defined.
Use sliders for scales from 0-X for numerical inputs.
Allow the user to connect via an integration to some external service to retrieve metrics if possible.

Can we process the data to make it numerical?

This mostly involves using NLP(Natural Language Processing) to process text information, and CV(Computer Vision) to process images. For most teams and organizations this is more of a last resort, as it involves an extra layer of complexity, required competency and up-front work before actually starting to make use of the data.

Object detection in action

Image recognition is getting to the point of almost being able to recognize anything in a picture that you yourself can recognize in it, such as “Is there an orange in this picture?”, “how many people are in the image?” and “how are the people standing?”. NLP however still has a ways to go before interpreting texts in the same way as humans, so for most teams I would highly advise against this.

If you’re thinking about adding a text field because it might give you good heuristics to model your users behavior with normal programming, or allow something to be searched, remember that it can probably not be used in machine learning. Which means you’ll have to redo the work of capturing that feature some other way, and your data scientist might be sad.

Good to go?

As long as you can shape your data into something like this, you are pretty much good to go in terms to passing the data to a machine learning algorithm. Keep in mind that you should already have a close relationship with which indicators you are hoping to improve on as well. This could mean instrumenting the collection of data for things such as click through rate, revenue or number of conversions. This will help the data scientist that comes on board (and your team) to easily gain confidence in that ML is the right thing for you. For some teams, plain old programming might be enough, and better (depending on your needs / resources / data availability).

A few caveats

Also remember you need to have enough data to be able to tell something about the true distribution of what you’re trying to describe. If I have the sales prices for 3 Hondas, and I’m trying to predict the estimated sales price of a Honda CR-V 2015 edition, I hardly have enough information to reliably tell what it should be worth. Here you might find traditional statistics helpful to figure out when you have enough data. However if you have millions upon millions of examples of something, usually you should be fine, no matter what you’re trying to figure out about the world.

Another smart thing to keep an eye out for is, “Can you yourself execute the task of what you want ML to do for you, given your available data?” (on a small sub-sample of course). If I have only the brand name, and edition year for the Honda that I’m trying to predict the proper sales price for, I probably don’t have enough data, as most people are interested in the milage of the car, what extras are included, color of the paint-job, and so on.

Is the feature you have selected affected by time? Then you might need more complex models to model it, and proper time-keeping at the data-collection stage.

Is the feature affected only by you, and nobody else? Or is it dependent on the way your UI works, how another team tracks info to their database, or is it depending on something your other models already affects? These are important things to keep in mind before drawing too decisive conclusions about what the data tells you, and might also bias your model later.

And how much data should a team save? I would say, save as much as you can without annoying users and trampling privacy laws such as GDPR. It’s a well known fact that more data, means better results in the long run. Just don’t reduce the user experience to collect data you don’t know you’ll need later. Don’t set out to track everything about users right away dismissively thinking that as long as you have the data, AI can solve all your problems later. Remember to make this an iterative process your team or domain experts are a part of.

In conclusion

Normal programming can usually get you close to your goal, or even completely to your goal depending on what you want to achieve. However even if you end up replacing parts of your approach later when you get a data scientist, it is not in vain as you will have discovered useful features, and have a mountain of data the data scientist can sift through.

Just remember to format the data in a way which makes it easily usable by computers, and keep track of the metrics you want to improve ahead of time! Without this, even the normal programming approach will be hampered.

If this is something that interests you, or that your team is struggling with I recommend checking out Rules of ML by Google authors and/or Why you need to improve your training data by Pete Warden. Facebook even dedicated a purely to data to it in their new machine learning field guide.

Also, if you’re interested in machine learning and programming, I usually tweet whatever interesting stuff I stumble upon at @JoakimRi.

Thanks to Thorkild Stray and Simen Eide for reviewing earlier drafts of the post, and providing feedback!

5 learnings from Hack Days in FINN

2018-06-28T08:00:00+00:00

Personalized search with a custom Solr plugin

2018-04-10T06:00:00+00:00

Background

On FINN.no people can search for classified ads, where the backend system is using Solr as the search engine. Default sorting on the vertical “torget” is by relevancy, which is based on Solr score. The Solr score for a document is again calculated from query relevance and the ad’s published date. Here is an example searching for the word chair:

Non-personalized search

The first and third ad are bought positions, while the rest are sorted by published time and the importance of the word chair (Norwegian: stol).

In the fall of 2017, we started experimenting with ways to improve the relevancy sorting. Our first try was by boosting geo distance, ads close to my position would get a higher score. But we could not see any positive changes for our product KPIs. Then we wanted to try sort by mixing scores from both Solr and our recommendation system. Our recommendation system already had an api where we could send a user id and a list of ad ids, and receive a recommendation score for each of the ads.

A new solution

We evaluated a few different solutions, mainly:

learning to rank in Solr
custom Solr plugin using the existing recommendations api

Since we already had a system for recommendations, and awesome data scientists tuning the algorithms and so on, we chose to test the latter. An uncertainty was if we could get the response times needed for a search.

Solr SearchComponent

A Solr search component contains several phases used by the search handler. As we do not use sharded indices for the search we wanted to test, these are the important phases:

prepare - Preparing the response -> parsing request parameters
process - Processing the request for the current component

Requirements

the personalization score should influence the order of the search result
we want to easily be able to change recommendation algorithm, and test a new one against the current search (with or without personalization)
we also wanted to tune Solr score/recommendation score balance

We therefore added these parameters:

personalization (Boolean) - to switch on/off the personalization search
recommender id (String) - to set recommendation algorithm
rerank pages (Integer) - number of pages with personalization per search, typically the first 5 or 10 pages will be personalized. This is to increase performance, compared to calculating personalization scores for every document in a search result.
personalization score weight (Double) - recommendation score weight, 1= equal weight to Solr score
user id (String) - to be able to personalize the search

The solution we chose for the plugin is:
1) Let the search fetch all documents from the first x pages, set by the parameter rerank_pages.
2) Fetch recommendation score for each document returned by the search from the recommendations api.
3) Sort the documents by a combination of Solr score and recommendation score.
4) Return a subset of the search result, based on offset and rows parameters

We want the search from the default QueryComponent to run before the PersonalizationComponent, since we need the ad ids (Solr document ids) for the first x pages of the search result. Therefore we setup our component as a last-component in the Solrconfig.

<requestHandler name="dismax" class="no.finntech.search.Solr.searchhandler.SearchHandler" 
    default="true">
    ..
    ..
    <arr name="last-components">
        <str>personalization</str>
    </arr>
</requestHandler>

SearchComponent.prepare

The prepare step are evaluating and setting the search parameters count, offset and score.

@Override
public void prepare(ResponseBuilder rb) throws IOException {
   if (!shouldWePersonalize(rb.req)) {
       return;
   }

   final SolrParams params = rb.req.getParams();
   final SortSpec sortSpec = rb.getSortSpec();
   final int offset = sortSpec.getOffset();
   final int reRankNum = getReRankNum(params);

   if (reRankNum > offset) { //reranking only the first x (reRankPages) pages
       sortSpec.setCount(reRankNum);
       sortSpec.setOffset(0);
       if (params.get("score") == null) {//adds score for the search
           rb.setFieldFlags(SolrIndexSearcher.GET_SCORES);
       }
   }
}

First we are checking the parameters if it is a personalization search. This component will not manipulate a search without personalization.

if (!shouldWePersonalize(rb.req)) {
   return;
}

The personalization search will reorder the first x pages, set by the parameter rerank_pages, therefore count and offset is set to include the first x pages of the search result.

if (reRankNum > offset) { //reranking only the first x (reRankPages) pages
           sortSpec.setCount(reRankNum);
           sortSpec.setOffset(0);

Setting the flag which makes the QueryComponent calculate scores:

if (params.get("score") == null) {//adds score for the search
    rb.setFieldFlags(SolrIndexSearcher.GET_SCORES);
}

SearchComponent.process

In the process step, the recommendation scores are fetched, and the final sorting of the search result are set based on Solr- and recommendations scores.

@Override
public void process(ResponseBuilder rb) throws IOException {
   if (!shouldWePersonalize(rb.req)) {
       return;
   }
   final SolrParams params = rb.req.getParams();
   final String recommenderId = params.get(FINN_PERSONALIZATION_RECOMMENDERID);
   final String userId = params.get(FINN_USERID);
   final int rows = params.getInt(ROWS);
   final int offset = getOffset(params);
   final float personalizationScoreWeight = getPersonalizationScoreWeight(params);
   final int reRankNum = rb.getResults().docList.size();

   if (reRankNum <= offset) { //only reranking first x (reRankNum) documents of a search
       return;
   }

   final DocList initialSearchResult = rb.getResults().docList;
   List<ScoredAd> results = getInitialSearchDocs(rb, reRankNum, initialSearchResult);

   final Map<String, ScoredAd> recommendedItems = 
        getRecommendedItems(userId, recommenderId, reRankNum, results);

   if (recommendedItems.size() > 0) {
       results.forEach(result -> {
           final float normalizedSolrScore = normalizedSolrScore(result.getScore(), 
                                                                 initialSearchResult.maxScore());
           final float personalizationScore = personalizationScore(personalizationScoreWeight, 
                                                                   recommendedItems, 
                                                                   result.getAdId());
           result.setScore(normalizedSolrScore + personalizationScore);
       });
       results.sort((r1, r2) -> Float.compare(r2.getScore(), r1.getScore()));
   }

   final List<ScoredAd> searchResult = results.stream()
           .skip(offset)
           .limit(rows)
           .collect(toList());
   try {
       addToResponse(rb, reRankNum, searchResult, initialSearchResult);
   } catch (IOException e) {
       LOG.error("Could not add recommended items to search result: " + e);
   }
}

As in the prepare step, we start the process step by escaping non-personalization searches, checking if the search is a personalization search, and if the search is within the pages to personalize:

if (!shouldWePersonalize(rb.req)) {
   return;
}

if (reRankNum <= offset) { //only reranking first x (reRankNum) documents of a search
   return;
}

The QueryComponent runs its process step before the PersonalizationComponent’s process, and creates a doclist containing the search results, with lucene document ids and scores.

final DocList initialSearchResult = rb.getResults().docList;
List<ScoredAd> results = getInitialSearchDocs(rb, reRankNum, initialSearchResult);

Note that Lucene document id is not the same as Solr document id. Next, to get the Solr document ids, which we will use for recommendation scores, we need to do a index/document cache lookup per document. java searcher.doc(luceneDocumentId, <Set of fields to obtain>) is one of the more costly operations in this plugin. The Lucene document ids will change when anything in the index changes, it is therefore not trivial to cache these lookups.

private static List<ScoredAd> getInitialSearchDocs(ResponseBuilder rb, 
                                                   int reRankNum, 
                                                   DocList initialSearchResult) 
                                                   throws IOException {
   List<ScoredAd> results = new ArrayList<>();
   int counter = 0;
   final SolrIndexSearcher searcher = rb.req.getSearcher();
   for (DocIterator docIterator = initialSearchResult.iterator(); docIterator.hasNext() && 
            counter < reRankNum; counter++) {
       results.add(createDoc(searcher, docIterator.next(), docIterator.score()));
   }
   return results;
}

private static ScoredAd createDoc(SolrIndexSearcher searcher, 
                                  int luceneDocumentId, 
                                  float score) 
                                  throws IOException {
   final Document doc = searcher.doc(luceneDocumentId, Sets.newHashSet("id")); //there should be a better way to do the luceneid -> id mapping
   String adid = doc.get("id");
   return new ScoredAd(luceneDocumentId, adid, score);
}

Next we use the recommendation api to get a personalization score for each document. This is a http-request, and is therefore also a high cost part, latency vise, of the plugin. Our api takes the following parameters:

UserId
RecommenderId
List of AdIds (Solr document ids)
MaxRows

And returns a list of adIds with recommendation score. ScoredAd now contains both the Solr- and the recommendation-score. These are further used for sorting the search results: total score = normalized Solr score + (normalized recommendation score * recommendation score weight)

final Map<String, ScoredAd> recommendedItems = getRecommendedItems(userId, 
                                                                   recommenderId, 
                                                                   reRankNum, 
                                                                   results);

if (recommendedItems.size() > 0) {
   results.forEach(result -> {
       final float normalizedSolrScore = normalizedSolrScore(result.getScore(), 
                                                             initialSearchResult.maxScore());
       final float personalizationScore = personalizationScore(personalizationScoreWeight, 
                                                               recommendedItems, 
                                                               result.getAdId());
       result.setScore(normalizedSolrScore + personalizationScore);
   });
   results.sort((r1, r2) -> Float.compare(r2.getScore(), r1.getScore()));
}

Finally we are preparing the search response with a subset of the search results, set by offset and rows.

final List<ScoredAd> searchResult = results.stream()
       .skip(offset)
       .limit(rows)
       .collect(toList());
try {
   addToResponse(rb, reRankNum, searchResult, initialSearchResult);
} catch (IOException e) {
   LOG.error("Could not add recommended items to search result: " + e);
}

private static void addToResponse(ResponseBuilder rb, 
                                  int reRankNum, 
                                  List<ScoredAd> resultsToReturn, 
                                  DocList docList) throws IOException {
   DocListAndSet docListAndSet = buildDocListAndSet(reRankNum, resultsToReturn, docList);
   rb.setResults(docListAndSet);

   BasicResultContext resultContext = new BasicResultContext(rb);
   SolrQueryResponse rsp = rb.rsp;
   rsp.getValues().removeAll("response");
   rsp.addResponse(resultContext);
}

Response times

We will also look at the response times:

Response times for 95th percentile. Default search is search without any custom plugins, distribution search is the actual search we are trying to replace with personalization search. Distribution search is a search where every xth (typically every 4th) document is a professional seller ad, this is performed by doing one search for professional sellers and one for non-professional sellers, and merge them together by adding a professional ad for every xth position. The actual timings in the 95th percentile doubles from 225ms to 450ms. While the basic search takes around 80ms.

95th percentile latency

Lets also look at the 99th percentile for response time. We here removed the default search, because of too much noise, because of low traffic. The personalization search’s 99th percentile is between 60 to 100 ms slower than the search to be replaced.

99th percentile latency

Search examples

Here is a couple of examples of different user doing the same search at the same time, which will result in a completely different order of the ads:

Personalized search with a user interested in antiques

Personalized search with a user interested in furnitures for children

We are now doing a personalization search for the 5 first pages of the search results. Both of the searches includes the same 250 ads on the first 5 pages, but as you can see, the order is completely changed between the two examples. Published dates and the query relevance decides which subset of the search result to personalize.

Product results

We made a version of the personalization search, which included distribution, to take care of the professional sellers. The personalized search with distribution were tested against the distribution search for 4 months, increasing the share of users getting personalized search to 50% the last couple of months. These were the results from the most important KPIs we followed:

	Distribution search	Personalized search with distribution	Difference
Share of sessions with ad click	59.7%	62%	+3.9% (2.3% points)
Time from start of search to ad click	63 s	61.7 s	-2.3% (1.3 seconds)
Clicks to ads without any contact from buyers, per session	2.1	2.58	+22.9%

All of these were better with the personalized search, in addition our other KPIs were unchanged.

Summary

To understand what kind of data is populated in each of the SearchComponent’s steps, a lot of debugging the default Solr components was necessary, because of incomplete documentation.

The personalization search makes our search better for our users in FINN.no. But it does indeed costs more in terms of latency. Here we hope to be able to increase the performance a bit. There are several possibilities:

tune the document cache / index commits
cache recommendation matrices within the Solr servers, to avoid network overhead
find a better way to get the Solr document ids
hardware resource tuning
learning to rank feature in Solr
…and so on

We will still tune our recommendations algorithms and the plugin to get our results even better. As of April 2018, all users of the “torget” search gets the recommendation search.

Åpen fagkveld hjemme hos FINN

2018-02-06T14:14:30+00:00

Tradisjonen tro åpner vi også i år dørene hjemme hos oss i Grensen for å dele våre erfaringer rundt hvordan vi jobber med produktutvikling. Teknologi er i fokus, og vi gleder oss til å vise dere hvordan vi bygger - og holder et av Norges mest besøkte nettsteder vedlike. En kveld er dedikert studenter, og en de mer erfarne.

Presentasjoner

Åpen fagkveld 28. februar kl 16-20 (for de som jobber med teknologi)

Program

16:00	Velkommen til FINN ved Nicolai Høge (CTO)
16:30	Mat og drikke
17:00	Sosial rekruttering Hvordan vi tok i bruk våre egne data for å lage FINN Jobbmatch.
17:30	GDPR Samtykke - Hvorfor må vi ha det og hvordan håndterer FINN det i en mikroservices-verden?
17:30	Unleash Verktøy/rammeverk for A/B testing.
18:00	Oppetid Hvordan arbeider vi for høy oppetid i FINN. Ambisjoner, målinger og oppfølgning.
18:00	Podium Frontend i en mikroservices-verden. Hvordan FINN bygger en unison frontend på kryss av microservicer og team med forskjellige eierskap.

Åpen studentkveld 7. mars kl 16-20 (for studenter)

Program (endringer kan komme)

16:00	Velkommen til FINN ved Nicolai Høge (CTO)
16:30	Mat og drikke
17:00	Dette ser vi etter når vi ansetter utviklere
17:30	Sosial rekruttering Hvordan vi tok i bruk våre egne data for å lage FINN Jobbmatch
17:30	Universell utforming - hvorfor og hvordan man skal lage noe som funker for alle. Absolutt ALLE!
18:00	Unleash Verktøy/rammeverk for A/B testing
18:00	Podium Frontend i en mikroservices-verden. Hvordan FINN bygger en unison frontend på kryss av microservicer og team med forskjellige eierskap.

Takk til alle dere som kom.

Karrieresiden vår inneholder mer info om oss.

Browser statistics December 2017

2017-12-13T10:02:32+00:00

Browser statistics, December 2017 at FINN.no#

Yuletide is coming real soon now, and we haven’t published browser statistics for 2017 yet! The 2016 edition is here if you want to compare.

How many visitors use a desktop or laptop?

The trend continues, a higher percentage of the traffic is coming from a mobile terminal this year than in 2016, but the trend is that the growth of the curve seems to flatten.

Our first graph shows the share of our users using a mobile phone, tablet or a desktop computer to access FINN.no, regardless of whether they use our responsive web app (m.finn.no), or a native Android or iPhone app. 72% of our visits are now from a smartphone or a tablet. The traditional desktop/laptop has a market share of 28%.

Which offering/”app” of FINN are people using, regardless of whether the device is a desktop/laptop, mobile or tablet?

Given that a user is on a mobile or a tablet - are they using the web-version or the app? For the first time the majority of those accessing FINN from a mobile device use our native app - the www.finn.no share of the visits is down to 48%.

These graphs also shows that we changed the domain name again this spring, so that all web traffic is served from www.finn.no, and not m.finn.no any more.

Browsers and devices

The browser war is now only a memory of the veterans, but we still have to debug bugs in specific browsers from time to time, and it’s useful to have an idea of which browsers could or should be ignored.

Let’s also look at the which devices are the most popular:

Operating systems

We’re not using our mobiles all the time, so lte’s take a look at the operating system versions. The changes here go quite slowly, but the longer trend is that Windows 10 is still increasing over Windows 7 and 8.

Distribution of iOS versions - it’s interesting to see how quickly a new version is rolled out and dominating. (Note that iOS 11 did not fit into this graph, but the last week, iOS 11.1.2 was the biggest.)

…especially compared to Android:

A list fits more elements than a line graph, so here are the top 10 iOS and Android versions the second week of December 2017:

There are many models of iPhone, and though we have some doubts about the correctness of these data, this report still gives an indicator.

FINN wins Universal Design Award

2017-10-30T11:19:07+00:00

Family and Equality Minister Solveig Horne handed out the award to FINN employees from the Accessibility Team, Product, Innovation and Technology. From left: Maiken Solberg Olsen, Solveig Horne, Lotte Johansen, Tor-Martin Storsletten, Karina Birkeland Lome, Ingrid Vestby Fredriksen, David Håsäther, Nicolai Høge and John Arne Bjerknes (Jury leader).

By removing obstacles and involving users, FINN.no has created a service that allows everyone to participate in important social functions. On October 17th we received the DOGA Innovation Award for Universal Design in the Interaction Design and Digital Solutions category. Developers and designers run the grassroots work with accessibility at FINN.no. It started in 2014 when a new accessibility legislation was introduced in Norway. Since then the grassroots group has worked to make accessibility known to everyone who develops FINN.no. Sometimes the work gets hard, and it is extra encouraging to receive such an award. It helps us to show how important it is that we make our services available to everyone.

FINN.no has received the award for its web solution and corresponding mobile app.

House and Work for Everyone

It is a major responsibility being the country’s largest digital marketplace. If you do not have access to, for example, the job or real estate market on FINN.no, your opportunities are limited to participate in the community.

FINN.no is used by virtually everyone in Norway. Therefore, it is important that people can find a house to live in, a job to go to, trips, or other items they may need, regardless of any disabilities. Through user tests we have learned that where people with disabilities are struggling, others also tend to struggle. By removing obstacles, the solutions become better for everyone.

Images and icons are widely used in our online solution. Designs and texts are made to be as intuitive as possible, among other things by using self-completing text. The FINN.no app is adapted for use by screen readers, which are often used by blind and partially sighted people. The app easily integrates with the phone’s features, and supports push messages and GPS location.

In addition to the accessibility team teaching accessibility to our colleagues, FINN.no designers provide a design framework where developers can find elements used at FINN.no. These elements are all checked for contrast and other accessibility issues.

A user test regarding people with disabilities made it obvious that there was a confusion between My messages, My ads, My searches and My findings. The test made it clear that this confusion was a problem for everyone, focused our efforts on solving the problem.

Increases Competitiveness

The Innovation Award for Universal Design was awarded for the third time. The category winners also competed for a main prize. FINN.no competed for this award against the online retailer Kolonial.no, the children’s newspaper Aftenposten Junior, a changing room facility at Solvik camping, Hamaren Activity Park, the Vision of the Fjords sightseeing boat, and the soundproofing felt FeltRoll. The Vision of the Fjords won the main prize.

Universal design allows everyone to participate on equal terms. The seven categories are good examples of how universal design triggers innovation, boosting competitiveness and providing a more open and better society, says Program Director Onny Eikhaug in Design and Architecture Norway (DOGA).

Far Ahead of the Competition

In the jury celebration, FINN.no was praised for giving users a simple, safe, and engaging way to buy and sell items:

“FINN.no is innovative by being ahead in using new technology, and over the years has developed an entire ecosystem of services that constantly improve and develop further. Both the web solution and the FINN.no app rank high on ease of use and universal design, and are far ahead of many other digital products”, the jury writes.

(based on and translated from Pressenytt article in Norwegian on FINN wins Universal Design Award) by Henning Poulsen

FINN tweet after winning the category interaction design and digital solutions

Akka workshop at Finn.no

2017-10-17T06:56:49+00:00

Akka workshoppers hard at work!

In FINN.no we have experience in running distributed systems, and we often rely on asynchronous communication between our applications. Apache Kafka is the work-horse on the house, and it’s doing an excellent job for us.

But this doesn’t mean that we shouldn’t learn new things - or look at other solutions.

Akka is an implementation of the Actor Model - a concept of concurrent processing first published in 1973. The Actor Model is all about two ideas: Actors and Messages.

Actors sending messages

A number of Actors communicate to each other using messages. An Actor will process its received messages sequentially, and send messages to other actors asynchronously. It can also keep state, perform side-effects and supervise other actors. This small hand-full of operations makes the Actor Model easy to reason about and within one Actor we need not worry about the complexity of concurrency - one message is processed after another.

The power of the Actor Model comes from combining (often specialized) Actors in a supervised hierarchy. Concurrency and scalability comes naturally as the sending of messages between them are asynchronous. Reliability and uptime comes from a let-it-crash philosophy and supervision done through special Supervisor-Actors.

Arild wrapping heads around the async monad!

The workshop was a hands-on with a few slides between the exercises. You can do the tasks yourself by cloning and checking out the master-branch (for java) or the kotlin-branch.

Slides Resources

Deep NLP-based Recommenders at Finn.no

2017-09-08T13:56:49+00:00

During a hackathon at FINN.no, we figured we wanted to learn more about deep NLP-models. FINN.no has a large database with ads of people trying to sell stuff (around 1 million active ads at any time), and they are categorized into a category tree with three or four layers. For example, full suspension bikes can be found under “Sport and outdoor activities” / “Bike sport” / “Full suspension bikes”.

In our daily jobs we are working on recommendations. There, we already have a content based (tf-idf) recommender build on Solr’s More Like This. It seems to work well in areas where our collaborative filtering approaches does not. Would it be possible to build a deep learning NLP-model of similar performance?

To achieve a measure of similarity, building a classifier of the previously mentioned categories seemed like a good choice, since we already had a lot of pre-existing data. The NLP team at Schibsted had already tokenized around six million ads as well as trained a word2vec model for us - we were ready to roll!

Some preprocessing still had to be done. We ran through all ads, concatenated the title and description strings, and after a quick look at the data took the first 15 words of each ad.

Model architecture proposed by the paper

Our initial experiments were done with a simple “Bag of words” model included in the Keras repository, but we promptly switched over to “Convolutional Neural Networks for Sentence Classification” based architecture after hearing about it from our colleague, Tobias. By looking at the first 15 words of the ad, and using 200 dimensional embeddings for each word, our input is transformed into a 15x200 matrix. We apply three different convolutions on each document. The three convolutions looks at 2, 3 and 4 words (kernel sizes) in each convolution. It then max-pools each over the whole document, so that you end up with one value per document per convolution. For each kernel size you do 100 different filters. Finally you add a dense layer for classification. In addition to the standard model described in the paper, we experimented with different kernel sizes, number of filters, dense layers, batch normalization and dropout. We also added several losses, so that the model optimized both the higher and lower category at the same time. That helped.

Keras representation of our NLP model

So how did it go? Our hackathon model managed to categorize 10’000 ads into 359 categories with an accuracy of 50%. We were surprised it worked so well, it was about the same accuracy image models have achieved on roughly the same ads. After tuning and pruning it further and adding 1 million data points, we have reached an accuracy of 70% on the 359 classes. In comparison, the bag-of-words model we started with managed an accuracy rate of around 25% and image recognition models have reached accuracies around 50%.

Using the model in recommendations

It is usually the case that category-similarity translates decently to ad-similarity. Using our classifier model we can serve users more ads similar to what they’re already seeing, based on the text of a selected ad.

We use the model by cutting the last dense layer (called feature layer in figure above), then comparing normalized dot products (cosine similarity) between objects. Since our our benchmarks for judging anything a success or failure is based on how it performs in a production environment, we went ahead and did that. This gave us decent results using only text, performing about 5-6% percent worse than our top collaborative filtering approach. When we made an ensemble model combining text and collaborative filtering we managed to improve our existing best model by about 10%. This is likely due to better supporting “cold ads”, or ads without traffic, while still retaining the accuracy of the collaborative filtering-model.

An example of a “cold ad”, where we think our NLP model does a better job at finding relevancy than the traditional collaborative filtering approach

Collaborative filtering recommendations

NLP Recommendations

Further work

The pure text-model does not prioritize the popularity (or perhaps by proxy, how good the ad is) of the ad at all. This leads us to suspect that although users are being directed to similar ads, they could for example be missing an enticing image to make engagement likely. Seeing how the ensemble model in the end is optimized for click-rate, it likely only gives the NLP model high priority when the ad has low traffic. It would be interesting to somehow introduce this aspect into the NLP model.

We would like to eventually have a more thorough NLP representation of all our ads for other teams to build services and functionality on, and this recommender is an important first step to achieve that.

The code can be found at FINN.no’s github page.

Resources

The summer interns of 2017

2017-08-08T10:56:49+00:00

The starting point

In a galaxy very very close by, there were 5 brave summer interns working for FINN.no. They came from the far north (NTNU) and the far south (UiO) of the cold and rainy plains of Norway. 4 of them were taught in the arts of the computer, and one in the teachings of the design. They had a mission: to improve the current user experience for advertisers on FINN Torget, and to decrease the amount of zero-contact ads. A tough mission, but our heroes have valiantly fought during the summer of 2017 to resolve this quest.

During the first week of the summer internship we (the interns) were introduced to FINN.no, its systems and our coaches for the summer project; Martin, the brilliant programmer and Ingrid, the amazing designer. In order to ensure everyone on the team were on the same page a brainstorming session was held. During this session possible solutions for the summer project were developed, and our mutual ambition for this project was put into words. Our ambition has been to: “Increase the amount of ads receiving feedback/being sold on FINN.no” and “Help advertisers on FINN.no make better ads”.

In the past 2 months two concepts have been realized, with quite surprising results and developments. But you have to read the rest of the document to know exactly what surprising developments took place :) Sorry, that’s just life bro/hon!

Project 1: Tilbakemelding (Feedback)

Have you ever created an ad and received no feedback, no likes and no messages? This is the reality of approximately 50 % of all ads published on FINN Torget in Norway. Imagine if there was a way for advertisers to know exactly how they could improve their ads, what would it look like? “Tilbakemelding” is one possibility. Tilbakemelding is a button located at the front page of an ad for all potential buyers to see and use. If a user does not like an ad, and feels that sending a message is too much of a hassle, they can push this button. By doing so, a set of possible improvements to the ad will be revealed, and can easily be selected. Compared to sending a message this is less time consuming and the feedback can be used to create statistics to help the advertiser improve their ads. In this project, the focus was first and foremost to create the button and observe how users of FINN.no would experience the new functionality.

After implementing the solution and launching it on FINN.no website, it was determined the new functionality was used by barely any users. Some tears were shed, and a new direction was chosen. It was decided to rename the button, and make it into a notification button. By using the new button users can choose specific tags and be notified when this tag has been edited in the ad. It turns out this option was used more, however not enough to deem this a valuable asset to FINN.no’s website.

Tilbakemelding (feedback) accordion

Technical solution

There were several steps to go through before we could launch our solution. We were lucky to get the opportunity to develop it the whole way from backend to frontend. As we have understood, this is a common practice in FINN.no. This so that the developer can feel a greater ownership to his or her code, and can follow it the whole way through the process. Our solution took the form of a separate API running inside kubernetes in FINN.no’s tested and proven microservice architecture. PostgreSQL was chosen as the underlying storage, due to both simplicity and familiarity. Docker and Docker Compose were utilized to ensure that dependencies and environments were identical throughout development, staging, production, and to increase the overall speed of development.

Each tag that’s chosen by a user was stored in the database connected to different types of data like the Id’s of the seller, ad and the person clicking on the tags. We used Java, Spring Boot and IntelliJ IDEA to make our API. To represent our solution we used JavaScript, HTML and CSS. Thankfully FINN.no has its own implementation guide that helps both designers and developers to get an understanding of how different solutions should look like, and much of it has already been implemented so it saves developers a lot of time that they can use on other things more fun than CSS.

Results UX Design

Several prototypes were designed using Sketch and Invision, with people inhouse and on the streets helping us evaluate the design. The tests helped us rename several of the tags, helped us deciding the looks of the button and the tags, and the means by which a user should access them. In addition, more elaborate user tests in the prototyping lab of FINN.no were successfully completed, and valuable insights were obtained. Although the user surveys proved a majority of FINN.no users, both buyers and advertisers, wanted a system to improve ads, the user tests proved contrary. It was revealed buyers were not interested in helping improve ads. If they wanted information about the product, they would message the advertiser directly, thus limiting the competition for the ad and receiving a response quickly. Most users did not notice the button at first glance, partly due to the location, but also because they did not feel a need to use it.

Activated drop-down menu for Tilbakemelding (feedback), displaying tags with different ways to improve the ad.

Recommendations

During the first weeks of the project, several surveys regarding aspects of “Tilbakemelding” were given to a set of FINN users. Based on these surveys, we know advertisers want advice in how to improve their ads and buyers on FINN.no are not interested in ads if they are too poorly made. Making users help advertisers by giving feedback has proven to be difficult though. Another way of giving advice to advertisers would be to make an automated system capable of doing so. Using an automated system rather than user feedback would also minimize the possibility of buyers misusing or abusing the feedback module (giving inaccurate/wrong feedback). This system could analyse the amount of pictures, the quality of the pictures, the amount of text, the price etc. in terms of similar ads and compare them to the ad in question. The tags used in “Tilbakemelding” could be used as a starting point, with the most focus on number of pictures and picture quality. According to our results, users would be first and foremost interested in the pictures of the ad. Another recommendation based on our experience is to start user testing early on. By doing so, weaknesses in the prototype can be found before a lot of time is spent programming.

Project 2: Redesign ad dashboard

After finishing “Tilbakemelding”, it was decided to do a redesign of the ad dashboard page in order to improve advertisers’ user experience. As can be seen on the picture, several of the buttons have been rearranged, and the statistics have been given more space and importance. The old message system has been upgraded to improve the flow and aesthetics. Tips into how your ad can be improved have been added to motivate and help advertisers make even better ads. Lastly, a timeline showing the past, present and future status of the ad are displayed.

Technical solution

The redesigned ad dashboard page has been created using Node.js and React. By splitting FINN.no into multiple smaller microservices it allows developers to create/modify pages, while still keeping development environments rich and easy to use. We created a Node.js server for handling server rendering and proxying requests to other backend services and a React web application for the frontend. We still utilize existing backend services at FINN, e.g for getting the correct buttons for an ad in a specific state. This allows our newly developed front end to be switched out seamlessly without the user knowing the switched from one application to another. FINN.no has quite a lot of backend services that we can take advantage of to speed up the process. One example of this is the use of Finnlets which creates the identical footer and header while the server is rendering the page to the user.

Results UX design

New prototypes in Sketch and Invision were made. According the user tests in FINN.no’s lab, most people understood the new system, and felt it was fitting nicely into FINN.no current website. Several users did not notice the changes. Almost all users understood the ad timeline, the statistics graph, the new message system and how they could use the given information to improve their ad. One user mentioned he felt compelled to change his ad after inspecting the timeline, and noticing his ad would soon expire. In most cases it took some time before the advice for improving the ad was discovered. However, all users said they wanted to follow the advices after spotting them. In the user test, the most remarks concerned existing buttons and text on the current webpage. These remarks were mostly concerned with the “Skjul annonse i søkeresultater” (Hide ad in search results) button and the “Vanlig trafikk” (regular traffic) statistics, which were both found to be confusing. A majority of the users did not want to purchase “Synlighet” (visibility) because it felt like a waste of money, and told us they had never used this function.

The statistics module in the redesigned ad dashboard page

The message module in the redesigned ad dashboard page

Recommendations

As mentioned, the advice for improving an ad was not noticed by most users, even though they were perceived as helpful and motivating when spotted. By making the advices more noticeable (bigger, more visible colors, more central location) when an ad is either about to expire or has a very low number of views, the people who are in the most need of this information would be more easily exposed to this information. According to our user test results, people who noticed the advices would most likely follow them. Lastly we would recommend “Vanlig trafikk”, “Kjøp synlighet” and “Skjul annonse i søkeresultater” to be renamed. “Vanlig trafikk” could be called “Totalt antall visninger” and “Kjøp synlighet” could be called “Øk synlighet”. A new user test to establish new names should be performed, as well as an user test into making purchasing “Synlighet” more attractive and understandable.

Final thoughts

Working with FINN.no this summer has been very educational and challenging! Although there has been a lot of work, FINN.no made sure we always felt at home in their quarters (it’s actually an old prison for females, who would have known?). We have met and learned from a wide range of professionals this summer, used high-quality software to solve our problems and learned new methods for dealing with challenges. In addition, it turns out working at FINN.no is also amazingly fun! We have been treated to ice cream, pizza and other delicious food (some of us probably have to hit the gym soon), and lots of social activities such as “Fangene på fortet” escape rooms and FINN.no summer party. We, the summer interns, feel very fortunate to have been the chosen ones this year, and we know next year’s summer interns will feel the same!

The interns and our fantastic coaches

From Hadoop and Cassandra to Kafka Streams

2017-07-31T12:00:00+00:00

Some context

People who publish their classified ads on FINN.no get access to various statistics to see how their ads are performing. For a user it can look something like this:

Statistics the owner of a realestate ad gets to see

The top left bar chart shows the repartition of the incoming traffic by day and the legend to the right of it shows the total numbers for each type of incoming traffic. The lower section is divided in 3 parts:

the left part shows the number of views by unique users;
the one to the center shows how many users have been notified of the ad by email and how many added the ad to their favorites;
the right part shows how many viewers out of the total come from a specific type of traffic.

This gives the user basic insight into the reach of their ad, such as how many views it has and how many unique users have viewed it.

Around November 2016 there was a request to show more detailed information about the viewers, such as the age, gender and location distribution of the viewers (demographic information) so the owner could potentially change an ad to better fit the audience they wanted.

Existing solution

As users view ads, do actions (such as send a message to an ad’s owner or scroll down and read the whole page, for example), these actions are gathered and published internally on Apache Kafka. These streams of data then become the basis for computing the statistics above. The plan was then to also publish demographic data about the viewers (such as location, age and gender) on Kafka and join the user actions with this demographic data to provide enhanced statistics.

At the time, the action events published on Kafka were saved to Cassandra Apache Cassandra clusters, and statistics were being computed as batches on an aging Apache Hadoop cluster reading from Cassandra. Both our Hadoop and Cassandra clusters had not receive much love recently and were all on end-of-life versions. The old system also had an increasing tendency to fail, so we were also in the process to decide to either spend our time to refresh/renew the old system or replace it.

A simplified overview of the previous architecture

Replacement solution

We first tried to find a good way of joinining the latest demographic data being received through Kafka as part of the existing statistics computation done with Hadoop. We quickly learned that efficient communication with Kafka from a Hadoop job was not easy to get right. After some experimentation we had a solution that sort of worked, but it was complicated and far from elegant.

After doing some research we started to look at the possibility of using Kafka Streams. Reading the introductory blog post - especially the “Simple is Beautiful” paragraph - it felt like Streams was just what we needed.

The most interesting aspects were:

no more external runtime for jobs (in our case YARN/Hadoop);
no more intermediary external storage (Cassandra);
built-in fault tolerance;
only POJOs running as normal Java apps;
limiting the number of components by using Kafka for the whole solution.

This would lead to less moving parts, less maintenance and what we saw as a simpler and more elegant solution.

We also evaluated other solutions, amongst which:

upgrade our Hadoop and Cassandra clusters;
Apache Spark micro-batches on top of an Apache Parquet storage filled from Kafka;
hybrid solutions with hyperloglog and other similar cardinality tricks to estimate the unique users visiting an ad.

We were already using Kafka (and it is seen as a central part of our future architecture), so Streams was deemed to be the most attractive and straight-forward solution.

Streams in use

The first prototype took less than a week to get up and running, and from there it took 6 months to develop a fully operational, properly sized version running in production on our Kubernetes container cluster.

As part of development we also wrote tools for monitoring, error handling/diagnostics, reprocessing and for migration of the existing statistics out of the old Cassandra storage and into our new storage. During this development, the original plan to join in demographics data was put on hold (for non-technical reasons) and the project turned into a more technical one: move the statistics system out of the aging Hadoop and Cassandra clusters and migrate to something that incurs less maintenance and fits better with our overall technology strategy.

Requirements

The amount of data is is reasonably large, about 100 million messages a day, and the main requirements were:

keep the near-realtime computing of the statistics we had in the previous solution;
be able to reprocess data up to 3 months back in time (in case of failure or bugs);
be able to integrate late arrivals to some extent.

Architecture

We went through multiple versions and the components evolved both in size and responsibility during development, and below is how it looks now.

A simplified overview of the replacement architecture

The replicator copies a subset of short lived topics (3 days retention) into our own long lived topics (90 days retention) to support reprocessing of the last 3 months. It is a simple read-write (Kafka consumer and producer) that computes the hash of the message as the key for deduplication downstream.

The aggregator is the interesting bit that actually computes the base data for our statistics: the counts. A simplified versions of the Streams code to achieve this would look something like this:

// grouping by unique key which consists
// of the id of the ad, the day the action
// happened and the type of the action
// and then counting

kstream.groupBy((key, action) ->   // build the new key by which we want to count
            new Key(action.ad,     // ex: ad number 12345678
                    action.day,    // ex: day 17-02-2017
                    action.type))  // ex: "click"
       .count("count-store")       // counts the number of messages for each key
       .to("count-topic");         // outputs each update of the count the Kafka topic

The production code does more than that with Apache Avro serialization / deserialization, filtering of bad data, deduplicating, reporting metrics etc., but the above code shows the main logic the component consists of.

Finally, the sink is just an instance of Kafka Connect configured to use the JDBC sink - with no custom code. It reads from the output topic from the aggregator and writes into a PostgreSQL database.

All of these components are webapps running in Kubernetes and can be scaled up or down easily. And while the replicator and the aggregator are Spring Boot based webapps, the sink is just an off-the-shelf Kafka-REST webapp.

Data quality

Avro and Schema Registry

We use Avro and the Schema Registry to enforce schemas on the Kafka messages so that changes to the message structure maintain compatibility over time. This proves to be quite robust any incompatible change to the schema gets immediately rejected.

// build the serialization/deserialization object

GenericAvroSerde avroSerde = 
    new GenericAvroSerde(registryClient, registryProps);

// build the KStream with Avro

new KStreamBuilder()
    .stream(Serdes.String(), avroSerde, topics);

A note about developing with Avro and the schema registry: While in the early stages of development, using the schema registry proved cumbersome: breaking changes happens quite often during development and maintaining compatibility or overriding it takes time to get used to. This was however very good training for applying actual real changes to the schema that will inevitably be required later on when running in production. Special care is required with the internal Streams topics that also contain Avro messages.

Streams filtering

We filter out bad messages (missing or corrupted pieces of information) with the filtering method of the Streams and plain old Java.

// filter out actions for the owner of the ad
// and actions that are missing some data

kstream.filter((key, action) -> 
                   action.isNotFromOwner() 
                   && action.hasRequiredData());

Dealing with duplicates

There is no built-in support in Streams to deal with duplicates, so non-idempotent processing of the messages then becomes an issue. The strategy we adopted is to calculate a hash of each message value and set it as its key so that consumers downstream can deduplicate based on an in-memory LRU cache of these hashes (because all the identical keys are guaranteed to be in the same partition, and thus end up in the same consumer on the same thread).

In the replicator:

// compute a hash of the message and
// set it as the key

String key = DigestUtils.sha1Hex(message);
producer.send(new ProducerRecord<>(topic, key, message);

In the aggregator:

// cache processed key for 10 minutes

KEYS = CacheBuilder.newBuilder()
                   .maximumSize(100000L)
                   .expireAfterWrite(10L, TimeUnit.MINUTES)
                   .build();

// keep only messages whose key has not been processed

kstream.filterNot((key, action) -> {
    boolean duplicate = KEYS.getIfPresent(key) != null;
    KEYS.put(key);
    return duplicate;
});

This is a best effort strategy since if the Streams app dies, so does its cache, and the new app instance won’t be able to deduplicate the very first duplicate (but this should happen so rarely that it is acceptable).

Limiting the disk usage

Streams and Kafka rely on offsets when reading messages which guarantees that when things start again they pick up where they stopped. On top of that, when using aggregations, Streams stores a changelog of the operations in Kafka itself, meaning that also stateful operations (grouping, reducing, counting, etc.) will recover gracefully and efficiently from a stop or crash by starting from where they had stopped. This is really nice fault-tolerance feature.

The changelog topic is used on every start to build a “local state” for stateful operations on the machine where Streams is running. RocksDB manipulates that state internally. This local state resides partly in memory and partly on disk, all depending on the amount of data in question. Instances running on our Kubernetes cluster are stateless, so they do not keep data on disk between restarts. This means we lose the local state each time the app is shutdown. However, when the streams application starts up again it will rebuild this local state from the Streams internal changelog topic, so no data is lost. As a consequence, there are two things to consider:

rebuilding of the local state on (re)start can take several minutes depending on the volume of data and how it is partitioned
if the app runs for a long time the disk usage will keep on increasing, so we need to use TimeWindows to avoid the issue of never-ending growing local state

Below is an example of how to limit data on disk by using TimeWindows.

// count by buckets of 1 day
// and keep these buckets for 1 day in the local state

.groupByKey()
.count(TimeWindows.of(TimeUnit.DAYS.toMillis(1))
                  .until(TimeUnit.DAYS.toMillis(1)),
       "count-store");

Migrating existing data

The Cassandra table storing our data was not structured in a way that made it easy and fast to export the existing data. Since we were running on an older version of Cassandra, we used the Astyanax library to migrate the existing data over to PostgreSQL. It allowed partitioning of the ring buffer and reading in parallel from each of these partitions for higher throughput. The included checkpointing mechaninsm helped us start where it left off each time the migration would fail. Astyanax did a good job reading and writeíng the billions of rows of data needed within an acceptable migration time window without overloading the running Cassandra clusters.

The above proved to be quite tricky due to tombstone and compaction issues. Tombstones would often lead to queries timing out, so we first had to fix that. These problems were not, for the most part, visible in day to day use, so it was a surprise challenge when migrating data (which took quite a lot of time to do).

Integration testing

We use JUnit and spin up an embedded Kafka and Apache Zookeeper. Then we use producers and consumers to send and receive messages for the input and output topics that the Streams expects. Extra care should be taken to clear the state of the Streams between each test. Integration testing has proven to be quite challenging because there are a lot of details to take into account.

For manual testing when quickly prototyping something new, we both make use of our Kubernetes development environment, or a local Docker Compose configuration, and a bunch of scripts using the Kafka tools.

Reprocessing

When rolling out bug fixes or improvements, it may be necessary to reprocess the existing data. Reprocessing with Streams is sort of an artform - especially if you have non-idempotent processing. Sometimes it is possible to reset the app (with kafka-streams-application-reset) and just restart with auto.offset.reset to earliest to reprocess everything. This clears all of the app’s state, and depending on the case this can be acceptable or not. Our aggregator can start in a special reprocessing mode that spools all the messages of our 90-days long lived topic and only processes a given time interval.

Error diagnostic

Figuring out when the Streams output data is wrong is not always as easy as seeing a sudden spike on a monitoring graph or incoherent numbers. Figuring out why the numbers do not add up can be even more challenging in that there is no way to query the input Kafka topic to investigate. The same could be said of our Cassandra “input table” to Hadoop which was not structured after a diagnostic query need either. Putting a Kafka Connect Source from our input topic to a form of storage that allows easy querying might enable easier diagnostics.

However, we’re happy to say that we never experienced any bugs in Streams itself and the Kafka cluster have proven to be very reliable. The problems we have encountered have mostly been due to corrupted or duplicated input data.

Summary

The previous solution required a lot of knowledge on top of knowing Kafka, mainly Cassandra (dealing with tombstone problems, compaction strategies, cluster management, old composite columns vs CQL etc.) and Hadoop (maintaining HDFS, operations, batch-writing etc.). Using Streams instead, there is no extra required knowledge. The only thing to know about is Kafka (and the Streams framework).

We also end up with a less heterogeneous infrastructure to run with fewer clusters and fewer products. It ends up being just one Kafka cluster and a bunch of webapps. Which we have quite a lot of experience of maintaining over time.

With Kafka being more and more widely used at FINN (as well as Streams) the knowledge can also be shared between all and promotes cooperation across the organization.

Flappy Cat

2017-07-19T16:58:36+00:00

Why make a Flappy Cat game?

The Idea was to make a Flappy Bird with motion tracking control for our JavaZone stand. Some quick research told us it should be possible. Since cats are assosiated with FINN, it of course became a Flappy Cat game. Come and try it at JavaZone in September.

Made by search team.

How we made it

Step 1: Make the game. As neither of us are game developers, we found this flappy bird tutorial online that we decided to follow. This tutorial uses Phaser, an open source JavaScript/HTML5 game development framework. It was easy to follow and it explained all the steps in detail, so even the most back-end heavy developer in our team had no problem writing his first JavaScript game.

Step 2: Buy the motion tracker. We chose Kinect as we found videos on YouTube of people doing exactly what we wanted to do with the Kinect sensor. We only had one day to do this project, so this was not the time to reinvent the wheel.

Step 3: Tweaking one of the example apps that was included in the Kinect 2.0 SDK. We found an app that was made to do something when it detected a gesture. This was perfect for our purpose. We recorded our desired gesture, “the flap”, in Kinect Studio and used Visual Gesture Builder (also included in the SDK) to build a database of our gesture. We then used this gesture database in the example app and suddenly we could recognize a flap! Now, all we needed was for something to happen when a flap was registered.

Step 4: Creating a simple Node.js server that could not only serve our game, but also have an API endpoint and give the Windows app a way to notify the game that it detected a flap. This is a simple solution; a GET call to /flap is equivalent to pressing space in the game.

WAI-ARIA and its true impact on assistive technologies

2017-06-07T12:00:00+00:00

1. Introduction

This article is intended for web developers who want to improve their understanding of the effect that WAI-ARIA has on assistive technologies. It will therefore be assumed that you already have basic knowledge about terms such as ARIA and screen readers.

1.1. Generalize screen readers

From the perspective of web development, it serves no purpose to delve into technical details about specific brands of screen reading software. There are several reasons for this:

We can’t programmatically detect which assistive technologies our users are using. There are several good reasons for this, including privacy concerns pertaining to confidential information regarding personal health and diagnostics.
Universal design means design for everyone, regardless of which type of assistive technology is being used.

Therefore we won’t talk about specific brands of screen reading software in this article. We will focus on best practices with universal design in mind.

1.2. The main issues with ARIA

Firstly, support for ARIA in assistive technologies is still partial at best. A consequence of this is that only a handful of ARIA features are reliable across different implementations, and that is if we don’t count older assistive technologies that have no support for ARIA at all. Therefore it becomes important to differentiate between ARIA features that we can reliably use for correcting accessibility issues versus ARIA features that we should only use for improving UX. I.e. to a certain degree we can accept that UX will differ depending on which screen reader is being used, but keeping parts of our applications inaccessible to everyone except those who are fortunate enough to use the right screen reader is unacceptable. Naturally this is something that will slowly but surely sort itself out as makers of assistive technologies continue to work on their ARIA implementations, but for the time being we ought to remain mindful of this.

The second issue related to ARIA is that many developers who technically know how to use it aren’t aware of the actual effect it has on assistive technologies. This is probably the biggest root of widespread incorrect use of ARIA on the web today, which is a growing source of frustration for users of assistive technologies. This is definitely an issue that web developers can and should do something about, and a good step in the right direction is to read the rest of this article.

2. Screen readers and virtual user interfaces

Many screen readers create an invisible virtual user interface for presenting web content. This concept typically involves a process where the screen reader reads the entire web document by accessing its DOM, then it uses that information to construct a virtual representation of the document that the user can browse through. The screen reader will route most of the keyboard input to functions in the virtual user interface where it will be used for navigational purposes. Different screen readers have different quick keys, but typical examples are H for jumping to next heading, Shift+H for jumping to the previous heading, digits 1 through 6 for jumping between headings at levels 1 through 6 respectively, F for form elements, T for tables, and so on. These virtual user interfaces provide the user with a consistent and familiar interface that is highly optimized for quick and convenient keyboard navigation. On the flip side, the web application won’t receive keyboard input from the user while this virtual browsing mode is engaged, which can be a real issue in situations where the application requires keyboard input for optimal interaction.

2.1. Native interactive widgets

The most obvious example of an interactive widget that requires keyboard input from the user is an edit field (HTML textarea and certain input types). When the user starts interacting with such an element, their screen reader will automatically disengage its virtual user interface and pass keyboard input on to the application. At this point the user may freely take advantage of all keyboard features offered by the application and the web browser, such as typing in the edit field, move the cursor around using Arrow keys, etc. No ARIA magic is needed to make this work, since both input and textarea elements are native HTML widgets that web browsers and screen readers know how to handle correctly.

The select element is another example of a native HTML widget that web browsers provide keyboard navigation for. The user can use arrow keys for navigation between the options, page up and page down for quickly scrolling through the options, home and end for beginning and end of the dropdown, and alpha-numeric keys can be used to efficiently navigate to the options that match the input from the user. Again, screen readers will therefore automatically turn off the virtual user interface when the user starts interacting with this type of widget, and all this works right out-of-the-box because the select element is a native HTML widget.

2.2. Non-native interactive widgets

The same opportunity for automatic toggling of virtual user interfaces should also be applied to non-native interactive widgets that are expected to handle keyboard navigation. Examples include menus, list boxes, tree views, grids, and even modal dialog boxes. Why should all of these widgets process keyboard input? Firstly because some users rely on a keyboard for interaction. Not everyone can use a pointing device (such as a mouse). Secondly, some users want it, because keyboard navigation can be fast, efficient, and convenient for interaction with these types of widgets. Implementing keyboard navigation for custom interactive widget can improve UX for a surprisingly wide range of users.

Alpha-numeric keys for quick selection of items. For instance, if focus is on one of the grouped buttons in a dialog box, the user should be able to press e.g. the Y key to select the Yes button. And item matching should be implemented for list boxes, tree views, menus, radio buttons, and pretty much any widget that contains grouped text items that are available for selection.
Arrow keys. Up and Down for navigating between items. Left and Right for expanding and collapsing sub-menus and branches in treeviews. Up/Down for values and Left/Right to switch between year, month, and date in a date picker widget. Proper spatial navigation for grids. And users should be able to use arrow keys to move focus between the dismiss option in a dialog box (Yes, No, Cancel, etc.)
Home/End keys for moving focus to the first or the last item in a group of items.
Page Up/Page Down should be implemented as the equivalent of X amount of Arrow Up/Arrow Down key presses for faster browsing through the items. X could either be a fixed number, or you could make a formula to calculate X based on the size of the group.

We need to let screen readers know that we have taken the time to implement proper keyboard navigation for our non-native interactive widgets, otherwise screen readers will inadvertently hide this functionality behind their virtual user interfaces. It is true that these virtual user interfaces are effective for overall page navigation, but custom keyboard navigation is often even more effective when interacting with specific widgets. Below you’ll find three methods of using ARIA to help screen readers with automatic toggling of virtual user interfaces for non-native interactive widgets that support keyboard navigation.

2.4.1. Method 1: Defining roles

The WAI-ARIA specification lists many possible roles for HTML elements, and modern screen readers will handle presentation and interaction of elements according to the designated role. Screen readers will typically turn off the virtual user interface when the user starts interacting with elements that have any of the following roles: combobox, grid, listbox, menu, menubar, radiogroup, scrollbar, slider, spinbutton, tablist, textbox, tree, and treegrid. IMPORTANT! You MUST NOT set any of these roles unless the widget has a functional mechanism for keyboard navigation implemented, otherwise the user will be much better off with navigation features provided by the screen reader. These ARIA roles are not just meant to be used for announcing the presence of something that visually resembles a certain widget. Their purpose is to prepare assistive technologies to accommodate the type of interaction that users associate with the type of widget in question, and naturally this requires that the widget itself keeps the promise of providing methods for keyboard navigation. E.g. a tree view must not just look like a tree view, it must also behave like one. Just like a doctor must not just look like a doctor. In short, we have certain expectations about entities that have specific roles.

ARIA roles might also affect how screen readers will present the content of a widget to the user. A tree view for instance might be represented in a virtual user interface as a single starting point for interaction, effectively hiding the content of the tree so that it won’t clutter the virtual user interface with a lot of data. The user could then choose to interact with the tree if they want to browse through its items, but if keyboard navigation has not been implemented then the tree will essentially end up being totally inaccessible to the user since they can’t access the content of the tree in the virtual user interface either. So, let me emphasize that you MUST NOT set any of the ARIA roles for interactive widgets listed above unless you have also implemented a mechanism for keyboard navigation.

2.4.2. Method 2: Using the aria-haspopup property

This method can be used in conjunction with ARIA roles to aid screen readers with automatic deactivation of virtual user interfaces when an interactive widget pops up (like an overlay) in response to an action from the user. E.g. the user clicks on a button, and the expected result is that a menu pops up. Obviously the most convenient solution here from the user’s perspective would be to make the screen reader automatically initiate interaction with the popup widget, since the user has already implied that they would like to interact with the widget by clicking on the button that produced it.

By setting aria-haspopup on an element, we promise two things:

The application takes responsibility of moving focus to the starting point / default selectable element of the pop-up widget when the user starts interacting with the element that has aria-haspopup (e.g. by clicking on it).
The pop-up widget is accessible to users of assistive technology (i.e. keyboard navigation has been properly implemented for the pop-up widget).

Screen readers may then automatically deactivate the virtual user interface to facilitate optimal usability with the widget. And again, this means that the pop-up widget MUST be accessible with a keyboard in the first place. Think of the aria-haspopup property as a promise of implemented accessibility features. It does not make the pop-up widget more accessible by itself.

A few more things to keep in mind regarding aria-haspopup:

The mechanism for activating the element that has aria-haspopup MUST be accessible for everybody. This means that you can’t just rely on e.g. a mouse-over event to trigger your pop-up menus, since many users of screen readers can’t successfully use a mouse even if they had one. Use click or focus events instead/in addition.
The element that has aria-haspopup must have an interactive role (either implied by HTML semantics or set with the role attribute), otherwise screen readers might simply choose to ignore the aria-haspopup property altogether. The exception here is the link role (or the native a element), which might cause the screen reader to retain its virtual user interface upon activation in anticipation of browser navigation. Remember that you can always override implied semantics with the role attribute, and the button role is usually a good choice for a clickable element that triggers a pop-up widget in the application.
The pop-up widget must also have correct semantics. I.e. give the widget an ARIA role that semantically corresponds to the type of widget that is specified in the aria-haspopup property. ARIA 1.1 defines the following possible values for aria-haspopup: dialog, grid, listbox, menu, and tree. E.g. use a value of “dialog” even if the pop-up widget has its role set to “alertdialog”, not just if its role is “dialog”. Also, for backward compatibility with ARIA 1.0, a value of “true” for the aria-haspopup property is equivalent to “menu”. Therefore, if your pop-up widget is a menu, then you might get wider coverage by using “true” instead of “menu”.

2.4.3. Method 3: The application role

Setting the application role on an element tells screen readers that the content of the element is not intended for virtual user interfaces. The application claims responsibility of providing adequate keyboard navigation features for all interactive widgets that are part of an element that has the application role. IMPORTANT! You should think very carefully before using this method. Many users prefer to use the screen reader’s virtual user interface for overall navigation since it provides a consistent and familiar way of browsing web applications. Setting the application role (e.g. on the body element) is like blocking the highway and forcing drivers to use an alternative route where directions might be scarce. Unless you have a very specific reason to do so, you should refrain from doing it.

3. Defining relations between HTML elements

ARIA provides three different properties for explicitly defining relations between HTML elements in cases where the DOM hierarchy does not already do this implicitly (by nesting and ordering elements in a sequence that makes sense from an end user’s perspective).

3.1. The aria-owns property

The aria-owns property is used to rearrange elements in the accessibility tree (which is a version of the DOM tree that has been tailored to accommodate assistive technologies). Of the three properties for specifying relations between elements, aria-owns is preferred for the following reasons:

Aria-owns is the most supported property for specifying relations between elements.
No special interaction is required for aria-owns to take effect. The user won’t need any additional knowledge to benefit from this feature.

As an example of a scenario where aria-owns would be useful, let’s say we have a list where each item consists of a checkbox, an image, a heading, a paragraph, and a link. We might prefer to keep these elements in that particular order in the DOM for visual styling purposes. For someone who uses a screen reader however, this order could easily end up being confusing since the heading is not the first element. The heading is the element that introduces a new context to the user. Therefore the user wouldn’t know the context of the checkbox the first time they spot it. The second issue with this order is that the user might believe that the checkbox belongs to the previous list item since it’s technically under the heading of the previous list item. To fix this, we could create an empty div element at the top of each list item, and set aria-owns to refer to the id of the heading. This would alter the accessibility tree so that the heading would come before the other elements.

3.2. The aria-flowto property

Instead of rearranging the DOM hierarchy for assistive technologies (like aria-owns does), the aria-flowto property tells assistive technologies to present the user with an option to jump directly to the related element(s). Unfortunately this property is not widely supported by assistive technologies. Another issue with aria-flowto is that users might not know how to trigger the jumps, even if their assistive technologies support this feature.

3.3. The aria-controls property

The aria-controls property is defined by the WAI-ARIA specification as a way of indicating which element(s) are being controlled by the element that has aria-controls set. In practice it provides a jump point (similar to aria-flowto), but it also indicates that interaction with the element that has aria-controls might have an effect on the controlled element(s). Consequently this means that aria-controls must only be added to interactive elements. Unfortunately, just like with aria-flowto, few assistive technologies support this property, and it also requires that the user knows how to initiate the jumps to take advantage of this.

Interestingly enough, numerous web sites seem to be richly seasoned with aria-controls, and the golden rule about using ARIA to define relationships only when it can not be inferred from the DOM hierarchy is often forgotten. In cases where this feature is needed, aria-owns would be a better choice since it is more widely supported. In cases where this feature is superfluous, it just adds extra noise for those who use assistive technologies that support it. E.g. it makes no sense to create a jumping point from a button to a panel if the button is immediately followed by the panel in the DOM hierarchy. The user would spend more time and effort initiating the jump with their screen reader than it would be to just continue with the normal and familiar flow of navigation that would yield the same result.

4. External resources

Unleash Your Features Gradually

2017-03-10T16:58:36+00:00

FINN.no is the largest online marketplace in Norway, and we take continuous deployment seriously. We are about one hundred developers deploying new code to production 978 times each week. That is 978 / 100 = 9.78 deployments to production per developer every week. In order to get to these numbers and still keep it under control, we have had to both change our development process and establish the right set of tools. In this article we introduce feature toggles and how this technique has contributed to our continuous deployment success.

We will also present Unleash, an open source framework we created to support feature toggles at an enterprise scale. Unleash allows us to enable features for specific users via the activation strategies concept. Unleash is already set up and in use by multiple teams within Schibsted, including FINN, SPT Payment, and Knocker.

Unleash is now also available as a Software-as-a-Service offering, making it much easier for anyone to adopt. Check it out at unleash-hosted.com

Number of production deploys in “week 50”

Feature Toggles

‘Feature toggles’ is a simple technique to separate the process of putting new code into production from the process of releasing new features to our users. In its simplest form a feature toggle is just a “if” statement in the code, guarding the new feature:

if (unleash.isEnabled("AwesomeFeature")) {
  //magic new feature code
} else {
  //old boring stuff
}

The fastest way to get started with feature toggles in your project is to use a plain old properties file. This allows you to integrate unfinished features to the master branch and hide them from users in production. A simple plain properties files was exactly how we started playing with the features toggles in FINN.

We realised that feature toggles enabled us to move faster, because we were able to integrate new and unfinished features to the master branch early. This also avoids long-running feature branches that tend to be harder to merge. We learned the hard way that having many parallel feature branches adds a lot of extra complexity related to merging, but with the help of feature toggles we can merge them directly to master.

This went well, but we also felt we could do better. We wanted one simple dashboard to turn the feature toggles on and off. The dashboard should contain all feature toggles for all web apps and applications in FINN. After creating several proof-of-concepts using various technologies, we proposed a solution to our technology management group. They loved the idea and we created Unleash, our shiny and open source feature toggle system.

Unleash gives us a great overview of all feature toggles across all our services and applications. It also makes it super easy to activate new features in real time in production and of course instantaneous deactivation of the feature if there’s a problem.

System Overview

Unleash is comprised of three parts:

Unleash API - The service holding all feature toggles and their configurations. Configurations declare which activation strategies to use and which parameters they should get.
Unleash UI - The dashboard used to manage feature toggles, define new strategies, look at metrics, etc.
Unleash SDK - Used by clients to check if a feature is enabled or disabled. The SDK also collects metrics and sends them to the Unleash API. Activation Strategies are also implemented in the SDK. Unleash currently provides official SDKs for Java and Node.js

Unleash system overview

Unleash is written with performance and resilience in mind.

Performance - In order to be super fast, the client SDK caches all feature toggles and their current configuration in memory. The activation strategies are also implemented in the SDK. This makes it really fast to check if a toggle is on or off because it is just a simple function operating on local state, without the need to poll data from the database.
Resilience - If the unleash API becomes unavailable for a short amount of time, the cache in SDK will minimise the effect. The client will not be able get updates, when the API is unavailable, but the SDK will keep running with the last known state.
Built in backup - The SDK also persists the latest known state to a local file at the instance where the client is running. It will persist a local copy every time the client detects changes from the API. Having a local backup of the latest known state minimises the consequence of clients not being able to to talk to Unleash API at startup. This is required because network is unreliable.

Gradual Roll Out of Features

It is powerful to be able to turn a feature on and off instantaneously without redeploying the whole application. The next level of control comes when you are able to enable a feature for a specific user or a small subset of the users. We achieve this level of control with the help of activation strategies. The simplest strategy is the “default” strategy, which basically means that the feature should be enabled for everyone.

Another common strategy is to enable a new feature only for specific users. Typically I want to enable a new feature only for myself in production, before I enable it for everyone else. To achieve this we can use the “UserWithIdStrategy”. This strategy allows you to specify a list of specific user ids that you want to activate the new feature for. (A user id may of course be an email if that is more appropriate in your system.)

When I have verified that the new feature works as expected in production, I can activate the feature for some real users. To achieve this I can use one of the gradual rollout strategies. At FINN we use three variants of gradual rollout, which have slightly different purposes:

GradualRolloutUserId is the strategy we use when we want to gradually rollout a new feature to our logged-in users. This strategy guarantees that the same user gets the same experience every time, across devices. It also guarantees that a user from the first 10% will also be part of the first 20% of the users. Thus we ensure that users get the same experience. Even if we gradually increase the number of users who are exposed to a particular feature. To achieve this we hash the user id and normalise the hash value to a number between 1 and 100 with a simple modulo operator.

Converting a user id string to a number between 1 and 100

GradualRolloutSessionId is almost identical to the previous strategy, with the exception that it works on session ids. This makes it possible to target all users (not just logged in users), guaranteeing that a user will get the same experience within a session.

GradualRolloutRandom has no form of user stickiness, it just picks a random number for each “isEnabled” call. We have found this rollout strategy to be very useful in some scenarios when we are enabling a feature which is not visible to the end user. Because of its randomness it becomes a great way to sample metrics and error reports and it can be a useful way to detect JavaScript errors in the browser.

In Unleash 2.0 we added support for multiple strategies. This is especially convenient when you both want to expose your new shiny feature to a percentage of the users and at the same time make sure to expose it to yourself.

Metrics

When you enable a feature for a small group of users it is nice to know how many users actually get exposed to it. This is why we added metrics to Unleash 2.0. The SDK will locally collect metrics in the background and regularly share this information with the Unleash API. The API then aggregates the metrics from all clients and calculates how many times a feature toggle evaluated to enabled in a given time period.

When implementing the metrics feature we discovered that this was a nice way to display which toggles are actually in use in an application. It even made it possible to show feature toggles being used in the client code which are not defined in the Unleash API. Showing undefined toggles used in client applications makes it easier to discover feature toggles not defined in Unleash UI. This can happen when a developer starts using a new feature toggle, but has not defined it in Unleash UI yet. The undefined feature toggle is also a clickable link to make it easy to define it in Unleash.

Types of Toggles

We have seen how release toggles allow us to decouple deployment of code from the release of new features. At FINN we have ended up with three main categories of feature toggles:

Release toggles are the most common type of feature toggles we use. These are used for new features we roll out in the market and are mainly used to roll them out safely: instead of exposing a new feature to everyone at once we can try it out on a small subset of our users and verify that it scales, does not contain bugs, and that it moves our KPIs in the right direction.

Kill switches are sometimes necessary to protect our site. Like many other large sites we occasionally have some features that struggle under certain edge cases, often related to third party integrations. The Kill switches allow us to turn these features off, which is a simple way to degrade our service in order to keep the business running. Of course it would be nice to not need the kill switches, but they have proven their value over time.

Business toggles are used to turn on certain features for specific users, segments or customers. It’s pretty straightforward to solve this in Unleash because of the flexible activation strategies, but we generally don’t use Unleash for this. It’s considered acceptable for experimenting with new ways to segment our service, but we don’t want the business toggles to become a permanent part of Unleash. We think permanent business rules should be handled by the application code and not Unleash.

Tips and Best Practices

We have been using feature toggles for several years in FINN. Here are some tips about how to use them more effectively:

Feature Toggles Increase Technical Debt

One of the biggest issue we have found is that due to ease of use developers love to add new feature toggles but don’t seem to remove them afterwards. Sometimes developers keep them around because they feel they are “nice to have”. Other times they are just forgotten or no one has the time to refactor and remove them. A feature toggles is technical debt from the moment it is added to the code. The general advice is to remove the toggle as soon as it has served its purpose. Every feature toggle adds a new code path through the application making it harder to test and debug the code.

Don’t Use Feature Toggles if You Don’t Need Them

In many cases you can safely add a new and unused feature to the application without protecting it with a feature toggle. It might be a new API endpoint or a new page on a separate URL, either way a feature toggle is not needed.

If you have to protect your new feature you should try to use as few if-statements as possible, ideally just one if-statement should guard the feature.

Further Plans for Unleash

Unleash has been actively used in production by FINN from 2014. The project has been lucky to have many different contributors over the years.

Currently Ivar, Sveinung and Vegard actively maintain the project. We have some great ideas for the future and we hope to see some of them implemented. In this section we will briefly explain some of these ideas. We would love to hear your feedback on them!

Unleash as SaaS

Currently you have to host your own instance of the API and the UI in order to use Unleash. This includes setting up a database and running a node application in production. We believe it could greatly improve adoption if Unleash was provided as a software as a service (SaaS). It would of course require us to add some access control, but this is something we have on the roadmap anyway. This might require us to charge the users of Unleash for a monthly cost and spend that money on hosting and new development.

Update April 2019 We finally did this and you can now check out our SaaS offering called Unleash-Hosted at https://www.unleash-hosted.com!

“AND Strategies”

Unleash 2.0 added support for multiple strategies. This was a huge improvement! But it is implemented as a simple array where all the strategies are “OR”-ed in order to determine if a feature should be enabled. We believe we could improve this by adding support for required strategies. This would make it possible to combine strategies more freely to create new segmented roll-out groups. For example, this would make it trivial to only gradually roll out a feature toggle to our beta users.

Authentication and Access Control

The Unleash service already provides hooks for registering your own middleware, so that you can actually add your own access control. But why not add a few ready to use implementations, such as Google Authentication or Okta? This would make it much easier to integrate into an enterprise already using an identity provider.

Application Scoped Toggles

Today all applications receive all toggles for everyone in the same cluster. In Unleash 2.0 we added a client registration feature, where all clients register with the API using a unique identifier. We could use the application identifiers to provide a simple way to define which application you intend to use a feature toggle for. This would make it possible to only expose relevant feature toggles to relevant applications.

Time to Live

Because most features toggles should be used for a limited time it would be nice to be able to set a time-to-live field on them. Then Unleash UI could then inform developers about toggles they use which have expired, which would be a great motivator to clean up feature toggles which have served their purpose.

A/B Testing

At FINN we use Unleash as a way to setup and manage simple A/B tests. Our current implementation is a bit cumbersome and we believe there is room for generalising and standardising this, making it available to anyone. Also, in order to support multivariate experiments we might need to change the API slightly.

These are just some of the ideas we have for Unleash. Which features would you like to see in Unleash? I would love to hear from you!

Summary

In this article we introduced feature toggles and how FINN uses Unleash to gradually release new features. This approach gives us more control because it decouples the process of putting new code into production from the release of new features for our users.

Sound interesting? Check out the getting started guide. If you need help please do not hesitate file an issue or reach out to any of the core contributors.

Thanks to everyone who has contributed to the project. All contributions are highly appreciated.

Åpen fagkveld 7. mars

2017-02-09T14:14:30+00:00

Åpen fagkveld 7. mars hjemme hos FINN - kl 15-20

For ett år siden åpnet vi dørene hjem til oss i Grensen for å dele våre erfaringer rundt hvordan vi jobber med produktutvikling. Vi fikk gode tilbakemeldinger på kvelden og nå ønsker vi å invitere til en ny fagkveld. Denne gangen blir agendaen mer rettet mot teknologien. Vi viser dere hvordan vi bygger og holder vedlike et av norges mest besøkte nettsteder.

Program

15:30	Velkommen til FINN ved Anders Skoe (adm.dir)
15:45	“Slik jobber vi i FINN”. 3 lyntaler som bør gi deg en bedre forståelse av hvordan vi jobber i FINN. Vi kommer blant annet innom organisering, strategi, arkitektur, devops, testing og kultur.
17:00	Introducing Node.js in an Enterprise. In this talk we will introduce Node.js and how we introduce it to a large enterprise in a safe way.
17:30	Beyond “Hello World”, handling complexity and organization in large systems. How architecture and organization come together to address the challenges we face at FINN.no. How we believe decentralised ownership and decision making can help improve development speed and product quality over time.
18:15	Maskinlæring, anbefalingsalgoritmer og datadrevne produkter Hvordan bruker FINN.no sine data til å lage gode produkter for brukerne sine? Hvilke datadrevne produkter har vi, hvordan lager vi anbefalinger for brukeren og hvordan forbedres algoritmene bak.
18:45	Hvordan vi flytter FINN.no ut i skyen Vi jobber for tiden med å migrere mikrotjenestene våre til en mer moderne, dynamisk skalerbar infrastruktur basert på Kubernetes. Hvordan gjør vi det, og hvordan sørger vi for at vi håndterer brukertrafikk og kan fortsette å levere endringer til prod 1000 ganger i uka mens vi migrerer.

Takk til alle som kom for en hyggelig kveld.

Presentasjoner

Category guessing

2017-02-02T14:14:30+00:00

The problem

Making it easy to fill out forms is essential for getting data into our system at FINN småjobber, which is a service to get help with tasks like house cleaning or renovation, where the user submits a job that non-professionals bid on.

Submitting a job is done by filling out a form with a title, a short description, and selecting the category the job belongs to. The category hierarchy has two levels: a main category and a subcategory. For example the main category domestic help, has the subcategories cleaning and babysitting. There are 6 main categories and 26 subcategories, and our assumption was that it was a hassle for the user to go through this long list. We wanted to simplify this by guessing the category. Ideally on the given title.

The solution

Machine learning

A typical solution to this kind of problem is to use machine learning, where the goal is to classify job titles into categories. We had a large set of historical data (130 000 registered jobs) where category already was selected by the user. This was a good starting point for a type of machine learning technique called supervised learning. In supervised learning historical data is used as a training set, and results in a classifier that can classify job titles into categories.

Delving into the field of machine learning is quite a complex task, and requires both theoretical understanding and knowhow of various software libraries. And in many cases, it requires access to a special machine learning infrastructure. So, as a lean product development team with limited resources we looked for an easier solution.

Good old search

Search was a sentral part of our service. All the jobs were indexed, and we had all the necessary libraries and infrastructure for this. So, what about using normal search to find the category?

A simple search on the title “Cleaning kitchen”, gave the following ranked results:

Match rank	Category
1	Renovation
2	Cleaning
3	Cleaning
4	Renovation
5	Cleaning
6	Cleaning
7	Cleaning
8	Cleaning
9	Carpentry and assembly
10	Cleaning

Our first attempt was to simply select the highest ranked match. This gave us 87% chance for guessing the main category correctly, and 68% chance of guessing the subcategory correctly.

These probabilities where calculated in a simulation by guessing category for 10000 random jobs in our dataset, and comparing the guessed category with what was actually selected by the user.

We were quite impressed by these results, however, we wished to provide the users with even better guesses. When we analysed the search results for various job titles, we found that the correct category often was not the highest ranked match, but number 2 or 3, as shown in the table above.

So based on this insight we came up with a simple algorithm that finds the most frequently represented categories among the 10* best matches. This gave us the following results:

Category	Frequency
Cleaning	7
Renovation	2
Carpentry and assembly	1

It was safe to assume that the correct category was cleaning, as it is represented 7 times.

We only wanted to make a guess when we were quite certain, and concluded that when a category is represented 6 or more times among the 10 best matches, we make a guess. As for the rest, we leave it up to the user to manually select a category.

This algorithm improved our guessing, and in our simulation gave us a 95% chance of guessing main category correctly, and 85% chance of guessing subcategory correctly.

In production

The following graph shows that we guess category for about 60% of all the jobs.

The graph also shows that around 90% of the jobs are submitted with the the guessed category. As for the remaining 10% of the guessed jobs, we assume that we have guessed wrong category, and that the user has changed the proposed category to one they found more suitable.

What is interesting is that having the user correct our guesses, it will automatically improve our guesses over time, as the given title will be associated with a more correct category.

Thanks to

Håvard Nesvold for proofreading and feedback.
The brilliant FINN småjobber team!
Simen Eide and Fredrik Jørgensen for good talks on machine learning.

*) We experimented with the number of best matches to include in the calculation, but found that 10 gave the overall best results.

WTF er Programmatic?

2016-12-15T10:17:32+00:00

… og hvorfor trenger jeg å bry meg om det?

For å forstå programmatisk kjøp og salg av annonser, er det viktig å forstå hva vi legger i begrepene. Hva skiller det eksempelvis fra tradisjonelt merkevaresalg? Er det ene alternativet bedre enn det andre?

I korte trekk er det viktig å forstå forskjellene mellom direktesalg og programmatisk salg. Direktesalg omfatter alle salg som gjøres fra et menneske til et annet menneske, og er ofte relasjonsbasert. Toyota ringer selger Y med ønske om bannerannonsering på FINN. Selger Y lager et tilbud basert på kundens behov og målsetninger, får deretter godkjenning fra kunden, booker det inn i ROSE og får til slutt en annen medarbeider på Schibsted sin traffic-avdeling til å sette den avtalte annonsen LIVE i annonsestyringssystemet AppNexus.

FINN sine annonseprodukter kan i all hovedsak deles inn i to hovedretninger, men begge starter med samme utgangspunkt. Annonsøren velger et banner, eksempelvis netboard. Dette banneret kan du deretter velge om du vil sette i en kontekstuell plassering på FINN, eksempelvis på et bilmerke, eller om du vil vise budskapet til en gitt målgruppe. Se illustrasjon nedenfor.

Programmatisk kjøp og salg er en form for automatisert kjøp av bannerannonser, hvor det i teorien er minimalt med menneskelig kontakt. Programmatisk er ikke et eget produkt, men heller en annen måte å kjøpe de overnevnte merkevareproduktene til FINN på.

Det er heller ikke tilgjengelig for alle kundene våre. Hvorfor ikke? Jo, det fordrer at Toyota har tilgang til et teknikksystem som kalles «Demand Side Platform» (DSP), hvor de selv enkelt kan velge ønskede medier og målgrupper uten å involvere en selger. Kunden kan kjøpe disse medieplasseringene – og målgruppene - automatisert fordi mediene benytter seg av en teknisk motpart, nemlig en «Sell Side Platform» (SSP) til å tilgjengeliggjøre varelageret sitt i en såkalt annonsebørs. Når rett bruker dukker opp på rett tid, settes det i gang en «auksjon» hvor de ulike kundene (via sine DSP-er) har en budkrig for å nå drømmebrukeren sin. Den som vinner budkrigen får vise annonsen sin til brukeren. Se illustrasjon nedenfor.

Hvorfor kutter vi ikke ut salgsleddet og selger kun programmatisk?

Det er flere grunner til at programmatisk enn så lenge står for en liten del av totalinntektene til FINN Salg. Først og fremst koster en DSP penger, og det benyttes hovedsakelig av profesjonelle merkevarekjøpere i mediebyrå som håndterer såpass store reklameinvesteringer at de kan forsvare teknikk-kostnadene. En bilforhandler fra Oppdal har neppe råd til disse systemene, og vil nok heller ikke tjene på en slik investering. I tillegg er programmatisk kjøp og salg fremdeles ungt i Norge, det krever opplæring i systemene, og i praksis er det fortsatt mye menneskelig kontakt som må til for å få på plass en kampanje.

Hello world!

2016-11-01T10:17:32+00:00

As some of you may know, FINN is owned by Schibsted. What may not be that familiar, is that Schibsted has online classified solutions in several markets around the world, including Blocket in Sweden, Le Bon Coin in France and Vibbo in Spain. Schibsted has several hundreds of developers distributed around the globe, and there is a strong need and will to coordinate and cooperate across geographical boundaries.

The need to cooperate

Reinventing the wheel is generally a bad idea. With that in mind, Schibsted has a strong focus on building global components that can be useful for all the companies within Schibsted. However, it is not always easy to understand the needs and challenges across national borders. There are cultural differences and different companies have reached different maturity levels. FINN has been around for years, and is one of the more mature classified sites within Schibsted. In an effort to help building global components, Schibsted has set up an international hub in Barcelona and developers from the local companies are encouraged to come to Barcelona to help out, share knowledge and build a common platform. Several developers from FINN have taken up the challenge and spent some months in beautiful and vibrant Barcelona. Here are Øyvind’s impressions.

Anders, Øyvind and Per Jørgen. Three happy FINN-campers in Barcelona

Øyvind goes messaging

Øyvind has worked as a developer for FINN for 9 years, but when he got the opportunity to move to Barcelona for 6 months, the decision was easy.

- The decision was really easy. There was little risk involved and I thought it would be an excellent opportunity to experience Barcelona. I love the city and the international atmosphere at the office. I think we are around 25 nationalities working here.

- Tell us about the work you have been doing in Barcelona

- I work as a developer here, in a team that developes a global messaging-component to be used by Schibsted-owned classified-sites around the world. When I arrived in January, we had integrated 6 global sites and the platform handled hundreds of thousands messages/day. Today, we have even more sites using our platform. It’s worth mentioning that the messaging-platform I’m working on, originally was developed by FINN. In 2014, however, it was decided that this platform was to be promoted to become a global component, so the responsibility was moved from Oslo to the global components team in Barcelona. The developers in Barcelona inherited the FINN code-base and developed it from being a FINN-only solution to a full-fledged multitenant solution. It’s not easy to inherit code like that, but I feel the ownership has been transferred now. The Barcelona-team owns the product and is soon going to integrate with FINN as well. We’ll replace FINN’s messaging platform with the global component-version, but that is not an easy task. FINN needs to preserve it’s messaging history, and that means we have to import 150 million messages! It’s a big job for our Cassandra-nodes.

- What are the technical differences compared to working in FINN?

- We use Amazon Web Services and it is really great to see how easy it is to scale up and set things into production. The infrastructure team here in Barcelona has done an amazing job and I feel it is really at the bleeding edge of technology working in this environment.

- You are soon returning to Norway. What will you miss from your stay in Barcelona?

- The people! It has been great working here, in this including atmosphere.

- Learned any lessons?

- When working in FINN, it’s easy to forget that you are part of a bigger company. I see that clearly now, the global perspective. It’s inspiring to know that I am a part of a global company.

Epilogue

Øyvind spent 6 months in Barcelona and came home with lots of new knowledge, new friends and a Messi-shirt. He contributed in building global components and believes FC Barcelona will win Champions League again this season. The migration of the FINN-messaging platform went off without a hitch and we are still looking for great developers!

Åpen studentkveld hjemme hos FINN

2016-10-18T00:00:00+00:00

FINN.no skal ha en åpen studentkveld der vi forteller litt om hvordan vi jobber med fokus på teknologi, produkt og brukeropplevelse.

Dette vil skje den 3. november fra klokken 16:00 – 20:00, her i våre lokaler i Grensen 5-7 i Oslo.

Hvem kan melde seg på?

Alle som studerer ved en relevant linje på et universitet eller en høyskole

Våre tre mål for kvelden

Dele kunnskap om hvordan vi jobber
Fortelle litt om hva vi ser etter i kandidater, og hjelpe studentene med å få et innblikk i hvordan reisen er fra student til jobb. Enten det er i FINN eller andre steder.
Vi ønsker selvfølgelig også å profilere FINN Teknologi og Brukeropplevelse som et spennende sted å jobbe, uten at dette er et rekrutteringsarrangement

I disse timene kommer vi til å gi dere et innblikk i hvordan det er å jobbe med Norges mest besøkte nettside.

Vi viser hvordan vi setter brukeren i fokus og jobber med å skape brukeropplevelser som begeistrer.
Hør om hvordan vi jobber i landets ledende interne teknologimiljø. Hvilke teknologier vi benytter, hvordan vi organiserer oss og hva vi ser etter når vi ansetter folk i FINN.
Du vil også få tips om hvordan det er å gå fra studenttilværelsen til å bli en medarbeider i et profesjonelt selskap

Som i alt vi gjør så skal vi også ha det litt gøy, vi bjudar på pizza og øl :-)

Hilsen alle oss i FINN

Disse plassene står og venter på deg:

Takk til alle som kom for en hyggelig kveld. Her er presentasjonene.

Presentasjoner

Browser statistics October 2016

2016-10-04T10:02:32+00:00

Last year, we presented the browser statistics of FINN.no as of June 2015. It’s time for an update!

How many visitors use a desktop or laptop?

Our first graph shows the share of our users using a mobile phone, tablet or a desktop computer to access FINN.no, regardless of whether they use our responsive web app (m.finn.no), or a native Android or iPhone app. 66% of our visits are now from a smartphone or a tablet. The traditional desktop/laptop has a market share of 34%.

If we exclude the native app users, which are obviously on either Android or iOS, what is the distribution between table, mobile and “other” (desktop) on our web site?

Which FINN.no application do people use?

We have two major offerings of FINN.no - the responsive web (served from the domains m.finn.no and www.finn.no), and native apps for mobile devices. 24% of our visits are from our native apps, and 76% from our responsive web.

Since we’re in the middle of a migration, some of our internal details leaks out in this graph; some parts of the finn.no traffic is served from the domain www.finn.no and the majority from the domain m.finn.no. They are supposed to be merged “real soon now”.

Browsers

First of all, let’s take a look at the numbers of the browser vendors. The ranking is clear, Apple is the biggest, ahead of Google, Microsoft, Samsung and Mozilla. Opera is no longer in the top 5.

In case you wondered which browser is the biggest on the “desktop” here’s the trend. (Ignore the orange and blue triangle markers)

The complete browser statistics, across all applications and devices are as follows:

It still seems like a good idea to make sure your website works well with Safari! The old giant Microsoft now has only the 5th spot with IE 11 (6.8%) and the 12th with Edge 13 (1.7%).

Mobile details

We got a request to give some more details about the mobile devices that uses FINN.no. Apple still has a strong position in Norway!

The most popular browsers on mobile devices:

Then the question is - what is the “Android Browser”? This is the break down of devices that has delivered traffic from the “Android Browser”.

Similarily for the Samsung Browser:.

And finally the trend on mobile browsers, by vendor:

Which version of Android or iOS?

Our apps need to work well on several versions of Android and iOS. How fast are our users upgrading? This shows the distribution of operating systems among our apps users.

Given that the user have an iPhone, which models are in use? Unfortunately, the data are quite spotty, but this graph might still be an indication of the truth.

Getting down to business with Prometheus

2016-08-26T13:00:00+00:00

Having moved towards an architecture of microservices, FINN is already leveraging a number of technologies for identifying and dealing with service outages within this architectural style. With each piece of functionality being comprised of a growing number of individual services, specialized tools are required for detection, analysis and mitigation of errors, and we are already using our fair share: Zipkin, Hystrix, Kibana, Grafana and Sensu to name some.

When it comes to metrics and monitoring of time series data, FINN has traditionally employed an infrastructure based on StatsD and Graphite. Recently, however, we opted to switch to Prometheus.

Prometheus, which is heavily inspired by Google’s Borgmon monitoring system, was originally developed by SoundCloud as a reaction to scaling issues experienced with just StatsD and Graphite. Taking into mind Adrian Cockcroft’s rule #4 of monitoring:

Monitoring systems need to be more available and scalable than the systems being monitored

Prometheus stands out as a great choice for us. Not only is it directly supported by Kubernetes, the future container management platform of choice here at FINN, it has been engineered from the ground up to deal with issues of scale and stability. In this new area of monitoring microservices, Graphite seems to be losing ground.

The switch to Prometheus has been rapidly implemented by all our autonomous service teams. Along with monitoring of the “traditional” application metrics like latency and memory usage, special care has also been taken to ensure implementation of adequate monitoring of business metrics.

Business metrics monitoring

The implications of one failing or partly broken service can be hard to evaluate when you are dealing with a large number of services. Since a complex problem now can be broken up into units that are truly independent, all individual parts can continue to work fine separately, while the end result is just not working. Business metrics monitoring is a key tool to successfully discovering and mitigating problems in this world.

Monitoring business metrics means monitoring the core performance indicators associated with the services you provide. These are the primary features of the business which will potentially suffer if any operational or functional part of the system is not performing. Turnbull describes business metrics as

the next layer up from application metrics […] Business metrics might include the number of new users/customers, number of sales, sales by value or location, or anything else that helps measure the state of a business

(from Turnbull, James: The Art of Monitoring).

Other examples of such metrics are orders per second at Amazon, or stream starts per second at Netflix.

My team at FINN, FINN småjobber (FINN småjobber is Norways leading marketplace for matching labor and demand for help with everyday tasks such as cleaning, moving, delivery and handyman work - and more!) has recently finished implementing metrics monitoring with Prometheus. The effort will require some tuning in the weeks to come in order to adjust thresholds for automatic alerts and so forth, but is already looking quite good.

Here I have collected five tips related to metrics monitoring with Prometheus based on our recent experiences, hopefully you may find some of them helpful.

Counters for the win

Counters are great. Unlike gauges, which can spike when you are not looking, counters has no loss of information between samples. But learn the core rule of thumb when working with them:
The only mathematical operations you can safely directly apply to a counter are rate, irate, increase, and resets. Anything else will cause you problems.

Tip courtesy of Brian Brazil.

Remember recording rules

Once your data are turned into an instant vector (like when aggregating with the sum function), further application of functions requiring range vectors is naturally not possible. However, there are times when you want to continue transforming your queries, and business metrics monitoring is one area where this fast becomes a reality. Say, if the business metric is number_of_purchases, you want to monitor that the total number of purchases is at a healthy level, across all your servers. Assuming counters of type number_of_purchases{server="X"}, a prometheus query for this is

sum(increase(number_of_purchases[10m]))

This yields a meaningful metric to reason about - the total number of new purchases the previous 10 minutes. To alert on deviances in this metric, functions like deriv, delta and holt_winters may be useful, but these all require range vectors. A solution is to record the data using a recording rule.

Recording rules allow for creating new time series from user-configured expressions, and means you can now query on these as if they were just any other metric.

Recording rules are otherwise used to precompute computationally expensive expressions and save them as new time series, and are particularly useful when used in monitoring dashboards (which are often refreshed at frequent intervals).

Business metric monitoring tip #1: Use time() to deal with seasonality

Business metrics, like number_of_purchases, are sometimes varying a lot throughout the day, typically correlated with the traffic to your site, and often also showing trends related to day of week or other signs of seasonality. Functions like holt_winters may help in alerting on such metrics, but sometimes this is not adequate, for example when the general volume is just too low.

At such times, the time function may come in handy. time() returns the number of seconds since January 1, 1970, and as such may for example be used to determine the current hour of day:

(time() / (60 * 60)) % 24

With this in hand, expressions can be built to incorporate special handling for chosen periods, for example to compensate for nighttime hours where traffic is low and when alerts on low volumes typically would be triggered.

Adding the expression + (((time() / (60 * 60)) % 24) < bool 8) * 1000 to a query would for example add 1000 to a metric for hours 00:00 to 08:00, which may then rule these periods out for any alerting thresholds which you may have set.

time() function for the current hour of day

(Depending on your alerting tool, the option to exclude time periods from your alerts may also be configurable in that tool. However, this will usually not remedy the status in any dashboards for the metric, something which using a time()-clause in the query accomplishes).

Business metric monitoring tip #2: Use micro-metrics

Not strictly related to Prometheus as a technology, you may often find that the volume of an interesting metric is so low that efficient monitoring and alerting is hard. Low volumes however, does not mean that monitoring is not useful, it merely means that it will take a longer time to find deviances in the metric in question. The problem is very much analogous to that of doing A/B testing on low-traffic websites (like explained here).

A tip in this situation is to instead, or in addition to your key metrics, monitor correlated micro-metrics (or micro-conversions to use the terminology from A/B-testing). Here, instead of monitoring the key metric number_of_purchases, one would monitor related metrics like clicks on the “add to cart”-button or the number of product page views. The main point is that the higher the volume, the faster one can detect deviances in metrics. This is fundamentally due to the fact that statistical significance increases with a higher sample population, and a higher sample population is, of course, faster to achieve with high traffic.

Use better graphing tools to effectively tune your queries

Ok, the prometheus GUI comes with some core graphing capabilities, but it really cannot compare to what is provided by a tool like Grafana. In Grafana, which has support for Prometheus out-of-the-box, you can stack metrics, override time intervals, template variables and much, much more, all which enables more efficient testing and analysis of your queries.

Wrapping up, I would like to recommend the previously referenced blog by Brian Brazil at Robust Perception for more interesting Prometheus stuff: http://www.robustperception.io/blog/, and of course the official docs, which also links to where you can get in touch with the community.

Summer Project 2016: Customer Insight

2016-08-12T11:04:35+00:00

This year’s summer intern team consisted of five technology students from UiO and NTNU and one in-house design intern. We have released continuously throughout an eight week period, and have ended with a final solution which is available to the public.

The problem presented for this year’s summer project was; What can FINN offer professional vendors at Torget to motivate them to continue advertising as professional vendors. To help us get started, we had a few meetings with FINN’s innovation team to work out an ambition and a few ideas. Our ambition was to create value for professional vendors that use Torget as a market to run their business. To achieve this we aimed to provide better advertisement insight for vendors, and give them a better look into their general performance at Torget.

Our application consists of three main parts. The first presents geographical data about the ad’s visitors, show n on a map of Norway. This is primarily shown as percentages on each of Norway’s five regions, with more specific information shown in the cat’s conversation box. The latter also changes when the user clicks around the map. The second part of the application shows demographic data about the customers, which includes age and gender. Here we hope to open a more insightful relationship between seller and customer, which might for example allow sellers to properly target their correct customer group when advertising new products.

The last part of the application is a competitive ranking list that compares ads in the user’s category based on number of visits. This is shown with green text to the right of each ad in the list. The user also has the option to change which category they want to see the ranking for, in the blue menu. One of the main goals here is to give users an incentive towards comparing their advertisements to their competitors’, by letting them see what part of the market their customers prefer.

Implementing the application

The architecture we went with for the application can mostly be divided into two parts: One that collects statistical data, and one that presents that data to the customer. These meet in the middle through a PostgreSQL database.

The collecting part of the architecture is made up of a Java application that listens to events using FINN’s messaging broker, Kafka. This is usually referred to as a consumer. In our situation the consumer receives a message every time a user clicks on a sales ad, so that we can continue to aggregate anonymous statistical data for that specific ad.

The presenting part is mostly made up of a three layer hierarchy. At the bottom we have a Java REST-API that reads from the database, which is built using Spring Boot. The next layer is a Node.js application, which is responsible for responding to all customer facing requests. Upon receiving a visit this application performs an initial server-side render of the website, which is built using React. This also includes a primary API request, so as to be able to present a mostly complete website to the user on the first request.

This has the positive side effect that the application works decently even in browsers with JavaScript disabled. Upon receiving the initial render of the website the browser client takes over, which we consider the topmost layer of the presentational hierarchy. From here we fire off more API requests, to retrieve the data which is needed to present the rest of the website. This is an important step in maintaining a low average response time, as the user will be able to use most of the application while waiting for the other API requests to finish.

The geographical map and the statistical charts are made using D3.js. The map itself is written using pure D3, while the charts make use of the React friendly wrapper Victory. The data that lays the foundation for the map is transformed from raw Kartverket data to a D3 friendly format, using TopoJSON, a library made by the same author as D3. This allows us to build interactions with the map using regular D3 code, which we at the moment utilise to merge the Norwegian counties into larger regions, and to react to the user’s hover and click events.

The front- and backend applications are deployed in Docker containers, and managed through FINN’s new Kubernetes cluster. This follows FINN Infrastructure’s new deployment strategy, and has made our meeting with pipelines and deployment work easy.

The designers’ perspective

When creating new features, technology is obviously important, but making sure people can use it is just as essential. As the team’s designers, keeping the user in mind was our priority number one. We spent the summer sketching, prototyping, testing, and continuously iterating to make the final design as user friendly as possible. This required a close collaboration with the developers in the team and attention to detail.

We started out with an idea workshop where we sketched out different layout suggestions. This worked as the basis for further development of the user interface. We then made clickable prototypes in Adobe Experience Design and Invision, which were used in both formal user tests in FINN’s usability lab and more spontaneous ones in the busy streets of Oslo. The purpose of these tests was to assess people’s understanding of the information presented to them on the screen, as well as general navigation and interaction with the interface. Based on this feedback we were able to improve the different features in the app and ended up with a result we’re very pleased with.

Final thoughts - this has been a fun summer!

After eight weeks of working at FINN we’ve had the pleasure of being introduced to a wide variety of both people and technologies. Asking for help has never been an issue, and we’re pretty sure we’ve bothered most of the people daring enough to work here this summer. To be able to release a product and receive feedback from actual users is definitely an exciting experience, especially for such a large customer base as FINN’s.

So long, and thanks for all the parrots.

Splitting up a storyboard in an Objective-C/Swift mixed legacy project

2016-06-22T11:19:07+00:00

Splitting up a storyboard in an Objective-C/Swift mixed legacy project

How do you go about splitting up your storyboard into multiple smaller storyboards? And why would you? In this article I’ll tell you why we wanted to do this, what kind of problems we encountered along the way, and how we solved them.

The problem: Large storyboards makes iOS developers into sad pandas

As the FINN app has grown the last few years, so has our storyboard. Using storyboards means you get several useful features, and it can give you a visual conceptual overview of how your app works. I say “can”, because this is not necessarily the case. Also, using storyboard has some quirks that are not always appreciated when multiple developers are working on the same project. Any small change to a storyboard might mean several changes to the storyboard xml. In addition to this, XCode has an annoying tendency to recalculate coordinates for several storyboard items just because you opened the file! And yet another annoyance is that the larger your storyboard, the slower it is to work with. We usually waited at least 5 seconds just for the file to open, and actions were often laggy.

So, the iOS team had long ago decided it was time to split it up, but that job is not necessarily easy. Or small. Or even fun. And the Jira task for it also stated “Beware of scope creep…”. Yeah, it crept.

The FINN app: Some history

The current FINN app was launched in August 2013. During these three years, the app has of course grown a bit, and during the last year or so several new features have been added using Swift, and quite a few of the older Objective-C classes have been rewritten in Swift. However, the majority of the code base is still Objective-C. Interoperability between Swift and Objective-C is therefore crucial.

A quick count of files says that we currently have 171 Swift files and 349 .m files in our project (not counting third-party code, of course). These files contain >44,000 lines of code, comprised of 30,000 lines of Objective-C code and >14,000 lines of Swift code (excluding comments and whitespace).

Starting out small

We decided to start small and extract the part of our app called “Min FINN” (My FINN) to its own storyboard first. This is a fairly autonomous, although not quite, part of the app. Most of the navigation within this part of the app is internal to that feature, but there are a few entry points into it from other parts of the app.

Some of the navigation is done by segues, but there are multiple places that instantiates the scene’s viewcontroller via the storyboard directly. This, of course, means code like this is littered throughout the codebase:

FINSearchListViewController *resultViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"resultListViewController"];

There are 2 problems with this:

It refers to “self.storyboard”, which means it will only work as long as the viewcontroller exists on the same storyboard as the current viewcontroller
It uses a hard-coded string to refer to the storyboard identifier “resultListViewController”, which is error-prone

We needed a better way.

When we started out, this is how our MainStoryboard_iPhone looked like:

Yes, kinda messy and not necessarily super-manageable. And not very informative either, conceptually. So how do we go about splitting it up?

With XCode 7, we got a new, nice feature - Refactor to storyboard… :

You just select all the scenes you want to extract and then this refactor feature will both create a new storyboard for you, and it will wire up any connections between scenes in the old storyboard and the new storyboard. This, however, doesn’t work if you’re supporting iOS 8 AND you’re using relationship segues (i.e. segues from a UITabBarController).

Which is the case for us, of course. But, at least it gives us a handy shortcut for extracting the scenes into a new storyboard. We just deleted the resulting storyboard references. Besides, for non-relationship situations we would be on our own anyway.

Juggling two storyboards

Ok, so now we had a new storyboard, with just the “Min FINN” scenes:

Most of the “self.storyboard instantiateViewControllerWithIdentifier calls spread out around the code still work, but not all. For example, in some areas of the Min FINN storyboard, we open up the FINObjectViewController that is still on the MainStoryboard. And there are a couple of scenes on the Min FINN storyboard that are accessed from scenes on the main storyboard. How can we know at call site which storyboard a given scene is located on? There are many such invocations. And as we later down the line continue to split up the main storyboard, this will be even more fragmented. We needed a common place to handle this so that the call site didn’t need to know where a given scene is located.

Generating common code

We started out by searching for already existing tools that could help us. We found several, among them Swiftgen, and tried them all. Swiftgen is a very thorough and well-written tool for generating enums and structs that handle multiple storyboards. However, it supports only Swift and cannot be used for Objective-C, making it a no-go for us. It’s also quite elaborate with enums, structs, protocols and extensions, and outputs a fair amount of code. Most other tools we found were either Swift-only or Objc-only, or they created only constants for the identifiers.

But we very much liked the Swiftgen approach of creating functions that can be called directly, and that will return an instance of the correct class.

So we decided to create our own generator. The first iteration was to create a Swift-class that did not rely on Swift enums that are unusable in Objc, and that could be called from both Swift and Objc. This seemed absolutely doable, and our first generated Swift-file had static functions like this:

static func instantiateWebViewController() -> FINWebViewController {
    return self.storyboard.instantiateViewControllerWithIdentifier(MainStoryboardIdentifier.WebViewController.rawValue) as! FINWebViewController
}

static func instantiateFrontPageSearchViewController() -> FrontPageSearchViewController {
    return self.storyboard.instantiateViewControllerWithIdentifier(MainStoryboardIdentifier.FrontPageSearchViewController.rawValue) as! FrontPageSearchViewController
}

Objective-C compiler complaining

Looks pretty good, right? One would think so. Except, it doesn’t work in Objective-C. Why? The FINWebViewController is an Objective-C class, and therefore has that name on both sides of the table. Win! The FrontPageSearchController, however, is a Swift class, and therefore has the name “FINFrontPageSearchViewController” on the objc side of the table. Doh! When the instantiateFrontPageSearchViewController was called from objc it didn’t work, because the expected class was FINFrontPageSearchViewController and the returned class was FrontPageSearchViewController.

After a lot of trial and error (I can assure you that I’m sparing you a lot of painful details here), we finally gave in and decided to generate separate instantiator classes for objc and Swift. Win! Or was it?

Not really. Since some of the Swift view controllers are used from both objc and Swift, we annotate them with the objc name:

@objc (FinUserAdListViewController)
class UserAdListViewController : UIViewController, UICollectionViewDataSource, UICollectionVi...

That’s all well and good, but since our Python script parses the storyboard files and extracts the storyboard identifiers and their respective custom class names (if any), we had class names with the prefix (objc classes) and class names without the prefix (Swift classes). This enabled us to check for this prefix while generating. When generating Objc code, we added the prefix to the Swift classname, and when generating Swift code, we left it as is. Remember, in the storyboard, the non-prefixed Swift class name was used.

Now, this resulted in compiler warnings like this:

Double-you tee eff? Even though the UserAdListViewController class is annotated with the prefixed name, this doesn’t work. Since we practise zero-tolerance for warnings in our project, we needed to fix this. What about casting it to the class it’s supposed to return? Let’s give it a shot:

Warning gone! Now it HAS to work! Yes?

No. Although we’re instantiating a viewcontroller that is listed in the storyboard as a UserAdListViewController, which is annotated as a FINUserAdListViewController, this does not return a FINUserAdListViewController objc instance. What does it return? A UIViewController…

(To be clear, the hurdles I’m listing here are probably less than half the hurdles and dead ends we’ve met. You would probably be quite bored if I had listed them all. And frankly, I don’t even remember them all anymore. Which is probably a good thing.)

By this time, I was starting to feel somewhat inundated. And was repeatedly asking myself why I had picked this task as my first task as a newcomer to the team.

Allright, what if we define this view controller as “FINUserAdListViewController” (the objc annotated name) in the storyboard scene? Surely, that has to work in Objc?

Lo and behold, it did! Now Objective-C recognizes the class.

Swift compiler complaining

Oh, wait. Now the generated Swift class doesn’t work?

Of course, in the Swift realm, there is no such class. It’s supposed to refer to UserAdListViewController. No problem, we’ll just remove the prefix now when we’re generating the Swift code. But wait. The way we’d known whether a class was an Objc class or a Swift class was to check for this prefix, right? So how do we figure out whether a class found in the storyboard is a Swift class when they ALL have prefixes now?

Ok. What does Objc classes have that Swift classes don’t, that is easily accessible from a script? Header files. So, we created a Python function that crawls through all the files in the project and collects all header-filenames in a Set. Then, when generating the Swift code, we test every class name (with an added “.h”) against this Set. Does the Set contain an entry with this name? Yes -> Objc class. No -> Swift class.

Would it be possible to do that for Swift files instead? No, because there isn’t necessaritly a 1-1 correlation between Swift classes and filenames.

Believe it or not: this was the last hurdle, and we now have generated code for Objc and Swift. For Objc, the generated code looks like this:

And at call site:

Swift generated code:

At call site:

That’s about as simple as you can get it.

Well, almost. You might have noticed we have a Swift enum for the storyboard identifier. We could, of course, skipped the enum and just used the identifier string directly in the instantiate method. We’re undecided on that for now, as we thought we might use the enums for something else. But we might end up removing the enums at a later point.

As you can clearly see, this was not a straightforward task, and Swift/Objective-C in(ter)operability has some rough edges. So, to wrap this up I will list the steps you should follow, and at the end of this list there is a link to the Python script and a demo project if you’re facing the same challenges as we did. The script is unfortunately not optimized and generic so that you can plug and play, but it should be fairly easy to adjust it to your own needs. Feel free to generify it and create a pull request to enhance the script’s usefulness for others!

Setting up your project for generation

1 - Choose a small, reasonably autonomous part of your app that will get the first separate storyboard.

2 - Select the necessary scenes, go to Editor -> Refactor to Storyboard…

3 - Name your new storyboard. If you’re not using relationship segues, the generated storyboard references should work well for you, and you can leave them!

4 - Create a Run script under Buile Phases that will trigger the Python script (follow the guide in the GitHub project for this).

As you can see in the demo project, the MWStoryboardScenes.py file should be placed in your project somewhere. In our projects, we have a Scripts folder for these things, and this is not added to the XCode project (but it is handled by Git as a part of the project, of course).

Now, this script needs to run every time the project is built, before building source files. For a guide on how to set this up, take a look at the GitHub repository readme. You also need to set up paths to your storyboards.

When you’ve run this script for the first time, you have the functions that are necessary to instantiate your view controllers.

5 - Find all places where view controllers are instantiated via the storyboard (and not via segues) and change these invocations so that they use the generated storyboard functions.

As mentioned, a more thorough explanation on how to use the generator is provided in the README of the GitHub project.

Good luck with splitting your storyboard :-)

How to Make Your Colleagues Think Accessibility

2016-06-13T11:19:07+00:00

What to do when your colleagues don't make accessible products.

When developing a website which affects the life of almost every person in Norway, accessibility should be a main priority. Norwegians who are not able to use FINN.no are unable to find a home without assistance. Accessibility considerations are too often an afterthought, and this can have profound effects.

In the Spring of 2014, a small group of employees got together to find out how to make our colleagues care more about accessibility. After two years, the measures we think work best are workshops, user tests, and automatic validation.

Workshops

On a workshop we reveal accessibility issues and discuss how to solve them.

At our workshops we demonstrate how FINN.no works for people with disabilities before we let the team members test it the same way themselves. People find this a lot more inspiring than memorising WCAG 2.0 recommendations and running automatic validations locally.

How to test

Scaling Pages

The simplest thing to test for is page scalability, both for fonts and for the entire page. Most desktop browsers make it possible to scale both fonts and page no matter how poorly coded the pages are, but on mobile phones it’s still quite common to lock page size to screen size. It must be possible to pinch zoom every page on mobile phones.

The Straw Test

Another simple method is to pretend that you are looking at the page through a straw. This simulates how the page will appear for partially sighted people who either have a limited field of vision or need to magnify the page to read it. This reveals relevant elements spaced too far apart in a way that will make it difficult to navigate the page.

Keyboard Navigation

Most pages are designed and coded with mouse click and touch screens in mind, and we easily forget about people with broken arms, motor disabilities, or other reasons for not being able to use a mouse. By navigating the page with only a keyboard, we test that focus on links and buttons appears in a logical order.

High Contrast

Many partially sighted people are sensitive to bright light or struggle to see the contrasts on a page. They often invert the colors on the screen or switch to high contrast mode. Check that your pages are still readable in these modes. If you are using sprites or in other ways put relevant information in a background image, you may notice that this disappears on computers running Windows.

Screen Reader

Blind people depend on a screen reader to use a computer. Most cell phones have a screen reader built in by default. On Apple devices this is called VoiceOver and on Android it’s called TalkBack. You can turn this on in the accessibility settings. It takes some practice to use a screen reader, but when you handle it, you are likely to uncover a huge number of accessibility flaws.

Devopers actually think accessibility testing is fun (but you don't really have to use a straw).

User Tests

By testing our site during the workshops, we uncover the most obvious bugs. However, we cannot know how it actually feels for users with disabilities to use our site. That’s why we run user tests with blind and partially sighted users. We have people who run user tests at FINN, but because we don’t know what it’s like to be blind, we don’t always understand why they react as they do. Therefore, we need some external expertise to assist us. We are cooperating with Blindeforbundet (The National Association of Blind and Partially sighted, NABP) who run these tests for us.

Designers, developers, and product managers watch the tests, which are also videotaped for further analysis. After the tests, NABP hands over the videotapes with a commented report and recommended improvements.

Automatic Validation

Our next step is to include WCAG validation as a part of our automated tests. There are several tools offering this, including Pa11y and Tenon.io. We are likely to use Pa11y in our first effort, because it’s free.

Automatic validation gives an overview over detectable WCAG violations, and may work as a KPI for accessibility awareness or even break builds with new violations. If all developers and designers pay attention to this, we hopefully will stop deploying code with new WCAG violations.

Is It Working?

Accessibility used to be looked on as some kind of geeky pedantry for something that otherwise works fine. This attitude has changed over the past few years, and people take complaints about accessibility more seriously. In an internal speech I once mentioned that many of our pages weren’t scaleable on mobile. The next day, three teams had fixed that. A blind user complained that our iPhone app didn’t work with VoiceOver. This was fixed before the next release (several hours later). Another user complained that our banner ads caused epileptic seizure. This resulted in a huge debate on our forums, but sadly we still haven’t found a perfect way to use banners in a way that make both users and advertisers happy.

Do you have "maximum-scale=1, user-scaleable=no" in your viewport metatag? Delete it! It took three of our teams less than a day to make their pages scaleable.

Whatever we do, the most effective way to get our accessibility flaws fixed is feedback from the users. We are not perfect yet, and we don’t always find accessible solutions that make everyone satisfied. Knowing that a user is unable to use FINN.no is far more convincing to our managers than being told that they have a WCAG violation. So please, if you have disabilities that make FINN.no hard to use, contact our service desk. It takes only one complaint to put your problem on our agenda.

Åpen fagkveld, presentasjoner

2016-02-15T22:04:17+00:00

Onsdag 10. februar så hadde vi 140 personer på besøk hos FINN. Vi fortale litt om hvordan vi jobber i FINN og serverte pølser og øl. Nedenfor kan du finne presentasjonene fra fagkvelden.

Programmet

15:15 - 15:30	Velkommen til FINN v/ adm.dir Sondre Gravir
15:30 – 16:10	Brukerorientering og datadrevet produktutvikling Hvordan involvere brukeren? (Dag Olav) Brukerinvolvering + data = suksess (Marthe)
16:10 – 16:45	Pølser og øl
16:45 – 19:00	To gjennomkjøringer av temasporene
19:00 – 21:00	Mingling med øl og snacks

Presentasjonene var delt inn i tre spor.

Temasporene

Spor 1: Teknologi - kultur og organisering	Spor 2: Produktutvikling	Spor 3: Innovasjon og intraprenørskap
Why & How we do DevOps in FINN (Jo) Slik reorganiserte vi 100 utviklere (Nicolai)	Brukerorientert, målstyrt og iterativt (Bjørn Henrik & Bente Mari) Eksperimentering på skjema (Per Gunnar) Brukeropplevelse i FINN (Kaija)	Innovasjon (Jens)

Lær mer om FINN.no og se bilder og videoer her:

Åpen fagkveld hjemme hos FINN

2016-01-23T20:27:47+00:00

Velkommen til åpen fagkveld onsdag 10. februar, fra kl 15-20 hjemme hos FINN i Grensen 5-7.

Link til påmelding: (Beklager men arrangementet er fullt.)

Før jul kom vi opp med en idé om å invitere til en fagkveld der vi skulle dele av våre FINNske erfaringer. Vi bestemte oss for å sjekke interessen for denne ideen ved å sende ut en forespørsel i sosiale medier. Vi fikk nesten 100 svar! Det har gjort oss ganske sikre på at det er interesse for en fagkveld på FINN, og vi har satt sammen et program basert på tilbakemeldingene vi mottok.

Tilbakemeldinger

Program

14:45	Dørene åpner
15:15	Velkommen til FINN v/ adm.dir Sondre Gravir
15:30 – 16:00	Brukerorientering og datadrevet produktutvikling Hør om hvordan vi jobber for å forstå brukernes behov. Vi vil blant annet presentere hvordan vi jobbet med lanseringen av den nye FINN-plattformen og hvordan vi jobber med brukerinnsikt på Torget.
16:00 – 16:30	Matbit og drikke
16:45 – 17:45	Første gjennomkjøring av temaspor
18:00 – 19:00	Andre gjennomkjøring av temaspor
19:00 – 20:00	Enda mer mat og drikke!

Alle som deltar kan velge to av tre spor. Hvert spor varer én time, deretter rullering og én time til. Temasporene blir en blanding av lyntaler og diskusjon.

Link til påmelding: (Beklager men arrangementet er fullt.)

Temasporene

Spor 1: Teknologi - kultur og organisering	Spor 2: Produktutvikling	Spor 3: Innovasjon og intraprenørskap
Hør om hvordan vi jobber for å ha landets ledende teknologimiljø. Vi vil bla. presentere veien til DevOps og hvordan vi reorganiserte 100 utviklere, transparent og inkluderende!	Produktutvikling i teori og praksis. Vi vil presentere hvordan vi jobber innsiktsdrevet og målstyrt med å utvikle produkter som begeistrer	Hør innovasjonshistorien vår og hvordan vi jobber for å opprettholde en sterk innovasjonskultur. Vi vil presentere både hva vi har lykkes med og ikke lykkes med. Her er det takhøyde og stort rom for læring!

Link til påmelding: (Beklager men arrangementet er fullt.)

Husk å merke hvilke to spor du ønsker å delta på, og om du blir igjen til mingling fra 19-20. Vi lover deg – det kommer til å bli en fin(n) kveld!

Why Try - Java 8 lambdas and checked Exceptions

2016-01-20T07:13:00+00:00

Why won't this compile? (And why does riskyTask throw a PrinterException?!)

One does not have to use lambdas in Java 8 long before running into the obstacle of checked Exceptions. Because the PrinterException is checked, the compiler forces us to deal with it, even within a lambda:

private void demonstrate() {
    IntStream ones = IntStream.generate(() -> 1);
    ones.map(i -> {
        try {
            return riskyTask(i);
        } catch (PrinterException e) {
            System.out.println(
                "Your printer is out of ink or laser beams!"
            );
        }
        return i;
    });
}

private int riskyTask(int a) throws PrinterException {
    return 2;
}

But we don’t like to do this. We use lambdas to express intent in a concise and elegant fashion. The try/catch-brackets feels like noise.

For this reason, FINN.no’s open source lambda-companion project introduces a useful structure for using lambdas in a world with checked Exceptions : Try.

A Try represents a computation which might fail, and is always represented as a Success or a Failure (but never both). The concept borrows from Scala’s Try, and shares several properties to other monadic functional structures :

Future (completable success or failure)
Optional (present or empty value)
Either (one of two values)

FINN.no’s Try is right-biased, meaning that one can map and flatMap on a Try without having to add specific logic to handle a Try being a Failure; the computation will simply only take place if it is a Success.

Using a Try we could refactor our riskyTask-example:

private void demonstrate() {
    IntStream ones = IntStream.generate(() -> 1);
    Stream<Try<Integer>> tryStream = ones.mapToObj(this::riskyTask);
    tryStream.forEach(this::print);
}

private Try<Integer> riskyTask(int a) {
    return new Success<>(2);
}

private void print(Try<Integer> t) {
    Integer defaultNumber = 0;
    Integer i = t.recover(success -> success, failure -> defaultNumber);
    System.out.print(i);
}

This is contrived of course. And one should rarely accept a Try as a method argument. (And one should rarely generate an infinite stream of 2s and print them out.)

Lets look at a slightly more realistic example. Here we want to validate an order using different services that can throw a checked Exception:

public class OrderProcessor {

    CustomerLookupService customerLookupService = 
            new CustomerLookupService();
    ProductLookupService productLookupService = 
            new ProductLookupService();
    PaymentService paymentService = 
            new PaymentService();

    public static void main(String[] args) {
        new OrderProcessor().validateOrder(14L, 21L);
    }

    private Boolean validateOrder(Long customerId, Long productId) {
        return Try.of(customerLookupService::lookup, customerId)
                .flatMap(customerFound ->
                        Try.of(productLookupService::lookup, productId))
                .flatMap(price ->
                        Try.of(paymentService::pay, customerId, price))
                .recover(Function.identity(), this::handleFailure);
    }

    private Boolean handleFailure(Throwable throwable) {
        System.out.println(
                "Could not process order because of " + 
                        throwable.getCause());
        return Boolean.FALSE;
    }
}

class CustomerLookupService {
    Boolean lookup(Long customerId) throws Exception { 
        return Boolean.TRUE; 
    }
}

class ProductLookupService {
    Long lookup(Long productId) throws Exception { 
        return 50L; 
    }
}

class PaymentService {
    Boolean pay(Long customerId, Long price) throws Exception { 
        return Boolean.TRUE;
    }
}

In this example we combine services to determine if a Order is valid or not. We use Try.flatMap() because this guarantees that the provided lambda will only be run if the proceeding Try is a Success. If any of the 3 service calls result in a Failure, the flatMapping call-chain will simply roll the Failure forward. The recover-method takes two lambdas: One to be executed in the case of Success, and one to be executed in the case of Failure.

You can read more about Try, Either, StreamableOptional and more in FINN.no’s open sourced lambda-companion project available at github and for use in your project through the central maven repository.

Happy flatMapping, and may all your Tries be Successes!

The great big Schibsted programming language survey 2015

2015-11-21T21:28:28+00:00

During September, I collected answers on a survey about programming languages from people who work in one of the many companies that make up the Schibsted family. The survey is neither requested, sanctioned, approved or even suggested by anyone in management, in any Schibsted company. It is entirely my own personal pet-project, because I’m interested in these things.

The survey opens with the following introduction:

In 2012, there was a discussion about technology choices, and particulary programming languages to use in FINN. As always in such discussions, claims like “It’s impossible to hire people who know language X, so we need to always use Java” or “We can’t expect programmers to learn a new language just because we think language Y would be a better fit” were put forward.

Personally, I’ve never liked those arguments, because they contradict what I think defines a “good programmer”, and I’m optimistically hoping that we only hire “good programmers”. Also, instead of opinions, beliefs, rumours and hearsay, let’s be data driven in these discussions! So I decided I wanted to put it to the test, and created a survey and posted it for all the company to see.

The results from that survey were blogged about here: http://tech.finn.no/2012/09/04/leaving-the-tower-of-babel/

Now, almost three years later, a lot of changes are happening. FINN is becoming more close with it’s Schibsted brethren, Schibsted itself is turning towards a more technological outlook, and I felt it was time to get an updated view, and why not involve all of Schibsted while we’re at it.

So, “The great big Schibsted programming language survey 2015” was created, and hopefully it will make the rounds in all of Schibsted and gather some interesting, fun, surprising and (with a bit of luck) useful insights.

About the responses

309 people responded to the survey, which is more than many of my co-workers thought would respond.

Over 50% of respondents are in their thirties, and if we look at the ages from 26 to 40, we cover over 75% of respondents. On gender, it’s even worse, with a whopping 93.5% of respondents admitting to being male. We also seem to shun inexperienced people, with 42% having 10 years of experience or more, and only 2.3% of respondents fresh from school.

It seems we have some work to do with regards to diversity.

According to Arena (The Schibsted-wide intranet), there are just under 125 companies in the Schibsted familiy. In those companies, 9355 people are employed.

That means somewhere between 20% and 30% response rate in the target group, which isn’t too bad.

Unfortunately, our results are not representative. FINN is by no means the largest company, but supplied the most answers with 80 respondents. The second largest group was the various incarnations of Schibsted Product & Technology (SPT), which supplied 53 responses if we combine them all. Those two supplied over a third of all responses. Third place belongs to Schibsted Tech Polska (16 responses), which beat Wilhaben (15), Le Bon Coin (13), Blocket (12) and Aftonbladet (10) to close out the top 7.

This probably means that the results are more a description of FINN and SPT than a description of Schibsted in general.

Those 309 people listed 103 different companies, which is about 20 less than the total number of companies. That sounds great! Except I’m fairly certain that’s not what I have. Trying to normalize the data entered here, fixing variations in spelling and other issues I’ll get back to in a minute, I reduced it to 45 companies, one of which was Schibsted itself.

Those issues are companies that don’t exist (atleast according to Arena)! I’ve tried my best to guess which company was really intended, but might have made mistakes here and there. In some cases, I had no idea which company to use instead, so I just left it as is, even if the company is not listed as part of Schibsted. Some examples: Schibsted Tech Madrid, SPT - Madrid, SCM Spain, SCM Central, Schibsted Norge Tech, Plan3 and Schibsted ATE. None of these were listed anywhere, but I’ve tried my best to place them some logical place.

Another problem was SPT aka Schibsted Products & Technology. SPT is techically four companies, SPT Norway, SPT Sweden, SPT Spain and SPT UK. Most people in SPT simply answered SPT, but they estimated as little as 20 people working in SPT, and as high as 250. I’ve disregarded their disagreement, and combined them all to just SPT.

Two people said they work in Schibsted, but they had wildly differing opinions about how many people work in Schibsted. One said 10000 total, 1500 technical, the other said 1000 total, 200 technical.

Disregarding the problems here, let’s look at the numbers entered. Using the averages for each company and summing up, we arrive at roughly 4500 people working in those 47 companies. For technical positions, the sum of averages is roughly 1600. Since we have responses from about half the companies, that fits nicely with the total being just under 10000.

Languages in regular use

The most commonly used language, if you can call it that, is Bash. 52.1% of respondents use Bash regularly. Close behind, at 50.2% is JavaScript. Neither of these are surprising, considering the business we’re in. Java clocks in at 49.2%, and from there it’s quite a drop to Python at 28.2% and PHP at 24.9%. C manages 18.1%, Ruby and Scala are both used by 17.8%, and Groovy (11.7%) and Go (10.7%) conclude the top 10.

Of the previously mentioned top 7 companies with regards to number of responses, Le Bon Coin and Blocket are the only two companies that are not predominantly Java shops. Those two have a more wide array of languages in common use, with high numbers for multiple languages. When we combine those two companies, 88% use C, 80% uses PHP, 72% uses Bash and sh, 64% uses Python and only 56% uses Java. Ruby, R are popular in Le Bon Coin, while Blockets own template language and JavaScript are popular in Blocket.

Singling out FINN and SPT in a similar way, it’s a different story. Java is used by 76.7% in these companies, JavaScript and Bash/sh are around the same, at 48.1% and 46.6% respectively. Scala is popular with a share of 33.8% in the two companies. Next in line is Python at 24%, Groovy and R both at 21.8%. Ruby is used by 18%, and a group of PHP developers in SPT brings PHP into the top 10 with 6%.

What kind of developers are we?

The most popular self-description is “a backend developer” (124), followed closely by the “full stack developer” (82). There are slightly more devops people (29) than “frontend developers” (25). Of course, the most interesting fact about this question, is all the various roles I failed to think of. Over 40 other descriptions were entered in the “Other” box, some of them even by more than one person! Most notably the 10 or 11 data scientists (depending on how you interpret descriptions). Mobile developer or App developer is another one that should have been one of the choices.

The languages we know

Only 13 people do not know any Java at all, the remaining 296 respondents knows enough to tick the box. This makes Java the most known language, but it is closely followed by C and JavaScript. 268 respondents know C, while 263 know JavaScript. If ever we wanted to use bashttpd for anything, hardly anyone would need to be retrained, with 234 people already knowning enough bash to get around.

PHP and Python follow, at 212 and 204 respectively. Personally I’m pleased to see so many knowing Python, and can only blame childhood mistakes for so many people knowing PHP. Dropping below 200 respondents we find R at 188 and C++ at 185. Ruby and Perl close out the top 10 languages, with 164 and 146. Hopefully all these people have learned Perl as a defence mechanism: Know thy enemy and all that…

Of the more hip languages on the JVM, Scala comes out on top, with 117 people claiming to know it, while Groovy at 94 and Clojure at 53 haven’t convinced quite as many people that they are useful to know. Clojure isn’t even in the top 20.

Outside the JVM, the top three contenders are all in the Microsoft family: C# (106), Basic (104) and Visual Basic (103). From far out in left field comes AWK at 98 respondents, while Pascal, the old favourite of so many, are known by only 86 people. Go has a sizeable following, at 81, narrowly beating Objective-C at 77. To close out the top 20, Haskell lays claim to the twentieth spot, with 61 people knowing what the heck Monads are.

Moving into the twenties, CoffeeScript ties with Clojure and Prolog at 53, while Common Lisp has managed to stick in the head of 47 people. Lua is known by 42, while Erlang and Scheme are tied at 31. Microsoft hasn’t quite managed to get their PowerShell into as many heads as Bash, but 30 people are willing to admit knowing it. Swift (29) and Tcl (27) close out the top 30.

Fortran (26) and PostScript (21) are starting to fall out of fashion, and we can point at 20 people working in various norwegian companies who probably got their education at the University of Oslo, starting with Simula as their first language. The story behind the single non-norwegian Simula-proficient person at Aftonbladet would be interesting to hear I imagine (or is it simply a norwegian who has moved to Sweden?).

Other notable mentions are 19 people who know Smalltalk, 11 people knows D, a whopping 10 people have managed to learn Brainfuck, while 9 people know Rust and F# (not the same 9 though). The tail is quite long, and old heroes like Forth (8), OCaml (8), Standard ML (4), Delphi (2), Modula2 (2) and Ada (1) are still hanging on. There’s even a handful of people who knows some dialect of assembler.

What can we use?

Let’s take a close look at which languages some of the larger companies (in terms of number of respondents) could and should use.

When starting their next project, SPT could choose Java, C, Python or JavaScript, and send less than a third of their people back to school. Allowing half the company to re-train, they could select from Scala, C++ or PHP, while Ruby, Perl, Groovy or C# would require sending close to two thirds of the company to school. Other popular choices are Go and Clojure, but that would require around 80% of the company to hit the books, so maybe not just yet.

Blocket is fortunate enough to have four languages known by all respondents: Java, C, Python and PHP. If they want to use JavaScript, C++ or Perl, they are still quite prepared, while Ruby and Go haven’t quite the same following. If they want to be on the JVM, but not use Java, Groovy is their only choice, but they would still need to train 11 out of 12 people.

At FINN, some people would be surprised to see that we are better prepared for a switch to C or Ruby than we are prepared for a switch to Scala. With 48 people knowing Scala, and 42 knowing Groovy, any of those two would be doable. We’d be stuck quite quickly if we go for Go, with only 16 of 80 knowing anything about it, and Clojure isn’t doing much better, at 26 out of 80. Half the company already knows C++, Python or PHP, so we could use those, while Perl, Pascal and Basic requires training more than half the company. Last year we had both a Clojure course (again) and a Haskell course for those interested, and 26 and 24 people managed to pick those up enough to click the right boxes.

Le Bon Coin have a story similar to Blocket. They can hardly go wrong, with most of the company knowlegeable in Java, JavaScript, PHP, C and Python. A few don’t know Ruby or C++, while Go and Perl are only known by half the respondents. Groovy is the top alternate JVM language, but with only 2 out of 13, it’s not really a good choice. Scala and Clojure aren’t known at all, so functional programming on the JVM clearly isn’t a “thing” at LBC.

Schibsted Tech Polska delivers to a number of other companies, so they need to know a bit of everything. This is reflected in their knowledge, with Java, C and JavaScript being known by almost all. Python and C++ are known by half the company, while a third of the company is able to hold their own in Groovy, Scala, Clojure, Ruby and PHP. a few of them even know Perl.

Willhaben in Austria also know a few languages, with Java, C, JavaScript and C++ being known by almost all. PHP and Python is known by two thirds of the company, with Perl and Ruby edging onto the “known by half” list. Groovy, Scala just missed the 50% mark, along with Haskell.

SQL and its many variations

When talking about databases, it’s easy to think SQL is just “one thing”, and if you know SQL, you can work with any database. And it’s somewhat true, but each database has it’s idiosyncrasies. This becomes especially clear when writing stored procedures, where basic SQL is lacking in control structures. So which database should we be using, really?

PL/pgSQL, the dialect used in PostgreSQL, is known by 193 respondents, so PostgreSQL seems like a good choice. SQL/PSM, which is used by MySQL and a few others, is known by 159 people, is a good second choice.

Of the large commercial databases, Oracle is clear out in front, known by 143 people, while runner up T-SQL used by both Microsoft SQL Server and Sybase is at 106. Any other contenders in the fight can just pack it in and go to bed. Fifth place goes to SQL PL, used by IBM DB2, known by only 32 respondents!

PSQL, used in Interbase and Firebird, makes a surprise showing at 18, while PL/PSM, the less used dialect supported by PostgreSQL has 17 users. Of the remainders only Watcom-SQL, used in some versions of Sybase, manages double digits, at 10 respondents.

So, how good are we at what we do?

The results here are somewhat skewed by the fact that people will naturally be more skilled in the languages they use regulary, and the companies with the most number of respondents use only a few of the languages regulary. As a result, there are more Java-experts than any other kind of experts. 102 respondents consider themselves Java-experts. Java is also the only language where there are more experts than any of the other “ranks”.

Other languages we feel we know well are JavaScript and PHP. Just under 50 people consider themselves experts in those languages. Next is Python, with 22 experts. Objective-C and Scala have 11 experts each, and there are 7 Groovy-experts and 6 Ruby-experts in all of Schibsted.

Languages we don’t really know are C#, Clojure, CoffeeScript, Go, Groovy, Objective-C, Scala and Swift. In fact, Clojure is also the only language where nobody considers them selves experts.

Most people consider themselves above average when it comes to JavaScript (205 respondents scored 3 or higher), while only 15 don’t know it at all. Similary, 255 people consider themselves above average in Java, and only 13 don’t know it at all. Of the three most used languages, Bash is the only one where the skills are concentrated below average. 223 people scored themselves at 3 or less for Bash knowledge, of which 18 don’t know it at all. At the other end, a single person considers themselves a Bash-expert.

The numbers are perhaps more useful if we separate out the people who don’t use a language in their current work, and then look at the distribution of skills in each group.

I’ll pick a few interesting ones, graphs for all are available at the end.

Java is almost symetrical among the “amateurs”. Most amateurs are about average, but of the rest, less and more knowledge is more or less equally distributed. This is in sharp contrast to the “pros”, where a large majority consider themselves experts. While it’s easy to think that in the group of “pros”, it will always be many experts, Java, Objective-C and PHP are the only languages where the experts are the largest group.

JavaScript is a similar story to Java, except there are fewer experts in JavaScript. It’s good to notice that everyone who works with JavaScript, actually do know the language, which was not the case with Java.

Apart from Java and JavaScript, C and PHP are the only languages where a significant amount of amateurs consider themselves above average in skills. At the other end, Clojure, CoffeeScript, Go, Groovy, Objective-C, Scala and Swift are all examples of languages where a large portion of the amateurs consider themselves to either not know it at all, or know less than average.

When comparing pro and amateurs, Objective-C really stands out. Most of the pros are experts, but among the amateurs it’s virtually an unknown language.

What next?

The most popular choice for the next project is Java. 55 people would go with Java if allowed to choose freely. 43 people prefer JavaScript. Go, Scala and Python are in the next group at 37, 36 and 31. 17 people would select Ruby, 12 would go with Swift or PHP, 11 thinks Clojure is a good choice, and 10 people would go with C.

In this kind of survey, there are always some smartypants. Five people avoided the question with variations of “I’m not able to conceive of a project where the language I like the most is the perfect choice, so I’ll say it depends on some hypotetical project specification that I have no say over”. Other clever answers are Fortran, Visual Fox, and T-SQL. Some people prefer the simple over the complicated, and would go with languages usually reserved for small utility-scripts like AWK and Bash.

Self-improvement

73.6% of us will sit down and learn a new language just to learn something new. 65.1% will accept learning something new if they need it for work, while only 41.4% are willing to do the same for non-work.

If we are so easy to motivate, what holds us back?

It could be the number of languages we think is “neccessary” to know. Knowing more than 6 languages is not neccessary in most peoples view, only 9.9% of us thinks it’s worth knowing 7 or more languages. 29% of us think 5 languages is good enough, while 6.8% thinks knowing one language well enough is all you need to be a good programmer.

The majority seems to think between 3 and 5 languages should be our target. 70% think knowing 3, 4 or 5 languages defines a good programmer.

Nobody thinks you’re any good if you know 9 languages, so if that’s you, you should set out to learn atleast one more as soon as possible. Or just forget one. :-D

There’s probably more here

I initially said I would blog about the results a couple weeks after the end of the survey. That ended up being a couple months instead. For that reason I have skipped looking at a few questions that could be interesting, but time consuming to look at.

How many languages do you know vs. How many languages does a good programmer know
Which language currently not in use in a company, is most popular in the company?
Does experience correlate with number of languages?
What else does experience correlate with, if anything?
Skill level in your favorite language

I’m sure there are a number of other interesting things to look at, so I’m hoping someone else wants to take a look at the results and see what they find. The link is below. The first sheet is the raw, unedited responses. The second sheet is where I have tried to manually clean up some of the answers to get more sense out of the results. And all the other sheets are various views and tables I’ve used to make this post.

Let me know if you blog about these results, and I’ll add a link here. Or just post in the comments below.

Javascript: from callback hell to heaven

2015-10-16T14:00:00+00:00

This blogpost explains how some nifty features in ES7 will make it easier to write asynchronous code, and how ES6 generators will pave the way for this.

Async/await

My main annoyance with javascript and Node has been the tedious asynchronous programming model of callbacks, leading to nested callbacks and the so called “callback hell” or “pyramid of doom”. Take for example the following code where we are creating a function that reads a file and parses it to JSON, and see how cumbersome it is to read and follow the code:

const fs = require("fs");

function readJSONFile(callback) {
    fs.readFile("file.json", "utf8", (err, data) => {
        if (err) return callback(err);
        
        let json;
        let parseError;
        
        try {
            json = JSON.parse(data);
        } catch(e) {
            parseError = e;
        }
        
        callback(parseError, json);
    });
};

readJSONFile((err, json) => {
    if(err) {
        //handle error
    }
    console.log(json);
    //continue program
});

This was slightly improved and simplified with ES6 promises, however, as we can see in the following code, we are still stuck with nested .then-calls:

const bluebird = require("bluebird");
const fs = bluebird.promisifyAll(require("fs"));

function readJSONFile() {
    return fs.readFileAsync("file.json", "utf8")
        .then(data => {
            return JSON.parse(data);
        })
};

readJSONFile()
    .then(json => {
        console.log(json);
        //continue program
    })
    .catch(err => {
        //handle error
    })

Callbacks and nested .then-statements will be a thing of the past with the upcoming version of Javascript (ES7). It will provide us with the keywords async and await, which will enable us to write asynchronous code as we would have written it in an imperative synchronous way:

const bluebird = require("bluebird");
const fs = bluebird.promisifyAll(require("fs"));

async function readJSONFile() {
    const data = await fs.readFileAsync("file.json", "utf8");
    return JSON.parse(data);
}

async function() {
    try {
        const json = await readJSONFile();
        console.log(json);
        //continue program
    } catch(err) {
        //handle error
    }
};

The await keyword can either take a promise or an async function, and pauses until the promise resolves, or the async function returns. If the promise rejects or the async function throws an error, the error can be caught with a normal catch-statement. This provides a significant improvement in readability, which likely leads to less bugs and a pleasure to write.

Generators

Along with promises, ES6 brought generators, which allows a function to pause at a certain point (yield) and continue when requested with .next(), as illustrated in the following code:

function* hello(name) {
    console.log(name);
    const age = yield;
    console.log(age);
}

const helloGen = hello("Tor");
helloGen.next(); // => Tor
helloGen.next(37); // => 37

Generators are not very useful for normal application logic, but they provide the buildings blocks for some nifty features.

First of all, ES7 async/await can be transpiled (by Babel) to generators, so that javascript environments that supports generators (Chrome, Firefox and Node), also supports async/await.

However, because Safari and Internet Explorer don’t support generators yet, it is to early to start using generators on the front-end, however, on the backend with Node, it can be safely used, because async/await keywords are transpiled to generators.

A web framework in Node that is based around generators, and uses it for all it is worth, is Koa. They utilize generators to make the code behave similar to async/await:

const Koa = require("koa");
const app = Koa();
const router = require("koa-router")();

function* getName() {
    return Promise.resolve("Tor");
}

app.use(router.routes());

router.get("/hello", function* () {
    const name = yield getName();
    this.body = "Hello " + name;
});

app.listen(3000);

The yield keyword behaves as await, and generator function* behaves as async functions. The main motivation for this is to simplify writing middlewares, and avoid the mentioned callback hell. In the next version of Koa, the yield keyword and generator functions will be replaced with await and async.

Conclusion

So, ES7 provides a fantastic improvement in simplifying asynchronous code that makes code easier to read and write. These features can already be used on the backend with Node and web frameworks like Koa.

Credits

Thanks to Sveinung Røsaker for improving the callback example.

Setup nginx with HTTP/2 for local development (OS X)

2015-09-25T21:57:00+00:00

HTTP/2 became an official standard in May earlier this year, and support is starting to land in servers already. The most recent, and very welcome addition, is nginx 1.9.5 [1]. What makes nginx extra awesome is that it’s very easy to set up in front of any other HTTP 1.x server (or HTTP/2 for that matter). Server push won’t work just yet, but at least we can start to test how multiplexing works. Multiplexing is said to eliminate the need to concatenate resources into bundles. At FINN.no we do multiple releases a day, and it just feels wrong that users have to download the whole 100+ kB JavaScript bundle after every release, even though most of the time only a few lines have changed. If we don’t need to bundle anymore, the users only need to download the few scripts that had changed since their last visit!

HTTPS over TLS 1.2 is a requirement to use HTTP/2. Service Workers also require secure connections, and probably other features soon.

This guide will help you set up nginx for local development on OS X, with proxy passing requests to your local server on port 8080 (or whichever port you prefer). In plain English, that means we put nginx in between your browser and your local development server. Your browser communicates securely over HTTP/2 to nginx, and nginx forwards the requests to your local server over unsecured HTTP/1.1.

Installing nginx

You have to compile nginx with the --with-http_v2_module configuration parameter, but Homebrew makes that a breeze. It’s one of my favorite tools on OS X.

If you don’t have Homebrew, see Install Homebrew further down.

To compile and install nginx with http2:

$ brew update   # update list of packages
$ brew install --with-http2 nginx

Test that the install works (make sure port 8080 is available):

$ sudo nginx
$ open http://localhost:8080/

You should see a page with the title “Welcome to nginx!”

# stop nginx
$ sudo nginx -s stop

Now it’s time to set up the https server with HTTP/2 enabled. Open /usr/local/etc/nginx/nginx.conf in an editor, and comment out the existing server section. Then copy-paste in the config below instead^[2]. If your local server runs on a different port than 8080, you can change it in the proxy_pass URL.

server {
    listen                     443 ssl http2;
    server_name                localhost;

    ssl                        on;
    ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
    ssl_certificate            cert.pem;
    ssl_certificate_key        cert.key;

    location / {
        proxy_pass          http://localhost:8080;
        proxy_set_header    Host      $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-HTTPS   'True';
    }
}

To use https we need to generate a self signed certificate. It will give you a warning in the browser, but it works fine for local development. This command will generate the certificate ^[3]:

$ cd /usr/local/etc/nginx/
$ sudo openssl req -x509 -sha256 -newkey rsa:2048 -keyout cert.key -out cert.pem \
   -days 1024 -nodes -subj '/CN=local.finn.no'

When asked for «Common Name», fill in the hostname you use locally. Most FINN.no developers use local.finn.no

To set up nginx to start automatically on boot, Homebrew Services can set it up very easily.

# install Homebrew Services
$ brew tap homebrew/services

It is required to run nginx as root to open ports under 1024, and we want it to run on port 443. Homebrew Services will set it up correctly just by using sudo.

Start nginx (and install to /Library/LaunchDaemons):

$ sudo brew services start nginx

Homebrew Services can also be used to stop and restart nginx

$ sudo brew services stop nginx
$ sudo brew services restart nginx

Now you should be able to open https://localhost/ and nginx should proxy pass to your local development server.

If you get a certificate warning, then it is working. It is only because you’re using the self signed certificate and not one signed by a certificate authority. Most browsers allow you to ignore the warning.

By adding the self signed certificate as a trusted certificate in System Keychain, we’ll get the green lock icon ^[4]:

$ sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /usr/local/etc/nginx/cert.pem

If it doesn’t work, see the Troubleshooting section further down.

To see that you really are using HTTP/2 in Chrome, you have to open the Network tab in Developer Tools. Right click (or CTRL-click) the column heading above the network requests, then make sure Protocol is checked.

Then you should see HTTP/2 traffic show up as h2 in the protocol column.

Congratulations, you now have HTTP/2 and HTTPS working!

Thanks to Sveinung Røsaker, Rune Halvorsen, Tor Arne Kvaløy, Frode Risøy and Martin Solli for feedback and tips

Update Oct. 1, 2015: How to make the self-signed certificate trusted. Replaced custom aliases with brew services and added tips for common problems. Update Jun. 20, 2016: Updated brew install command with the new nginx options.

Credit:

Extra

Install Homebrew

Run this one-liner in Terminal to install Homebrew:

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# if you haven't installed Command Line Developer Tools from Apple already
$ xcode-select --install

Troubleshooting

Connection refused

A “Connection refused” error probably means that nginx is not running correctly.

Check that the ports in nginx.conf is not already in use
Check that you run nginx as root (sudo)
Check the error.log: /usr/local/var/log/nginx/error.log

502 Bad Gateway

If you get a 502 error, nginx is running, but nginx is not able to connect to your development server. Check that your development server is running, and that you have the correct port in nginx.conf. Copy the value of proxy_pass and try to open it in your browser. For the config above, that would be http://localhost:8080.

The Process of Using Kafka

2015-09-22T13:58:41+00:00

Use and misuse

Actually, the title is a bit too broad. We’ll look at how different number of acks, async and timeouts will affect the reliability of your messages.

Abstract / TL;DR

Kafka can be used in many ways, and there are also many options that you need to understand and decide upon before you start using Kafka. As always, the combination of options you choose is a compromise - a trade off - and in my opinion, explicit trade offs are much better than accidential ones.

We have seen many ways of using Kafka internally, and to demonstrate the properties of some of the different permutations, we’ve run a small test to demonstrate how it works.

Before using Kafka, you need to understand it! Read the documentation, the confluent.io blog, and Martin Kleppman’s blog.

Configuration Details

For a topic, you’ll have to choose a replication factor and number of partitions. In this small exercise will fix the replication factor at 3, we’ll split each topic in 4 partitions.

You can either produce to a topic synchronously or asynchronously. In the first case, the message is sent on the same thread as your producer code. In the async case, the kafka producer library will put your request in an internal queue, and unless that queue is full, you’ll get control back to your main thread immediately.

In the synchronous case you have to choose how many servers need to acknowledge the message before your thread can continue. The fastest case is to not wait for any acknowledgement. That means, your thread continues immediately after the message is sent on the network to leader. If the network goes down - your message is lost. For a higher level of reliability you can wait for acknowledgement from the partition leader, and finally, you can wait for acknowledgement from all in-sync-replicas.

The main trade off is thus between reliability - the possibility of losing data - and (you guessed it) speed. That is a business decision; is it acceptable to lose some data?

These simple demonstrations were run on a laptop connected to our dev-cluster, consisting of 3 brokers. The response times we got are as expected - i.e matches the intuitive expectation, and clearly demonstrates the trade offs.

The code connects to Kafka, does timing of the sending of 8 events, and then sends a big chunk of events. The various combinations of options in this exercise demonstrates actual patterns of code we have observed. Some of the combinations are NON-SAFE. Beware!

Timing information

For various combinations of producer type (sync/async), request.required.acks and ways of exiting the program, we’ve run the small test program included at the end of this post.

Run 1.1 - synchronous producer, wait for all in sync replicas to acknowledge the write. This is the most secure way of producing messages.

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
100%  send 10000 kafka events
000%  closing

Run 1.2 - synchronous producer, wait for the leader to acknowledge the write. If the leader crashes after acknowledging, but before any replica receives the data, the message will be lost.

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
100%  send 10000 kafka events
000%  closing

Run 1.3 - synchronous producer, but do not wait for acknowledgement.

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
099%  send 10000 kafka events
000%  closing

Let’s switch to the asynchronous producer. The producing thread will continue, but we’ll need to be aware of the need to drain any queues before exiting. Messages will be batched together within a 5 s window, so just exiting can be a serious problem if you care about your data.

Run 2.1 - Async, blocking on producer.close() before exiting:

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
005%  send 10000 kafka events
095%  closing

100% of all messages were delivered.

Run 2.2 - Async, immediate System.exit()

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
100%  send 10000 kafka events

0% of the messages were delivered!

Run 2.3 - Async, main method returns normally, without calling close()

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
100%  send 10000 kafka events

The process does not terminate, as there is a non-daemon producer thread left. The default value of queue.buffering.max.ms is 5000 ms, and queue.buffering.max.messages is 10.000 messages. This means that the queue will be sent after 10.000 messages, or after waiting at most 5000 ms.

Run 2.4 - Async, sleep 8 seconds after sending, then System.exit()

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
002%  send 10000 kafka events
098%  sleeping

100% of the messages were delivered

As 8 s is more than 5000 ms, the messages will be delivered.

Run 2.5 - Async, sleep 4 seconds after sending, then System.exit()

-----------------------------------------
ms     %     Task name
-----------------------------------------
000%  send 1 kafka event - 0
000%  send 1 kafka event - 1
000%  send 1 kafka event - 2
000%  send 1 kafka event - 3
000%  send 1 kafka event - 4
000%  send 1 kafka event - 5
000%  send 1 kafka event - 6
000%  send 1 kafka event - 7
003%  send 10000 kafka events
097%  sleeping

99.89% (9989/10000) of the messages were delivered.

As 4 s is less than 5000 ms, the first 10.000 messages will be delivered, and the rest will be left in the queue. As we’ve sent more than 10.000 message during the session, some messages were lost.

Summary

How you configure your Kafka producer will determine whether you have a super-fast-and-lose-everything messaging system, or something slower but reliable. YMMV.

Before using Kafka, you need to understand it! Read the documentation, the confluent.io blog, and Martin Kleppman’s blog. Hermes is a message broker built on top of Kafka.

Milliseconds / event	What
8.2 ms	Sync, wait for all in sync replicas
3.9 ms	Sync, wait for leader
2.0 ms	Sync, don’t wait for ack
0.13 ms	Async, wait for the queue to finish
0.02 ms	Async, exit without waiting (aka /dev/null for the last messages)

The sample code

package no.finntech.demo.ecclient;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.Date;
import java.util.Properties;
import java.util.Set;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.util.StopWatch;

public final class CleanBulkSenderDemo {
    interface Acks {
        int ALL_ISR = -1;
        int LEADER = 1;
        int NEVER_WAIT = 0;
    }

    public static void main(String[] args) throws InterruptedException, UnknownHostException {

        boolean featureAsync = true;
        int featureSleep = 0;
        boolean featureClose = false;
        boolean featureSystemExit = true;

        String brokers = "super-dev1.finntech.no:1337,super-dev2.finntech.no:1337,super-dev3.finntech.no:1337" ;

        final Properties props = new Properties();
        props.put("metadata.broker.list", brokers);
        props.put("producer.type", featureAsync ? "async" : "sync");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
        props.put("request.required.acks", "" + Acks.ALL_ISR);
        props.put("client.id", InetAddress.getLocalHost().getHostName());
        Producer producer = new Producer(new ProducerConfig(props) ) ;

        // connect, warm up
        sendEvent(producer, "42", "42_0");
        sendEvent(producer, "42", "42_1");

        System.out.println("Starting");

        StopWatch sw = new StopWatch("kafka");
        for (int i = 0; i <8; i++) {
            sw.start("send 1 kafka event - " + i);
            sendEvent(producer, "42", "42_" + i);
            sw.stop();
        }

        int n = 10000;
        sw.start("send " + n + " kafka events");
        for (int i = 0; i <n; i++) {
            sendEvent(producer, "42", "x2_" + i);
        }
        sw.stop();

        sw.start("closing");
        if (featureClose) producer.close();
        if (featureSleep>0) Thread.sleep(featureSleep);
        sw.stop();

        System.out.println(sw.prettyPrint());
        if (featureSystemExit) { System.exit(0); }
    }

    private static void sendEvent(Producer producer, String s, String s1) {
        producer.send(new KeyedMessage<String, String>("Test.ignore.performance", s, s1+ " - " + new Date()));
    }


}

Browser statistics June 2015

2015-06-25T23:00:32+00:00

In July 2013 more than 40% of the visits to FINN.no were from a tablet or a phone. Nearly one year later, in April 2014 - the share was 48%.

What do you think has happened the last year?

In April, Google rolled out their “mobile-friendly update”, which boosts the ranking of sites that are mobile friendly in the Google search. We’re clearly not the only one seeing the increased market share of mobile devices.

How many visitors use a desktop or laptop?

Our first graph shows the share of our users using a mobile phone, tablet or a desktop computer to access FINN.no, regardless of whether they use the traditional somewhat-responsive-desktop-web (www.finn.no), our responsive mobile web (m.finn.no) or an Android or iPhone app. 62% of our visits are now from a smartphone or a tablet. The traditional desktop/laptop has a market share of 38%.

Which FINN.no application do people use?

We have three major versions of FINN.no - the desktop, the responsive m.finn.no, and native apps. Our reponsive site is now the biggest! 14% of our visits are from our native apps, 42% from www.finn.no, and 44% from our responsive m.finn.no.

Browsers

First of all, let’s take a look at the numbers of the browser vendors. The ranking is clear, Apple is the biggest, ahead of Google, Microsoft and Mozilla - and Opera is still in the top 5, representing more than an amazing 0.2% of our visits.

Do you remember the “browser wars”? When IE 8.0 ruled the web? In case you wondered which browser is the biggest on the “desktop” here’s the trend:

The complete browser statistics, across all applications are as follows:

It seems like a good idea to make sure your website works well with Safari!

In case you still have Internet Explorer 9 as your favourite, it’s still making it to the top 15, with a total share of 1.2% of our traffic, marginally behind Internet Explorer 10 with 1.4%. In 2015, poor IE8 is down to 0.3%.

Thanks to Tobias Flatin, Chris Searle, Gregers G Rygg, Trond Hjorteland, and others for feedback.

When flying from Oslo to London, you don't want to go via Moscow

2015-06-15T12:47:32+00:00

If you want to travel from Oslo to London, you may end up getting results which include layovers in far-away places. Do you really want to spend a night at the airport in Moscow if you only want to go from Oslo to London? Probably not, but nonetheless, some of the results we receive at the FINN Flight Search are those kind of travels. Those travels may be really cheap, ending up at the top of our result list as we sort by price, but they are mostly garbage and we do not want our users to be confused by those outliers. We want them to go away!

The old solution

We have tried to fix this before. With limited success. A solution we used a couple of years back was something like this:

x = 3 * Average of the three shortest flights

We removed results with duration > x. This works fine on short travels, like a two-hour flight from Oslo to London. With the given formula, all trips longer than 6 hours to London will be removed. That’s ok.

However, it doesn’t work too well on longer flights. If you want to go from Oslo to Bangkok, you would probably spend at least 11 hours in the air. In that case, it would be too conservative to just remove the flights that are 33 hours or more.

A theoretical solution

The correct way to remove the outliers is by finding the quartiles. When the quartiles are found, the upper fence is defined by

Upper fence = Q1 + 1.5 * IQR

Where IQR is the Interquartile Range; Q3 - Q1.

We are using Solr at FINN, but the stats query in Solr does not give us the quartiles out of the box, so we would need to do a separate query to calculate the quartiles. Should be easy enough.

What is too long?

So what is a too long flight? That’s not an easy question to answer. Racking our brains (and doing some guesswork), we came up with the following suggestions on what the upper fence on duration to different destinations should be. The durations are per leg:

OSL - LON 300 mins = 5 hrs. Given the minimum flight time OSL - LON = 115  
OSL - BKK (Bangkok) 1000 mins = 17 hrs. Minimum = 674  
OSL - KOA (Kona, Hawaii) 2000 mins = 33 hrs. Minimum = 1335

The minimum flight duration is clearly of interest to us. We store that away in Solr, and using the stats query from Solr on that field, we get the following data on a flight OSL-LON:

"stats_fields": {
      "minimumLegDuration": {
        "min": 115,
        "max": 815,
        "count": 3610,
        "missing": 36,
        "sum": 615655,
        "sumOfSquares": 123968775,
        "mean": 170.5415512465374,
        "stddev": 72.5080446083144,
        "facets": {}

A lot of good information there! We get the standard deviation, mean and min values. Missing the quartiles, though, but we can fix that!

And what is too short?

Sometimes, the supplier sends us incorrect data. That’s bad. But the good thing is that when the data is incorrect it is really incorrect. We sometimes receive flight times of 0 minutes. That is obviously wrong, and we filter out those. However, we have not seen any occurences of, say, 1 minute flight times. That would be wrong as well, but more difficult to handle. So we close our eyes and don’t care about those.

But hang on…

So, we were ready to go! We wanted to fix this problem once and for all. But before we started coding, we talked to emeritus Harald Goldstein at the University of Oslo. He’s been teaching statistics at the university level for years and had some good feedback on our theoretical solution. First of all, he pointed out that the solution we have outlined is a good solution if we have a symmetrical distribution of the flight durations. The quartiles and the standard deviation works well on a symmetrical distribution like the normal distribution, but what about our data? Is it normal distributed? That was a relevant question, and as one of FINN’s main strategies is “data in our backbones”, we surely needed to have a closer look at this.

So here goes, flights from Oslo to London. Trip duration on the x-axis, number of occurences on the y-axis:

The shortest trip duration OSL - LON is 230 minutes in the graph above (115 min per leg) and there is even an outlier - which is not visible - at 2750 minutes! However, it is quite clear that this is not a normal distribution. It might look more like a gamma distribution for the part where x > 230.

Darn!

Back to square one

All the fancy statistics didn’t provide the solution, but it helped us really understand the problem. Obviously, we needed to re-think.

Logarithm

A problem with the solution from ages back (described at the beginning), was that it didn’t work very well on long hauls. The relation between the minimum trip duration OSL-BKK and OSL-LON is approx. 6:1. Alas, the upper fence must be relatively lower on longer trips. After some more brain-racking, we came up with the idea that we should look at this logarithmically.

A first suggestion

log(min BKK)/log(min LON) = log(675)/log(115) = 2,83/2,06

However, the relation is too far away from 6:1 and we are not able to compensate for long hauls.

Square root

Finding quartiles doesn’t work, using a logarithmic approach doesn’t work very well. So what about the square root? That looks a bit like a logarithmic function.

Let’s try

sqrt(min BKK)/sqrt(min LON) = sqrt(675)/sqrt(115) = 26/11

This is much closer to 6:1 than the logarithmic approach. The relation is relatively more correct for both short- and long hauls.

The final solution

After some trial and error, we finally ended up with the simple formula of

x = min + A * sqrt(min)

Where x is the upper fence, min is the minimum duration for that leg and A is a tunable constant.

This formula seems to remove most of the unwanted outliers. Some examples (from our first iteration, with the constant A=25)

SVG-BGO. min = 70   => x = 279  (4.5 hrs)
OSL-LON. min = 230  => x = 609  (10 hrs)
OSL-BKK. min = 1350 => x = 2269 (37 hrs)
OSL-KOA. min = 2671 => x = 3963 (66 hrs)
OSL-NYC. min = 920  => x = 1678 (28 hrs)

That seems to work quite well. In the SVG (Stavanger) - BGO (Bergen) case, the formula removes approx. 30% of the offers. Some of them with a layover in Oslo.

No need to hesitate then, this simple formula is now in our production systems. If you want to go from OSL to LON, you will get a slider which is preset with an upper fence of 5hrs 15min.

You still have the possibility to slide it to the max, though. All the way up to 27hrs 30min. The result list will then show you the unfiltered result set. But nobody wants to use 27hrs 30min on a flight from OSL to LON, right?

Conclusion

The simplest explanation is usually the correct one.

Apache Cassandra in a Microservices Enterprise Platform

2015-04-28T14:00:00+00:00

In this article we’ll explore how Apache Cassandra, the world’s most popular wide column store and 8th most popular database overall, will only grow as a cornerstone technology in a microservices platform. With a little theory to microservices, to some examples of microservices and underlying required infrastructure, we’ll show that any solution both capable of scaling and dealing with time-series data-models is going to need to depend upon Apache Cassandra as a persistence layer, despite having a polyglot persistence model at large.

microservices

Microservices is a term that’s come out of ThoughtWorks’ Martin Fowler and James Lewis. It’s a bit of a buzzword, basically a fresh revival of the parts of service orientated architecture that you should be focusing on and getting right. A lot of it hopefully is obvious to you already. If you’ve been doing service orientated architecture or even generally just unix programming properly over the years it might well be frustrating just how buzz “microservices” has become. But it’s worth keeping in mind how much garbage we’ve collected and how many aspects of service orientated architecture that we’ve gotten badly wrong over the years. Younger programmers certainly deserve the clarity that ThoughtWorks is giving us here.

Microservices, following the tips and guidelines from Sam Newman, can basically be broken down into four groups.

interfaces

Ensure that you standardise the systems architecture at large and especially the gaps or what we know as the APIs between services. Standardise upon practices and protocols that minimise coupling. Move from tightly coupled systems with many compile time dependencies and distributed published client libraries, to clearly defined and isolated runtime APIs. Take advantage of REST, especially level 3 in richardson’s maturity model, for the synchronous domain driven designed parts of your system. When it comes to event driven design use producer defined schemas, like that offered by Apache Thrift’s IDL which gives you embedded schemas for good forward and backward compatibility along with isolated APIs that prevent transitive dependencies creeping through your platform. Getting this right also means that within services teams get a lot more freedom and autonomy to implement as they like, which in turns diminishes the effects of Brook’s law.

deployment

Simplify deployment down to having just one way of deploying artifacts, of any type, to any environment. The process of deployment needs to be so easy that deploying continuously each and every change into production becomes standard practice. This often also requires some organisational and practical changes like moving to stable master codebases and getting developers comfortable with working with branches and dark launching.

monitoring

It isn’t just about the motto of “monitor everything” and to have all metrics and logs accessible in one central place, but to include synthetic requests to provide monitoring alerts that catch critical errors immediately, and to use correlation IDs to be able to easily put all the moving parts together for any one specific request.

architectural safety

Addressing the fallacies of distributed computing, ensure that services are as available as possible and consumers handle failures gracefully by using such mechanisms as circuit breakers, load balancing, and bulkheads.

If you want more than I can only highly recommend Sam Newman’s just published book on Building Microservices.

In this article, and when talking about microservices, i’m most interested in how Cassandra, one of the most popular databases in our industry and the database most realistic to the practical realities of distributed computing, comes into its own. Here Cassandra is relevant to the monitoring and architectural safety aspects of microservices, from looking at how monitoring is typically time series data, a known strength for Cassandra, and looking into how modern distributed systems should be put together.

turning the database inside out

Having worked in the enterprise for over a decade it’s clear that the relational database is at juxtaposition to the rest of our industry. The way we code against the RDMS, layer domain logic upon it, and at various layers up through the stack add additional complexity and cyclic dependencies with caches that require invalidation, makes it all too obvious we’ve been doing things wrong. You can see it in plain sight when watching presentations where people show their wonderful service orientated architectures or microservices platforms, and despite talking about how their services scale, are fault-tolerant and resilent, maybe even throwing in some fancy messaging system, there is still in the corner of their diagrams the magic unicorn – the relational database. Amidst all the promotion and praise for BASE architectures, the acceptance that our own services and those infrastructural like Solr, Elastic Search, or Kafka, need to work with eventual consistency so to achieve performance and availability, the relational database somehow gets a free ride, an exception, to all this common sense.

Martin Kleppman presented at Strange Loop last year and afterwards wrote an article “Turning the database inside out with Apache Sanza” that properly hits the nail on the head, perfectly describing my own woes around why the relational database has ruined back-end programming for us. And he sums it up rather elegantly to that the replication mechanism to databases needs to come out and become its own integral and accepted component to our systems designs. And this externalised replication is what we call streams and event driven design, and it leads us to de-normalised datasets and more time-series data models.

product examples

Let’s look at a few examples from FINN.no and see how these things work in practice.

First of all we know that Cassandra has a number of known strengths over other databases, from dealing with large volumes of data and providing superior performance on both write and read performance, to time series data and time-to-live data.

But one of the areas that’s not highlighted enough is that Cassandra’s CQL schema often provides a simpler schema over the SQL equivalent, something that’s easier to work with and for programmers today something that uses more natural types and fluent APIs.

After all the whole point with your microservices platform is that you’re writing smaller and smaller services, and those smaller services each come with their own private data models. As services and their schemas get smaller and simpler we find that we don’t need relationships and constraints and all the other complexities that the RDMS has to offer. Rather the constructs available to us in CQL are superior, and faster.

The first example is how we store the users search history. This shouldn’t be a product that needs to be explained to anyone. The CQL schema to this is incredibly simple and it takes advantage of the combination between partition and clustering keys. And it operates fast, just make sure to apply the “CLUSTERING ORDER BY” or you’ll be falling into a Cassandra anti-pattern where you’ll be left reading tons of tombstones each read.

CREATE TABLE users_search_history (
  user_id      text,
  search_id    timeuuid,
  search_url   text,
  description  text,
  PRIMARY KEY  (user_id, search_id)
)
WITH CLUSTERING ORDER BY (search_id desc);

Another example is fraud detection, and while fraud detection is typically a complicated bounded context at large, breaking it down you may find individual components using small simple isolated schemas. Here we have a CQL schema, much simpler than its relational SQL schema counterpart not only because is it time-series using the clustering key, but using Cassandra’s collection type to store the scores of each of the rules calculated during the fraud detection’s expert rules system.

CREATE TABLE ad_scores (
  adid        bigint,
  updated     timeuuid,
  rules       map<text, int>,
  PRIMARY KEY (adid, updated)
)

So it shouldn’t be of any surprise that Cassandra is going to hit the sweet spot for particular services in a number of different ways in any polyglot persistence platform. But bring it back to the bigger picture and we can look at how we can remove that magic unicorn we keep seeing in systems designs’ overviews.

brewer’s theorem

Looking at the CAP theorem you recognise that to build a BASE microservices platform it means building AP systems. When you look at Martin Kleppman’s message that the replication is its own concern in your BASE platform, when you look at domain driven design and how to focus keeping your services within clear bounded contexts and then taking it further to use event driven design to further break those bounded contexts apart, you see that it ties back to the CAP theorem and it is for the sake of scalability and performance and even just simplicity in design, a preference for availability over consistency. Looking into it deeper in how streaming solutions often still need to write to raw event stores, and similar to a event sourcing model when a service needs to bootstrap its de-normalised dataset from scratch from data beyond that to which is stored in the stream’s history, you can see there is a parallel to partition tolerance and how it, just like partition tolerance within the CAP theorem, is a hard fast requirement to any distributed architecture.

Here’s a simple example of a web application (named “xxx”) making three synchronous requests to underlying services in our platform when the user logs in. One service call to do the authentication, and the other two to fetch user data due to that user data being stored/available in different back-end systems.

It’s not difficult to see this isn’t a great design. First of all it’s keeping all the logic on how these services calls are initiated and how the data joined together high up in the presentation layer. It’s also not a great performer unless you’re willing to introduce concurrency code up in your presentation layer.

The obvious thing to do is introduce an aggregate service so that the web app only needs to make two inner requests and much of the logic, including any concurrency code, is pushed down into the platform and into the bounded context where it belongs. Another thing that typically happens here is that a cache, one that requires invalidation, is added into the aggregate service to address performance and availability.

But it’s a hack. Now you have more network traffic than before and more overall complexity, and just a poor and possibly very slow system of eventual consistency.

There is a better way, imagine there was but one service and all the data in a shared schema and then bring the replication mechanism of the database out into a stream to realise this. That is de-normalise the data from the auxiliary user-profile service back into the original user service. What you end up with is faster, more available, better scaling solution. Once you’re in the swing of event driven design and de-normalised datasets this is simpler solution too.

infrastructure examples

With some ideas of how Cassandra can become important for a successful microservices platform within product development let’s look into how Cassandra fits into the infastructure and operations side of things. A trap i suspect a lot of people are getting themselves into when starting off with microservices is that they haven’t got the infrastructure required in place first. Even if James Lewis and Sam Newman puts extra emphasis on the needs for deployment and monitoring tools it still can be all too easily overlooked just how demanding this really is. It’s not just about monitoring and logging everything and then making it available in a centralised place. It’s not just about having reproducible containers on an elastic platform, but about all the infrastructure tools and services being rock stable and equally elastic. You don’t want to be running a microservices platform and have crucial monitoring and logging tools fail on you, particularly in any crisis or in the middle of any critical operation.

When it comes to correlation IDs a brilliant tool out there is Zipkin from Twitter. Zipkin provides for you in all your applications and services this correlation ID, a unique ID for each user request, which you can for example put into your log4j thread context or MDC and then via a tool like Kibana be able to put together all the logs from across your whole platform for one specific user request. But Zipkin goes a lot further than this, based off Google’s Dapper paper, it provides for you distributed tracing or profiling of these individual requests.

Naturally Zipkin can be put together with Cassandra, the best fit as it’s perfect for large volumes of time series data. We also use scribe for the sending of the trace messages from all the jvms throughout our platform over to the zipkin collector which then stores them into Cassandra.

Below is the typical page in Zipkin. Under the list of services the first row is the user’s request, here we can see that it took 195ms. Then under that we can see when and how long all the individual service calls took place. We can see which back-end services are running properly in parallel and which service calls are sequential. Services like Solr, Elastic Search, the Kafka producers, and of course Cassandra, are all listed as well. Not only is this fantastic for keeping your platform tuned for performance but it’s a great tool for helping to figure out what’s going on with those slow requests you’ve got, for example in the top 5th percentile.

It’s also a great tool to help keep teams up to date with all the constantly evolving moving parts that exist in a microservices platform, something that’ll no doubt be outdated a week after any manual catalog documentation was written. This is especially useful for front end developers that usually haven’t the faintest idea what’s going on behind the scenes.

This visualisation can also be offered from within the browser, both Firefox and Chrome have plugins, so that developers can see what’s happening near real-time as they make requests.

Something that we’re added to Zipkin is a cascasding job that runs nightly in our hadoop yarn cluster, that aggregates all the different traces made during the day and builds up a graph of the platform showing which services are calling services. In this graph on the left hand side you will see our web and batch applications, then to the right of that the microservices moving down the stack the further to the right you go. Legacy databases with shared schemas end up as big honey pots on the very right while databases with properly isolated schemas appear as satellites to the services that own them. If you’re undertaking a move towards event driven design then you’ll see the connections between services and especially across bounded contexts break apart, and you should see those bounded contexts become more grouped neighbourhoods for themselves.

This cascading job that aggregates this data should now be available in the original Twitter Github repository, otherwise you’ll find it in FINN’s fork of it.

Another crucial infrastructure tool is Grafana, and the Graphite and StatsD stack underneath it. Grafana and Graphite is one of those must-have tools in your infrastructure, but the problem is it just doesn’t scale. Indeed the carbon and whisper components to graphite are dead in the water. We’ve hit this problem and looked into the alternatives, and there’s one interesting alternative out there based on Cassandra, which only makes sense as it’s another perfect match for time series data.

The plugin to Graphite is called Cyanite and very simply replaces all the carbon and whisper components. In an earlier version it was quite limited and you couldn’t for example get wildcarded paths in graphite working, but it now bundles with Elastic Search to give you a fully functional Graphite.

If you want to take a go at setting this up and see for yourself just how easy it is to get running, and how easily Grafana, Graphite, Cyanite, Elastic Search, and Cassandra, are configured together take a look at the GitHub repository docker-cyanite-grafana. It’s a docker image – just run build.sh and once everything has started up run test.sh to start feeding in dummy metrics and test away all the grafana features you’re used to working with.

a compliment to the modern enterprise platform

With this run through of just a few product and infrastructure examples it’s quickly obvious how useful Cassandra is to have established in any polyglot persistence model. And with an understanding on distributed computing it becomes even harder to deny Cassandra its natural home in the modern enterprise platform.

Finding Gold in Big Data

2015-04-27T09:00:00+00:00

In the previous article we introduced our own introduction into the world of Big Data, and explored what it meant for FINN. Here we’ll go into the technical depth about the implementation of our Big Data needs.

rehashing the previous article

FINN is a busy site, the busiest in Norway, and we display over 80 million ad pages each day. Back when it was around 50 million views per day, the old system responsible for collecting statistics was performing up to a thousand database writes per second during peak traffic. Like a lot of web applications we had a modern scalable presentation and logic tier based upon ten tomcat servers but just one not-so-scalable monster database sitting in the data tier. The procedure responsible for writing to the statistics table in the relational database was our biggest thorn. It had indeed gotten so bad that: during peak traffic; operations had to at the first sign of trouble turn off this database procedure – that is when users were getting the most traffic to their ads we had to stop collecting their statistics.

At this time we were also in the process of modularising the FINN web application. The time was right to turn our statistics system into something modern and modular. We wanted an asynchronous, fault-tolerance, linearly scaling, and durable solution.

the design

The new design uses the Command Query Separation pattern by using two separate modules: one for the collecting of events and one for displaying statistics. The event collecting system achieves asynchronousity, scalability, and durability by using Scribe. The backend persistence and statistics module achieves all goals by using Cassandra and Thrift. As an extension of the push-on-change model: the event collection stores denormalised data and it is later aggregated and normalised to the views the statistics module requires; we use MapReduce jobs within a Hadoop cluster.

the statistics module

At FINN all our modular architecture is built either upon REST or interfaces defined by Apache Thrift. We discourage direct database access from any front-end application. So our Statistics module simply exposes the aggregated read-optimised data out of cassandra, with a Thrift IDL like:

service AdViewingsService {
    /** returns total viewings for a given adId */
    long fetchTotal(1: i64 adId)
    /** returns a list of viewings for the intervals between startTimestamp and endTimestamp **/
    list<long> fetchRolled(1: i64 adId, 2: Interval interval, 3: i64 startTimestamp, 4: i64 endTimestamp)
}
enum Interval { YEAR, MONTH, DAY, HOUR, QUARTER_HOUR }

the event collection module

The collecting of events we wanted to happen asynchronously and in a fail-over safe manner so we chose a combination of thrift and Scribe from Facebook. Each event object, or bean, is a thrift defined object, and these are serialised using thrift into Base64 encoded strings and transported through the network via Scribe. The event collection module is nothing more than a Scribe sink and it dumps these Thrift event beans directly into Cassandra.

Each event is schemaless in its values field, it’s up to the application to decide what data to record, and is defined by Thrift like:

struct Event {
    /** different categories generally won't be mixed in the normalised views. */
    1: required string category;
    2: required string subcategory;
    3: required map<string,string> values;
}

For the persistence of events in Cassandra we store events in rows based on minute by minute buckets. The partition key goes a little further and looks like <minute-bucket>_<random-number>. The reason for the additional column random_number, named “partition” in the CQL (Cassandra Query Language) schema, is it ensures write and read load is distributed around the cluster at all times, rather than one node being a hotspot for any given minute. Then each event is stored within a clustering key “collected_timestamp”. The value columns are essentially the category, the subcategory, and the json map keys_and_values.

CREATE TABLE events (
  minute text,
  partition int
  type text,
  collected_timestamp timeuuid,
  collected_minute bigint,
  subcategory text,
  keys_and_values text,
  PRIMARY KEY ((minute, partition), type, collected_timestamp)
);
CREATE INDEX collected_minuteIndex ON events (collected_minute);

We have moved the category column in as the first clustering key since the aggregation jobs typically only scan one type of category at a time. In hindsight we might not have done this as it would be better to be able to use the DateTieredCompactionStrategy. Within this schema there are two timestamp that we have to work with, both the real_timestamp representing when the event happened and the collected_timestamp when the event got stored in Cassandra. Analytical jobs, like “how many bicycles were sold in Oslo on a specific day?”, are interested in the real_timestamp. While the incremental aggregation jobs are interested in aggregating just those events that have come into the system since the last incremental run.

the technologies

Cassandra is truly amazing and refreshingly modern database: linear scalability, decentralised, elastic, fault-tolerant, durable; with a rich datamodel that provides often superior approaches to joins, grouping, and ordering than traditional sql. The Scribe sink simply stores the Thrift event objects directly into Cassandra. After that we use Apache Hadoop to then in the background aggregate this denormalised data. The storing of denormalised data in this manner extends the push-on-change model, an approach far more scalable “in comparison with pull-on-demand model where data is stored normalized and combined by queries on demand – the classical relational approach”¹. In hadoop our aggregation jobs piece-wise over time scan over the denormalised data, normalising in this case by each ad’s unique identifier, this normalised summation for each ad is then added to a separate table in Cassandra which uses counter columns and is optimised for query performance.

Many of these incremental aggregation jobs could have been tackled with Spark streaming or Storm today. But 4 years ago when we started this project none of these technologies would have been capable of keeping up with our demands. Even today we’re happy that we have this system as our underlying base design, since there’s always aggregation jobs that can’t be solved with, or better solved with, streaming solutions. Having the raw events stored provides us a greater flexibility and a parallel to partition-tolerance when streaming solutions fail.

Painting a wonderful picture, hopefully you can see it puts us one foot into an entirely new world of opportunity, but it would be a lie not to say the journey here has also come with its fair share of pain and anguish. The four key technologies we adopted here: Cassandra, Hadoop, Scribe, and Thrift; involve changes in the way we code and design.

Cassandra, being a noSQL database, or “Not only SQL”, takes some investing in to understand how datamodels are designed to suit queries instead of writes. We also been bitten by an unfortunate bug or two along the way. The first was immediately after the 1.0 release when compression was introduced and before we had a CQL schema and were storing serialised thrift events spearately in individual rows. With the compression we jumped on it a little too quickly, also modifying its default settings for chunk_length_kb to suit out skinny rows for the denormalised data. This hit a bug leading to excessive memory usage bringing Cassandra nodes down. The Cassandra community was very quickly to the rescue and we were running again with hours. The second bug was again related to skinny rows. Each of our count objects were stored in individual rows resulting in a column family with billions of rows stored on each physical machine. This eventually blew up in our face and the fix was to disable bloom filters. It’s obvious to us now that skinny rows should be avoided, and you’ll see in the CQL schema presented above that it isn’t how we do it anymore.

Hadoop is a beast of a monster, and despite bringing functional programming to a world of its own (don’t go thinking you are anything special because you’re tinkering around with Scala) in many ways much of the joy Cassandra gave us was undermined by having to understand what the heck was going on sometimes in Hadoop. We were also using Hadoop in a slightly unusual way trying to run small incremental jobs on it with as high frequency as possible. In fact it could be that we have the fastest Hadoop cluster in the world with our set up of a purely volatile HDFS filesystem built solely with SSDs. We’ve also had plenty of headaches due to Hadoop’s centralised setup. Our Cassandra and Hadoop nodes co-existed on the same servers in the one cluster, providing data-locality – a key ingredient to good Hadoop performance. But these servers came with a faulty raid controller locking up the servers many times each day for over a year, it took HP over a year to “look” into the problem. For our Cassandra cluster it successfully proved how rock solid fault-tolerance it really has – just imagine the crisis to the company if your monster relational database was crashing from faulty hardware twice a day. But one of these faulty server was running the hadoop masters: namenode and jobtracker; so aggregation jobs were frequently crashing. Since then we’re moved these services to a small separate virtual server, together they use no more than 600mb memory.

Using Thrift throughout our modular platform we’d already broken the camel’s back on it. But Scribe has given us plenty of problems. Like most code open sourced from Facebook it seems to be “code thrown over the wall”. It contains traces of private Facebook code, has very little logging+documentation+support, and has settings that rarely work without peculiar and exact combinations with other settings. We believed scribe to be fault tolerant when it simply wasn’t and we lost many eventsc when resending buffers after a downtime or pause period. We have since moved to Kafka, as our general messaging bus for all event driven design throughout our microservices platform.

These problems have all since been addressed – they are but part of our story. Dealing with each new technology on its own was no big deal but a strong recommendation in hindsight is not to trial multiple new technologies in the one project. The reputation of that project could well fall to that of its weakest link.

a bright prosperous future

Of all these technologies it is Cassandra that has proved the most successful, and today is the essential and fundamental technology to our “big data” capabiilities.

On top of this platform we’ve come a long way. Today we use predeominantly Spark over Hadoop, with Spark jobs submitted to our YARN cluster. Although more and more use-cases are only requiring Cassandra. Some of these examples are messaging inboxes for each user, storing users search history, fraud detection, mapping probabilities of ipaddresses to geographical locations, and providing collaborative filtering using Spark’s ALS algorithm.

Eric Evans in the house

2015-03-10T14:37:49+00:00

In our continuing quest to break up our old monolith and create a more flexible and scalable architecture, we have realized that we need to look at some new tools and methods. One of the more promising approaches was Domain Driven Design, a term coined by Eric Evans in his Big Blue Book

Quite a few people in FINN has read (at least parts of) the Domain-Driven Design book, and we did some preliminary efforts to map our domains and look at our bounded contexts. We quickly realized that this was a very complex exercise, even with highly motivated tech and product people present. We figured we needed some help to get us on the right track. If you need help and information about DDD, why not get it straight from the horse’s mouth? So we got Eric Evans to visit us for a week at the end of February.

We needed to make the most of our time with Eric, so we set up a rather extensive agenda:

Day	Subject	Attending
Monday	Strategic Domain Driven Design Course	FINNs CTO, our functional product directors, the members of our enterprise architecture group including our Chief Enterprise Architect, several key developers and a couple of our international friends from Schibsted Classified Media
Tuesday	Strategic Domain Driven Design Course cont’d	Same as Monday
Wednesday	Workshop: Ad concept and our miscellaneous vertical	Chief Enterprise architect, EA group, Functional Product Director, Product Owner, Lead Developers
Thursday	Workshop: Job vertical and company profile	Chief Enterprise architect, EA group, Functional Product Director, Product Owner, Lead Developers
Friday	Presentation for everyone in FINN.no, plus workshop: Communications	Chief Enterprise architect, EA group, Functional Product Director, Product Owner, Lead Developers

The course on Monday and Tuesday was extremely helpful in getting everyone up to speed on the terms and method of strategic domain driven design.

The workshops the rest of the week focused on understanding the challenges involved in each area, and trying to map the bounded contexts of the existing systems.

Some lessons we learnt during the week.

When you are trying to create a context map, start with a very concrete example of how the system works. Which components interacts with each other, what data is passed between them, exactly what does the input and output look like
Use a huge whiteboard
Plan to use at least a few hours on every session. Doing a context map for even a part of a system may take a lot longer than you think.

Then you can manage to turn something like this:

Into a nice map like this:

Our key findings on bounded contexts:

Bounded contexts represent a higher order logical representation of a solution
They may span several of FINN’s current “microservices”
They are useful for understanding the high level architecture, and how teams and solutions depend on each other

The context map is a tool for discussions, and the boundaries and relationships are subjective.

Sample map focused on small subset of our system.

Example trying to map a larger part of our system.

Another thing we focused on was the concept of distilling the core. Figuring out which parts of your system that gives you a competitive advantage, and deserves extra attention. As part of this process it is also important to identify your supporting and generic domains. The tricky part is that it is really your business strategy that dictates what your core domain is.

So to sum things up: Having Eric Evans on-site for such a long time gave us invaluable insight into DDD, and how to apply it in our own business. Together the bounded context map, and our identified domains can help us drive development in the right direction. They are tools for discussion, not a framework that removes the need for highly competent developers.

JavaScript is evolving

2015-02-26T09:23:01+00:00

As a die-hard java-developer for several years I was quite annoyed by the fact that Java as a language lagged several years behind C#. (E.g. Java got lambdas last year, 6 years after C#, so standards are nice for compatibility, but tends to be slow to evolve.)

Polyfill

Now, as a frontend-developer and getting to know JavaScript, I have realized that JavaScript is lagging even further behind. Take for example the everyday case of checking if a string contains another string:

"hello world".indexOf("hello") > -1
//=> true

This is awkward and you know it, however JavaScript is evolving. The next version ECMAScript 6 is in its final phase, and will have, among several features, the following method:

"hello world".includes("hello")
//=> true

Another new nice feature is for finding the first item in an array. It used to be written like this:

for(int i = 0; i < array.length; i++){
  if(array[i].something === something){
    return array[i];
  }
}

And now it can be written like:

array.find(it => it.something === something);

ECMAScript 6 is feature complete, and is started to be supported by modern browser like Chrome, Firefox and Internet Explorer 11. As for older browsers, great polyfills like core-js come to our rescue.

Transpile

While polyfills extend current objects with new methods, ECMAScript 6 also contains new language syntax. Let’s look at the arrow function (also known as fat arrow):

["one", "two", "three"].map(v => v + "!")

Older browser will not understand “=>” (for a long time!), so this code has to be transpiled to ECMAScript 5:

["one", "two", "three"].map(function (v) {
  return v + "!";
});

There are some great transcompilation tools out there, and the one we have used is Babel. You can even play around with it here.

Scoped variables

JavaScript will also get scoped variables (let and const):

const hello = "Hello"

if(true) {
  const hello = "Inner scoped hello"
}

This will transpile to:

var hello = "Hello";

if (true) {
  var _hello = "Inner scoped hello";
}

Classes

And finally, JavaScript is getting support for classes (which is sugar around prototype), and the syntax looks like this:

class Hello {
  constructor() {
    this.hello = "Hello World"
  }
  getHello() {
    return this.hello;
  }
}

This will be transpiled to the following, not very readable, but working code:

var _prototypeProperties = function (child, staticProps, instanceProps) { if (staticProps) Object.defineProperties(child, staticProps); if (instanceProps) Object.defineProperties(child.prototype, instanceProps); };

var _classCallCheck = function (instance, Constructor) { if (!(instance instanceof Constructor)) { throw new TypeError("Cannot call a class as a function"); } };

var Hello = (function () {
  function Hello() {
    _classCallCheck(this, Hello);

    this.hello = "Hello World";
  }

  _prototypeProperties(Hello, null, {
    getHello: {
      value: function getHello() {
        return this.hello;
        },
        writable: true,
        configurable: true
      }
      });

      return Hello;
})();

Conclusion

ECMAScript 6 is here, and we should start using it. Older browser will be around for a long time, so I think we should get used to transpiling code on the web-platform.

Tor Arne Senior developer

27.feb.2015: Revised with improved code-examples from discussion on reddit.

Java 8 workshop at Finn.no

2015-02-16T07:50:16+00:00

Pondering on the right lambda for the job

With Java 8 comes a whole new set of language features. It even challenges the imperative coding-style of the Java programmer.

Java is a core language in Finn.no. More and more of our java-modules are being built with Java 8, adopting new features of the language. A workshop was warranted, with the goal of bringing every developer up to speed on the functional paradigm of Java 8.

In Finn.no, we want programmers to do stuff like this

db.fetchLastDayAds()
  .stream()
  .filter(ad -> "bap-webstore".equals(ad.getAdType()))
  .flatMap(ad -> ad.getContacts().stream())
  .distinct()
  .flatMap(per ->
    Optional.ofNullable(contact.getEmail()).map(Stream::of)
    .orElseGet(Stream::empty))
  .peek(contact -> LOG.trace("Sending notification to "+per))
  .forEach(this::sendNotification);

While avoiding stuff like this

IntStream.iterate(0, i -> (i + 1) % 2)
         .parallel()
         .distinct()
         .limit(10)
         .forEach(System.out::println);

(Locking up all cores on a CPU is bad, and should only be done when the machine is right about to become self-aware and turn against you.)

Several lambda-arrows pointing in the right direction here

We split the workshop into two half days. The first day was dedicated to streams and lambdas. Everyone seemed keen on getting those tests green (a rhyme!).

Day 2 we raised the bar with Optional and our in-house version of Either<L,R>. With these structures, much more code can be written functionally in a world where values might not exist (be null), and things may go wrong (throw exceptions).

We need power.. lots of power!

Both the word “monad” and the phrase “monadic domain” was uttered several times, but we still saw very few making the swoooosh-sound while flying a hand over their head (the internationally recognized sign of communicating that a topic is beyond mental capacity).

This might mean that the timing was good, and developers are interested in the new features of Java 8.

You may checkout the project and do the tasks yourself, by making the failing tests green.

Finn.no wanted to split up their monolith, you won't believe what happens next

2015-02-10T11:00:00+00:00

After working several years with (successfully) breaking up the legacy monolith, called iAD, Finn.no wanted to add structure to this process, and make sure that all 20 development teams work towards the same target. The target is a microservices architecture with small-ish autonomous services, owning their own data.

With that in mind we arranged our own two-day architecture summit for all Lead Developers, with a combination of external and internal presenters. The focus was on microservices, domain driven design, event driven architectures and modern programming techniques.

We started off the first day with a long session. Finn.no’s Chief Enterprise Architect Sebastian Verheughe wanted to communicate his vision. This included topics such as strategic domain-driven-design, data-highways, and how to use event-driven architecture to further decouple our services.

This theme continued, with the first external presenter. Jan Ove Skogheim from Particular Software talked about his experiences from Rikstoto, and how they had broken up a monolith into multiple small services. He also talked about the composite-UI pattern.

After lunch we continued with a presentation by Karsten Mevassvik from finn.no, of the most important properties for a finn.no microservice. These include:

Is RESTful
Owns its own data-store
Team ownership
Consumer Driven

The first day of the summit was concluded with a presentation of the CQRS architectural pattern. This presentation was held by Idar Borlaug and Andreas Berre from WebStep. They showed all the central concepts in CQRS, such as commands, event-stores, aggregates, sagas using very good examples to illustrate their concepts.

The second day of the summit started off with some hardcore programming. The frameworks of choice was node.js and RxJava. Christopher Kolstad started with an example of how a microservice could be implemented using Reactive Streams and RxJava. After that Phillip Johnsen presented a case using node.js in a real time context. These two talks gave a good introduction to new concepts that are not widely used in finn.no today.

During the second day several lightning talks were held, whenever there was a gap in the program. The talks, included topics as diverse as:

Testing of MicroServices
Unleash - Feature Toggles
Event Driven Architecure in Finn.no
Docker and Microservices.

Two more presentations should be mentioned. Firstly, one of the app-developers at Finn.no Haakon Eriksen, shared his opinions on what a perfect service looks like from a front-end developers viewpoint. The key point here was that the service developers should talk to their clients before and during development.

The last presentation was held by Geir Pettersen where he talked about his approach to software architecture, “growing software” using examples from the development of the new airline-tickets search.

From Wordpress to Jekyll

2015-01-28T13:40:36+00:00

Last year we decided to move our developer blog from Wordpress to Jekyll. One of the main reasons were the constant security vulnerabilities of Wordpress. We were hosting Wordpress on our own hardware. We pleased our operations team when they could shut down the service. Github provides hosting for Jekyll projects and that was one of many reasons we landed on that solution.

Other reasons why Jekyll is a good fit:

Static files (blazing fast)
Version controlled files (git) and the source is available here
Github handles username and passwords
Easy to run locally
You can use your favourite editor to write in Markdown or just plain HTML

Github restricts the number of plugins you can use down to this list. We really just wanted a simple blog platform so that was no deal breaker. We even get both @Andersos (@mentions) and :+1: (Emoji).

We landed on using Disqus for comments which is loaded on the client side. We made a design template post. We use that post to check out the design of different elements. That post is also a good starting point for people new to markdown. We are using Html-proofer to test the HTML of the site. Html-proofer checks that all the urls and images on the site are working. Another tip for improving blogposts is to use a tool like Hemingwayapp.

This developer blog is a place where we at Finn.no can share what we are working on and our interests. If you have suggestions or ideas for what we should write about please leave a comment under this post.

List of 10 most visited posts:

Finn.no year in review 2014

2015-01-07T10:30:17+00:00

The data in this post was gathered by Hans-Erik Hallheim, Henrik Faller and Lene Kallum.

The three most popular ads in 2014 were Audi A6, Walkman wanted badly and Skis with crap boots, used twice

We had over 7,4 million sofa related searches in 2014 and around 7000 unique permutations containing sofa.

React in the Real World

2014-12-15T12:36:31+00:00

At FINN oppdrag, a year ago we came to realize that our front-end code, written in JSP/jQuery, had become so large and complex that it affected our ability to produce features. We spent more time making sure the DOM manipulations were correct and did not break, than actually creating value for our users.

So we agreed that we needed a front-end framework to simplify our development and after looking into Ember and Angular we decided to go for React, recently open sourced by Facebook.

Our attraction to React was based on its promise of providing components with encapsulated state, fast automatic DOM manipulations, and the ability to render on the server.

Migration

We started by porting some functionality that already existed, and our first discovery was that the lines of codes needed to make the exact same functionality was almost reduced by half. Secondly, the code became clearer and easier to understand. And thirdly, as a combination of the reduced lines of code, the cleaner code and no manual DOM manipulations, we got less bugs.

Since then all new features have been written with React, and when we are requested to make modifications to functionality written in the old code, we rewrite them to use React.

When migrating large pages to React we start from the inside, the deepest DOM nodes, and let it gradually grow till the whole page is reactified. This often happened through several iterations and deployments, as React and the old JSP/JQuery can quite peacefully coexist.

Bundles

With React, HTML is represented as JavaScripts, so the JavaScript bundles will contain all the HTML, and hence grow as the pages grow. Instead of having a bundle for each page (very small), or having a bundle for the whole site (very big), we found a middle way where we have bundles based on roles, in our use cases contractor and owner. A possible improvement on this is loading JavaScript on demand with technologies like webpack.

Flux

So if React is our first great discovery, the Flux pattern is very much in second place. The Flux pattern decouples the logic from the views, in a different way than the traditional MVC-pattern. Logic and state are located in stores, and actions from the views to stores are asynchronous. When state in stores change, they will update React components that listen to changes, providing a single source of truth and unidirectional flow of data.

Server-side rendering

Our next step, which is yet to be solved, is server-side rendering. Server-side rendering means having the server generate and return the whole HTML page to the browser, instead of having the browser render HTML after downloading the JavaScript. This is especially important for SEO (search engine optimization) and mobile users, as search engines rarely run JavaScript, and mobile users have low bandwidth hence will take longer time before the page shows. However, having the server return a fully rendered HTML to browser is in general good for perceived page load performance.

Server-side rendering requires running JavaScript on the server, and that is straightforward with Node. However it is not that straightforward in our Java/Tomcat/Spring backend stack. We envision three possible solutions for this problem: 1) Running JavaScript on the JVM with the Nashorn JavaScript engine. 2) Delegating execution of JavaScript to an external process running Node. 3) Having Node run as smart-proxy in front of Java/Tomcat.

All these three solutions have their own benefits and challenges.

The first solution, running JavaScript in Java, has the benefits of close integration with the controller layer written in Java, and possibly making deployment easier, as the Java environment already has access to the JavaScript bundles. The downside of this approach seems to be performance, as the Nashorn JavaScript engine compiles and optimizes the JavaScript during the first runs, hence uses a long time to «warm up». So unless we write a mechanism that warms up the code, the first pages will be very slow to load.

The second solution, which involves interacting with an external process running Node, has the benefits that JavaScript runs natively and very fast. The downsides are more complicated deployment process where the Node instance needs access to the JavaScript bundles, and the overhead of interacting with an external process, possible through HTTP or Thrift, can impact performance negatively.

The third solution, where Node is placed in front Tomcat, and works as a smart-proxy, only enhancing certain pages, while simply forwarding the rest, has the benefits of executing JavaScript natively. While the downsides are more complicated deployment, and the possible complex process of executing certain JavaScript code found in HTML, and updating the HTML with the executed JavaScript, before it is served to the browser.

Conclusion

So, apart from server-side rendering, which we are working on, our transition to React has been smooth and a great success. It has improved our codebase, made developers happy, and most importantly, improved our speed of delivering features, which in the end is the most important thing.

Tor Arne Kvaløy Senior developer

Nova Hackathon

2014-12-11T17:00:00+00:00

We were invited by Nova for their first Nova Hackathon.

A Hackathon is period of time where computer programmers collaborate intensively on a given task or multiple tasks. For the Nova Hackathon Challenge Finn was invited to make a task for the students along with 2 other companies.

The challenge that Finn gave the students was to setup a simple web-server. It was based on Extreme startup by Robert Chatley. This software supports a workshop where teams can compete to build a software product that satisfies market demand.

We helped the students to setup their web-servers, each in their own respective language of choice. Their web-server’s url had to be registered on the Finn server. Once this was setup correctly, the Finn server would start to send http requests to their web-server. These http requests were various questions, the students had to get their web-server to respond to these requests not knowing the questions in advance, nor how they would be scored. The only thing they knew was that they would get a http request from the FINN server and had to respond to it. The questions were sent at a quick pace and were constantly evolving. This meant that their web-server’s code also was quickly evolving, a condensed real-life example of team-based agile programming.

Avanade’s task was to build an application for retrieving relevant data for a transport company. The students would then setup an prototype and a suggestion for further improvements. Itera gave the students a task centered around a calculator for mortgages, with focus on front-end javascript. It has to be mentioned that HP was so kind to borrow their offices for the Hackathon event. Read what HP has written about the Hackaton event.

The students was divided into 4 groups with different backgrounds. The business representatives helped the students underway and as you can see by this picture

the students was highely focused and competitive against the other groups.

Feedback from Nova was that no company in any of their events had ever been so involved throughout the whole day with the students as we had been. We had heaps of fun together and the extreme startup competition was a fantastic suggestion, hands-on and exciting. We had one team who implemented it in php and finished all 8 rounds in under 2 hours including questions that involved html scraping finn.no to find titles and prices for random finnkode! It was fun to see all the hard work and that the students put a tremendous effort in the challenge. The Finn ambassadors, Anders Olsen Sandvik , Mick Semb Sever, Nicolai Høge, Sverre Roos and Rida Aatif had a lot of fun collaborating with the students and we can agree that this is something we would definitely do again.

FINN techday 2014

2014-11-26T08:55:00+00:00

One of the ways of getting inspired and educated about technology here at FINN is through our “Tech-dag”. This year we have a packed program which consists of workshops, some talks and a bunch of lightning talks. Last time we posted about techday was in 2012.

Presentations

Throughout the day we had some long presentations and some lightning talks. Lightning talks are 10 minutes long and tightly timed.
The topics are broad ranging, and not limited to programming at all.

Starting #techdag2014 with @sisomm and the Internet of things #FINN_no pic.twitter.com/Xdx3HOMY3q
— Anders Olsen Sandvik (@Andersos) November 26, 2014

Here is the list of speakers and their slides:

Simen Sommerfeldt - Internet of Things
Rian Liebenberg
William Daraman, VG Tactus - "Egg og iBeacon"
Stefan Engelien - Eksperimentene som skapte FINN Småjobber
Geir Pettersen - Hospitering
Henning Spjelkavik - Strategisk planlegging med 'impact mapping'
Alexander Svanevik - Language Hacking
Tom Widerøe - Universell utforming – hva har blitt bedre
Per Jørgen Walstrøm - Nummi - om hockey og biler
Mikael Lindström - SPiD in the cloud
Anders Olsen Sandvik - Isomorphic Apps
Espen Dalløkken - a talk based on the blog post:I was wrong
Torgeir Waterhouse - hei FINN.no - hva skjer, internett og sånt...

Arduino Workshop

For this workshop we were all split out into teams and each team got a Arduino starter kit. The task was to make something cool. Check out this video from what group 10 did:

50 minutes of work. #techdag2014 pic.twitter.com/lbByxa7iJb
— Christopher Kolstad (@chriswk) November 26, 2014

#techdag2014 pic.twitter.com/J6qYJOrZBt
— PJ Walstroem (@pjwalstrom) November 26, 2014

#techdag2014 pic.twitter.com/H6uINBBsoH
— Sissel Sveum (@SisselSveum) November 26, 2014

Official hashtag #techdag2014

ApacheCon Budapest 2014

2014-11-25T13:00:00+00:00

Last week was Europe’s ApacheCon, held in Budapest.

A refreshingly “laid-back and no-nonsense” conference free from the douchebaggery that comes from big sponsors and marketing. This year it was held in the awesome 5-star Corinthia Hotel, and consisted of a welcoming and cheerful vibe, a place for apache members, committers, and all, to get together and better know each other. Apache continues to grow as the foundation for strong communities building trusted open sourced solutions, for running half the internet, and providing a substantial amount of code to your java stack.

Some hot topics were Docker, Mesos, Spark, Cassandra, CouchDB, Hadoop, Solr, OpenOffice, and the need for greater diversity in our communities.

Spark is all the rage because of its brevity and simplicity, but isn’t really a complete solution yet because it doesn’t scale in many situations.

Docker is awesome, and looks to take over the testing domain, but it remains limited and out of production as long as it has no network stack implementation. There was a very cool demonstration of Mesos and Aurora increasing production utilisation, even allowing in quiet periods for development/testing servers to come in.

The next release of YARN (2.6) will see support for long-lived services, whereby we can transform many of our hadoop jobs with very little effort into true streaming solutions. YARN will also be able to deploy docker containers through its clusters, this could be a very nice solution for our batch jobs. Putting these together and you see that the technical separation between streaming and aggregating solutions fades away and it really just boils down to what data structures you solve each use case with. Talking with a hadoop committer it came to light that FINN could be running the fastest hadoop cluster that they know of, given HDFS runs on SSDs and is dedicated just for hadoop internals.

Of no surprise Solr dealt a lot with scaling and performance, while Cassandra presented use-cases from across the industry. It felt that big tech companies are migrating their databases over to Cassandra not just for availability but often simply for the sake of performance. “Call of Duty” was rewritten from mysql to cassandra, resulting in the 95th percentile for requests going down to under 500μs. You don’t need complicated cache strategies when your backend is this fast!

Other Cassandra talks included Eric Evans presenting Newts, an lazy-aggregating time-series solution built with graphing and lucene search inbuilt. And Apache UserGrid, a fascinating project built, it is your LAMP stack, a BaaS framework, for the mobile-first world.

Patrick McFadin held a tutorial putting together kafka, cassandra, and spark, to create a very elegant scaling streaming solution. Patrick also discussed the performance gain of putting spark RDD data in cassandra rather than using files on disk. Chatting with both Eric Evans (he who coined “NoSQL”) and Patrick highlighted that the throughput per node of our Cassandra cluster is leading in the industry, it’s just to add more servers and reap the benefits.

And yours humbly presented to a packed room our Cassandra and Hadoop use-cases. Making mention to the need for better gender balance within Apache, Schibsted’s role in online classified markets across Europe, the need to look beyond current trends in our industry when designing systems, and having a healthy preference towards AP systems in the enterprise.

Absolutely a conference I’d recommend to infrastructure, operations, and back-enders serious about their technologies.

Oh… and Happy 15th birthday Apache!

More photos from the conference here

Email providers in Norway

2014-11-21T18:00:00+00:00

It all started with a simple tweet:

Er det noen som har noe norsk statistikk på bruk av e-post (gmail vs. outlook vs. online vs. yahoo osv)? F.eks @labs_finn_no eller @NRKbeta?
— Håvard Bergersen (@haavardb) March 7, 2014

“Does anyone have any Norwegian statistics on the use of email (Gmail vs. Outlook vs. Online vs. Yahoo etc.)? Eg @labs_finn_no or @nrkbeta.”

We have access to the data so if this will help someone why not post it. We have around 4 million email addresses in a database. These emails are not verified. The data was retrieved 2014-06-06. To answer the question “Gmail vs. Outlook vs. Online vs. Yahoo etc.” we would say:

Microsoft 33,57%
Google 15,32%
Telenor 12,53%
Yahoo 4,31%

The lists under shows the domain, number registered and percent of total. At the end is the table of the more popular email service providers (this list is cut off at 2500). Watch this space in the future if this is information that interests you.

Popular services:

Microsoft 33,57% (sum of hotmail, live, msn and outlook)
hotmail.com (1106084 - 28.06%)
live.no (96052 - 2.44%)
hotmail.no (67602 - 1.71%)
msn.com (29675 - 0.75%)
live.com (15014 - 0.38%)
outlook.com (9142 - 0.23%)

Google 15,32% (sum of gmail and googlemail)
gmail.com (599293 - 15.20%)
gmail.no (2982 - 0.08%)
googlemail.com (1766 - 0.04%)

Telenor 12,53% (sum og online, frisurf, telenor and adsl)
online.no (444897 - 11.28%)
frisurf.no (41768 - 1.06%)
telenor.com (3903 - 0.10%)
adsl.no (3643 - 0.09%)

Yahoo 4,31%
yahoo.no (94029 - 2.39%)
yahoo.com (75772 - 1.92%)

Universities:

stud.ntnu.no (7688 - 0.20%)
ntnu.no (1421 - 0.04%)
stud.nhh.no (1008 - 0.03%)
student.bi.no (925 - 0.02%)
hiof.no (921 - 0.02%)
medisin.uio.no (820 - 0.02%)
student.sv.uio.no (746 - 0.02%)
ifi.uio.no (725 - 0.02%)
uit.no (666 - 0.02%)
stud.hib.no (557 - 0.01%)
studmed.uio.no (522 - 0.01%)
umb.no (411 - 0.01%)
uus.no (408 - 0.01%)
uis.no (407 - 0.01%)

Telcos:

online.no (444897 - 11.28%)
frisurf.no (41768 - 1.06%)
telenor.com (3903 - 0.10%)
adsl.no (3643 - 0.09%)

c2i.net (67080 - 1.70%)
broadpark.no (48232 - 1.22%)
lyse.net (25964 - 0.66%)
getmail.no (19497 - 0.49%)
chello.no (18485 - 0.47%)
tele2.no (12218 - 0.31%)
bluezone.no (10187 - 0.26%)
netcom.no (7987 - 0.20%)
sensewave.com (7635 - 0.19%)
ntebb.no (6122 - 0.16%)

Kommune:

trondheim.kommune.no (1203 - 0.03%)
bergen.kommune.no (1156 - 0.03%)
baerum.kommune.no (906 - 0.02%)
kristiansand.kommune.no (688 - 0.02%)
asker.kommune.no (557 - 0.01%)
stavanger.kommune.no (500 - 0.01%)
fredrikstad.kommune.no (433 - 0.01%)
sandnes.kommune.no (420 - 0.01%)

Some random:

mil.no (2231 - 0.06%)
nrk.no (1942 - 0.05%)
finn.no (1731 - 0.04%)
vegvesen.no (1700 - 0.04%)
posten.no (1566 - 0.04%)
politiet.no (1549 - 0.04%)
operamail.com (1511 - 0.04%)
sas.no (1324 - 0.03%)
vg.no (656 - 0.02%)

Domain	Number registered	Percent of total
hotmail.com	1106084	28.06%
gmail.com	599293	15.20%
online.no	444897	11.28%
live.no	96052	2.44%
yahoo.no	94029	2.39%
yahoo.com	75772	1.92%
hotmail.no	67602	1.71%
c2i.net	67080	1.70%
broadpark.no	48232	1.22%
frisurf.no	41768	1.06%
start.no	33409	0.85%
msn.com	29675	0.75%
lyse.net	25964	0.66%
getmail.no	19497	0.49%
chello.no	18485	0.47%
live.com	15014	0.38%
me.com	12344	0.31%
tele2.no	12218	0.31%
wp.pl	10883	0.28%
bluezone.no	10187	0.26%
outlook.com	9142	0.23%
netcom.no	7987	0.20%
stud.ntnu.no	7688	0.20%
sensewave.com	7635	0.19%
tiscali.no	7583	0.19%
spray.no	7044	0.18%
bbnett.no	6887	0.17%
ntebb.no	6122	0.16%
epost.no	5975	0.15%
statoil.com	5835	0.15%
live.se	5273	0.13%
telia.com	4879	0.12%
mac.com	4588	0.12%
icloud.com	4532	0.11%
ebnett.no	4329	0.11%
haugnett.no	4294	0.11%
mail.ru	4091	0.10%
losmail.no	4069	0.10%
mimer.no	3995	0.10%
telenor.com	3903	0.10%
mail.com	3874	0.10%
adsl.no	3643	0.09%
o2.pl	3587	0.09%
yahoo.co.uk	3572	0.09%
inbox.lv	3348	0.08%
hjemme.no	3185	0.08%
halden.net	3021	0.08%
vikenfiber.no	3005	0.08%
gmail.no	2982	0.08%
student.uib.no	2953	0.07%
ymail.com	2901	0.07%
hydro.com	2852	0.07%
sf-nett.no	2689	0.07%
aol.com	2645	0.07%
interia.pl	2589	0.07%
web.de	2582	0.07%
loqal.no	2566	0.07%
yahoo.se	2517	0.06%

Workshop with blindfolded lunch

2014-11-19T18:00:00+00:00

Workshop with blindfolded lunch in cooperation with The Norwegian Association of the Blind and Partially Sighted (NABP)

At FINN a grassroots group of developers and interaction designers launched an initiative to bring up awareness of accessibility. The law of accessibility was introduced July 1st 2014 and m.finn.no is covered by that law because of its major design changes. The Accessibility group in FINN wanted to convey the same message as the NABP; caring for people with disabilities when we build our services, hoping it lead to better user experiences for all.

Wednesday November 19th, FINN arranged an accessibility workshop in cooperation with NABP. About 30 developers, designers and product owners attended the keynote by Stein Erik Skotkjerra who is blind and by Kristoffer Lium who is partially sighted. It was an eye-opener for people to see how these two who represented a fairly large group of blind and visually impaired, orients themselves on finn.no.

A smaller group of developers and interaction designers attended the whole workshop. We got an introduction to the law of accessibility and the requirements in practice. Stein Erik and Kristoffer both emphesized that the most important is to make sites user friendly; not the law itself. If we think of people with disabilities when we develop finn.no, it will lead to a better site for everyone. Even for those who have a headache one day or are sitting on a bumpy bus while browsing finn.no.

The highlight of the day was perhaps “blindfolded lunch”. All workshop participants put on ski goggles that were spray painted black so no-one could see anything. Then they were ushered around before being led to a prepared lunch table. The canteen staff from FINN helped to create an exciting lunch with different samples in small cups. It was a challenge to know where the food was, to spread the butter on the bread, and simply to understand what we were eating.

At the end of the day the workshop participants tried out various tools Stein Erik, Kristoffer, and other blind and visually impaired use. The screen was zoomed several hundred percent and the colors were inverted. This is what Kristoffer usually does. With this enlargement and conversion we should try to locate an item on FINN Torget, and try to post a new ad. Again, this was an eye-opener, and we wrote down the areas were we got challenges. This had to be improved! Along with Stein Erik we used a screen reader. And on the last station, we used only the keyboard; no mouse. In all the stations we found areas that could be improved.

The last half hour of the workshop was used to solve the challenges we had found during the day. The result was several pull requests and a good number of issues in the backlog to make finn.no a better site for everyone!

Log4j2 - dependencies

2014-10-27T11:50:00+00:00

To make it easy for all our codebases, to automatically have all logging abstractions pointing towards log4j2 along with logstash configured, we have bundled together the dependencies via one library. These dependencies look like:

<!-- core libraries -->  
org.apache.logging.log4j:log4j-api:2.0.2  
org.apache.logging.log4j:log4j-core:2.0.2  
com.lmax:disruptor:3.2.1  

<!-- commons-logging -->  
org.apache.logging.log4j:log4j-jcl:2.0.2  
org.apache.logging.log4j:log4j-1.2-api:2.0.2  

<!-- slf4j -->  
org.slf4j:slf4j-api:1.7.7  

<!-- JUL routed through slf4j -->  
org.apache.logging.log4j:log4j-slf4j-impl:2.0.2  

<!-- override old log4j with an empty jarfile -->  
<!-- incase it gets re-introduced transitively -->  
log4j:log4j:2-empty  

<!-- logstash -->  
net.logstash.log4j2:log4j2-logstash-jsonevent-layout:3.0.0-finn-2

Log4j2 in production – making it fly

2014-07-01T22:17:00+00:00

Now that log4j2 is the predominant logging framework in use at FINN.no why not share the good news with the world and try to provide a summary over the introduction of this exciting new technology into our platform.

let's just one logging framework

In beginning of September 2013 it became the responsibility of one of our engineering teams to introduce a “best practice for logging” for all of FINN.

The proposal put forth was that we can standardise on one backend logging framework while there would be no need to standardise on the abstraction layer directly used in our code.

The rationale to not standardising the logging abstractions was…

• nearly every codebase, through a tree of dependencies, already includes all the different logging abstraction libraries, so the hard work of configuring them against the chosen logging framework is still required, • the different APIs to the different logging abstractions are not difficult for programmers to go between from one project to another.

While the rationale to standardising the logging framework was…
• makes life easier for programmers with one documented “best practice” for all,
• makes it possible through an in-house library to configure all abstraction layers, creating less configuration for programmers, • makes life easier for operations knowing all jvms log the same way,
• makes life easier for operations knowing all log files follow the same format.

Log4j2 wins HANDS down

Log4j2 was chosen as the logging framework given… • it provided all the features that logback was becoming popular for, • between old log4j and logback it was the only framework that was written in java with modern concurrency (ie not hard synchronised methods/blocks), • it provided a significant performance improvement (1000-10000 times faster)
• it consisted of a more active community (logback has been announced as the replacement for the old log4j, but log4j2 saw a new momentum in its apache community).

This proposal was checked with a vote by finn programmers • 73% agreed, 27% were unsure, no-one disagreed.

when nightly compression jams

Earlier on in this process we hit a bug with the old log4j where nightly compression of already rotated logfiles were locking up all requests in any (or most) jvms for up to ten seconds. This fault came back to poor java concurrency code in the original log4j (which logback cloned). Exacerbated by us having scores of jvms for all our different microservices running on the same machines so that when nightly compression kicked in it did so all in parallel. Possible fixes here were to
a) stop compression of log files,
b) make loggers async, or
c) migrate over quickly to log4j2.

After some investigation, (c) was ruled out because there was no logstash plugin for log4j2 ready and moving forward without the json logfiles and the logstash & kibana integration was not an option. (a) was chosen as a temporary solution.

ready, steady, go…

Later on, when we started the work on upgrading all our services from thrift-0.6.1 up to thrift-0.9.1, we took the opportunity to kill two birds with the one stone. Log4j2 was out of beta, and we had ironed out the issues around the logstash plugin.

We’d be lying if we told you it was all completely pain free, introducing Log4j2 came with some concerns and hurdles.

• Using a release candidate of log4j2 in production lead to some concerns. So far the only consequence was slow startup times (eg even small services paused for ~8 seconds during startup). This was due to log4j2 having to scan all classes for possible log4j plugins. This problem was fixed in 2.0-rc2. On the bright side – our use of the release candidate meant we spotted early and getting the upcoming initial release of log4j2 to support shaded jarfiles, of which we are heavily dependent on. • Operation’s had expressed concerns over nightly compression, raised from the earlier problem around nightly compression, that even if code no longer blocked while the compressing was happening in a background thread the amount of parallel compressions spawned would lead to IO contention which in turn leads to CPU contention. Because of this very real concern extensive tests have been executed, so far they’ve shown no measurable (under 1ms) impact exists upon services within the FINN platform. Furthermore this problem can be easily circumvented by adding the SizeBasedTriggeringPolicy to your appender, thereby enforcing a limit on how much parallel compression can happen at midnight. • The new logstash plugin (which finn has actively contributed to on github) caused a few breakages to the format expected by our custom logstash parsers written by operations. Unfortunately this parser is based off the old log4j format, of which we are trying to escape. Breakages here were: log events on separate lines, avoiding commas are the end of lines between log events, thread context in wrong format, etc. These were tackled with pull requests on github and patch versions of our commons-service (the library used to pre-configure the correct dependency tree for log4j2 artifacts and properly plugging in all the different logging abstraction libraries). • Increased memory from switching to sync loggers to async loggers impacted services with very small heap. The async logger used is based of the lmax-disruptor which pre-allocates its ringBuffer with its maximum capacity. By default this ringBuffer is configured to queue at maximum 256k log events. This can be adjusted with the “AsyncLoggerConfig.RingBufferSize” system property.

simply beautiful

To wrap it up the hurdles have been there, but trivial and easy to deal with, while the benefits of introducing log4j2, and moving to async loggers, make it well worth it… • The “best practice” for log4j2 included changing all loggers to be async, and this means that the performance of the FINN platform (which consists primarily of in-memory services) is no longer tied to and effected by how disks are performing (it was crazy that it was before). • More and more applications are generating consistent logfiles according to our best practices. • More and more applications are actually plugging in the various different logging abstractions used by all their various third-party dependencies. • All the advantages people liked about logback. • An easier approach to changing loglevels at runtime through jmx. • Profiling applications in crisis easier for outsiders (one less low-level behavioural difference). • Loggers are no longer a visible bottleneck in jvms under duress. • And naturally the performance increase.

Because of the significant performance provided by the lmax-disruptor we also use the open-sourced statsd client that takes advantage of it.

We love npm

2013-12-03T10:38:00+00:00

What? Why is FINN.no donating to scale NPM? I thought you guys were a pure Java shop? It is true, we used to be a pure Java-shop. However over the past three years we have adopted new technologies to solve specific problems. We have used Ruby and Cucumber for some time to make a platform for continuos delivery and it has worked out beautifully! Our front-end developers have been forced to deal with out dated and not suitable tools for doing their job. This is largely due to the fact that all innovation when it comes to front-end development does not happen in the Java community. Most of the exciting tools are written in Node and this has become a frustration and a challenge for us.

In the past year FINN have been gradually making a transition away from using only Java-based tools for front-end development and towards a NodeJS powered tool set. We are now at a point were we are on the brink of rolling this out for our projects. Having worked with Node for a while we have learned to appreciate the Node ecosystem which is NPM. Being part of such a vibrant ecosystem of modules makes the transition easier and it also inspires us to become better at giving back. Therefore we are trying to give back to NPM, when we can.

When the scale NPM campaign was launched it was obvious that this was something we wanted to be apart of. It is an investment in our own happiness in a sense, as NPM is becoming a very important part for our technology portfolio.

Nodeify all the things

So were is it that we use Node in our technology stack today? Earlier this year we moved away from JsTestDriver in favor of Karma-runner. This meant that we needed to create a trojan horse containing the goodness of Node/NPM into existing Java projects without causing too many problems for developers with no knowledge of Node. A part of this scheme was the frontend-maven-plugin, which enables us to have control of which Node projects use and allows developers without Node previously installed to build projects and run tests without having to learn anything about Node.

Currently we are working towards removing the need for using Maven to build and deploy pure JavaScript projects using Node. The end result is of course to have more web applications built using Node. Today we have just two which are running in a production environment.

Low-fi Coffe Surveilence

2013-11-29T09:03:00+00:00

There is a Norwegian start-up called AppearIn which enables anyone to do online video conferencing with only a web browser. They utilize WebRTC to empower users to create rooms where they can hang out, do meetings, etc. We had the pleasure of having Ingrid form Appear In over for a visit and she suggested we’d contribute to their blog about use cases for appear in. That’s how we became use case #7 on their blog. Naturally this should’ve been an arduino powered thing with lot’s of sophisticated technology, but we decided we’d just get something simple working first and then make it better later on.

This is a really exciting platform as it empowers developers to have video/audio communication with little effort. Dealing with WebRTC, which is still a moving specification, can be quite challenging to implement properly. There’s numerous corner cases and differences between implementations.

Update: Through the wonders of the internet, it turns out that we have brought the webcam thing full circle. A user on Hacker News posted a comment which linked to a wikiepedia article about the first webcam. It turns out the use case for it was exactly the same as we had.

Package Management conflicts Continuous Delivery

2013-06-26T20:42:28+00:00

The idea of package management is to correctly operate and bundle together various components in any system. The practice of package management is a consequence from the design and evolution of each component’s API.

Package management is tedious

but necessary. It can also help to address the ‘fear of change’.

We can minimise package management by minimising API. But we can’t minimise API if we don’t have experience with where it comes from. You can’t define for yourself what the API of your code is. It is well beyond that of your public method signatures. Anything that with change can break a consumer is API.

Continuous Delivery isn’t void of API

despite fixed and minimised interfaces between runtime services, each runtime service also contains an API in how it behaves. The big difference though is you own the release change, a la the deployment event, and if things don’t go well you can roll back. Releasing artifacts in the context of package management can not be undone. Once you have released the artifact you must presume someone has already downloaded it and you can’t get it back. The best you can do it release a new version and hope everyone upgrades to it quickly.

Push code out from behind the shackles of package management

take advantage of continuous delivery! Bearing in mind a healthy modular systems design comes from making sure you got the api design right – so the amount one can utilise CD is ultimately limited, unless you want to throw out modularity. In general we let components low in the stack “be safe” by focusing on api design over delivery time, and the opposite for components high in the stack.

High in the stack doesn’t refer to front-end code

Code at the top of the stack is that free of package management and completely free for continuous deployment. Components with direct consumers no longer sit at the top of the stack. As components consumers multiple, and they become transitive dependencies, they move further down the stack. Typically entropy of the component corresponds to position in the stack. Other components forced into package management can be those where parallel versions need be deployed.

Some simple rules to abide by…

don’t put configuration into libraries. because this creates version-churn and leads to more package management
don’t put services into libraries. same reason as above.
don’t confuse deploying with version releases. don’t release every artifact as part of a deployment pipeline. separate concerns of continuous delivery and package management.
try to use a runtime service instead of a compile-time library. this minimises API, in turn minimising package management,
try to re-use standard APIs (REST, message-buses, etc). the less API you own the less package management. but don’t cheat! data formats are APIs, and anything exposed that breaks stuff when changed is API.

Dark Launching and Feature Toggles

2013-06-20T11:56:20+00:00

Make sure to distinguish between these two.

They are not the same thing, and it’s a lot quicker to just Dark Launch.

In addition Dark Launching promotes incremental development, continuous delivery, and modular design. Feature Toggles need not, and can possibly be counter-productive.

Dark Launching is an operation to silently and freely deploy something. Giving you time to ensure everything operates as expected in production and the freedom to switch back and forth while bugfixing. A Dark Launch’s goal is often about being completely invisible to the end-user. It also isolates the context of the deployment to the component itself.

Feature Toggling, in contrast, is often the ability to A/B test new products in production. Feature Toggling is typically accompanied with measurements and analytics, eg NetInsight/Google-Analytics. Feature Toggles may also extend to situations when the activation switch of a dark launch can only happen in a consumer codebase, or when only some percentage of executions will use the dark launched code.

Given that one constraint of any decent enterprise platform is that all components must fail gracefully Dark Launching is the easiest solution, and a golden opportunity to ensure your new code fails gracefully. Turn the new module on, and it’s dark launched and in use, any problems turn it off again. You also shouldn’t have to worry about only running some percentage of executions against the new code, let it all go to the new component, if the load is too much the excessive load should also fail-gracefully and fall back to the old system.

Dark Launching is the simple approach as it requires no feature toggling framework, or custom key-value store of options. It is a DevOps goal that remains best isolated to the context of DevOps – in a sense the ‘toggling’ happens through operations and not through code. When everything is finished it is also the easier approach to clean up. Dealing with and cleaning up old code takes up a lot of our time and is a significant hindrance to our ability to continually innovate. In contrast any feature toggling framework can risk encouraging a mess of outdated, arcane, and unused properties and code paths. KISS: always try and bypass a feature toggle framework by owning the ‘toggle’ in your own component, rather than forcing it out into consumer code.

Where components have CQS it gets even better. First the command component is dark launched, whereby it runs in parallel, and data can be test between the old and new system (blue-green deployments). Later on the query component is dark launched. While the command components can run and be used in parallel for each and every request the query components cannot. When the dark launch of the query component is final the old system is completely turned off.

Now the intention of this post isn’t to say we don’t need feature toggling, but to give terminology to, and distinguish, between two different practices instead of lumping everything under the term “feature toggling”. And to discourage using a feature toggling framework for everything because we fail to understand the simpler alternative.

In the context of front-end changes, it’s typical that for a new front-end feature to come into play there’s been some new backend components required. These will typically have been dark launched. Once that is done, the front-end will introduce a feature toggle rather than dark launch because it’s either introducing something new to the user or wanting to introduce something new to a limited set of users. So even here dark launching can be seen not as a “cool” alternative, but as the prerequisite practice.

Reference: “DevOps for Developers” By Michael Hüttermann

Conquering the kingdom of Java with a NodeJS trojan horse

2013-06-06T07:36:00+00:00

Most of FINN.no is built with Java and we’ve been a Java shop for 6-7 years. With the professionalization of front-end development we saw that the tools available in the Java Web Development ecosystem was preventing us from delivering rapidly and with high quality. The trend was that the tools we used weren’t keeping up with the pace set by people creating tools in Node’s NPM module ecosystem. When working on pet projects we could use these tools, but in our daily job we were working with second rate tools. This was a frustration for many of us, so we set out to work out how we could utilize Node without having to rewrite all our applications.

In the belly of the beast that is Maven

We use Maven to build and test all our applications. This includes continuous deployment and quality analysis. Our reporting tools requires projects to be built with Maven in order to work. To introduce Node-based tools for front-end tasks, we had to be able to run them from Maven.

The easiest way to get started is just to use the Maven exec plugin. It allows you to run commands like on the command line. This approach worked, but there was one catch: it relied upon Node + NPM to be installed on every developer machine as well as in our CI environment. Forcing 100+ developers to install Node and keep up with versions was something which wasn’t really acceptable for the rest of the organization. Enter The Maven Front-end Plugin

Behold! The Frontend Maven plugin

Erik Sletteberg, who started out working as a summer intern and is now working at FINN.no full time, helped solve the problem of Node having to be installed. He created the Frontend Maven Plugin which install Node+NPM in the project folder. The plugin also has tasks for running NPM, Grunt, Gulp and similar. Because of the tool we were able to use Node within a Maven web application module, which was awesome! It meant we could use build-, test and code analysis tools developed in Node and have them report into our existing tool chain.

Independence with Standalone Node modules

In the Java stack there is no real support for distributing front end dependencies for things like JavaScript, CSS, etc. There are plugins for things like Require-js for JavaScript, but they were all suffering from the same problem: the plugins couldn’t keep up with the pace set by the Node ecosystem when it came to iterating on these kinds of tools. Our setup at the time was to split out any front-end resource which needed to be distributed into Maven Web Application modules and include them like any other Maven dependency in the POM. The release process for Maven artifacts is tedious. When importing dependencies into an existing Maven module you had to perform all kinds of URL rewriting to make paths correct. In short, this was not a very smooth way of sharing client side resources.

Having Node and NPM running inside Maven, it was only natural for us to look at how we could create Common JS modules of our existing JS modules which were now in Maven. The short term goal was to get rid of Maven while developing these modules. The long term goal is to have client side dependencies declared as regular Node dependencies in projects.

Maven-deploy

In order to make the transition as smooth as possible we needed to have Node modules which we were able to deploy to Maven repositories in addition to our private NPM registry. Gregers G. Rygg from the Front-end Core Team set out to create the maven-deploy module which solves this (if you use Gulp, you can use the gulp-maven-deploy plugin). It can either install the artifact locally under M2_HOME or deploy it to a remote repositories. This way we could have Node modules behave as if they were Maven modules, which in turn meant that existing Java Web Application didn’t have to be changed in order for us to use Node.

In closing

This was by no means the fastest way to introduce Node into an existing architecture. If you’re a small shop with few people, you should adopt a more aggressive approach. Our approach however works beautifully for those of you working in larger organizations were you need to do things in a more subtle way. Taking small steps and always making sure everyone can perform their job as normal was one of our keys to succeeding. If we’d gone all out and created additional work for many developers, our Node adventure would’ve never happened.

Espen Dalløkken from the Front-end Core Team did a short lightening talk about this very subject at Booster Conference 2014 in Bergen:

Conquering the wicked kingdom of Java with a NodeJS Trojan horse - Espen Dalløkken from Booster conference on Vimeo.

given the git

2013-03-20T11:14:08+00:00

“There is no way to do CVS right.” - Linus

FINN is migrating from subversion to the ever trendy git. We’ve waited years for it to happen, here we’ll try to highlight why and how we are doing it.

Working together

There’s no doubt that git gives us a cleaner way of working on top of each other. Wherever you promote peer review you need a way of working with changesets from one computer to the next without having to commit to and via the trunk where everyone is affected. Creating custom branches comes with too much (real or perceived) overhead, so the approach at best falls to throwing patches around. Coming away from a pair-programming session it’s better when developers go back to their own desk with such a patch so they can work on it a bit more and finish it properly with tests, docs, and a healthy dose of clean coding. It properly entitles them as the author rather than appearing as if someone else took over and committed their work. Git’s decentralisation of repositories provides the cleaner way by replacing these patches with private repositories and its easy to use branches.

Individual productivity

Git improves the individual’s productivity with benefits of stashing, squashing, reseting, and rebasing. A number of programmers for a number of years were already on the bandwagon using git-svn against our subversion repositories. This was real proof of the benefits, given the headaches of git-svn (can’t move files and renaming files gives corrupted repositories)

With Git, work is encouraged to be done on feature branches and merged in to master as complete (squashed/rebased) changesets with clean and summarised commit messages.

This improves efforts towards continuous deployment due to a more stable HEAD.
Rolling back any individual feature regardless of its age is a far more manageable task.
By squashing all those checkpoint commits we typically make we get more meaningful, contextual, and accurate commit messages.

Reading isolated and complete changesets provides clear oversight, to the point reading code history becomes enjoyable, rather than a chore. Equally important is that such documentation that resides so close to, if not with, the code comes with a real permanence. There is no documentation more accurate over all of time than the code itself and the commit messages to it. Lastly writing and rewriting good commit message will alleviate any culture of jira issues with vague, or completely inadequate, descriptions as teams hurry themselves through scrum methodologies where little attention is given to what is written down.

Maintaining forks

Git makes maintaining forks of upstream projects easy.

With Git fork the upstream repository, branch, fix and commit, create the upstream pull request, while you wait for the pull request to be accepted/rejected use your custom inhouse artifact.

With Subversion file an upstream issue, checkout the code, fix and store in a patch attached to the issue, while you wait use the inhouse custom artifact from the patched but uncommited codebase.

Both processes are largely the same but it’s safer and so much easier using a forked git repository over a bunch of patch files.

Has Git-Flow any advantage?

We’re putting some thoughts into how we were to organise our repositories, branches, and workflows. The best introductory article we’ve so far come across is from sandofsky and should be considered mandatory reading. Beyond this one popular approach is organising branches using Git Flow. This seemed elegant but upon closer inspection comes with more disadvantages than benefits…

the majority needs to ‘checkout develop’ after cloning (there are more developers than ops),
master is but a sequence of “tags” and therefore develop becomes but a superfluous branch, a floating “stable” tag instead is a better solution over any branch that simply marks states,
it was popular but didn’t form any standard,
requires a script not available in all GUIs/IDEs, otherwise it is but a convention,
prevents you from getting your hands dirty with the real Git, how else are you going to learn?,
it goes against having focus and gravity towards continuous integration that promotes an always stable HEAD. That is we desire less stabilisation and qa branches, and more individual feature and fix branches.

GitHub Flow gives a healthy critique of Git-Flow and it helped identify our style better. GitHub Flow focus’ on continuous integration, “deploy to production every day” rather than releasing, and relying on git basics rather than adding another plugin to our development environment.

Our workflows

So we devised two basic and flexible workflows: one for applications and one for services and libraries. Applications are end-user products and stuff that has no defined API like batch jobs. Services are our runtime services that builds up our platform, each come with a defined API and client-server separation in artifacts. Applications are deployed to environments, but because no other codebase depends on them their artifacts are never released. Services, with their APIs and client-side libraries, are released using semantic versions, and the server-side code to them is deployed to environments in the same manner as Applications. The differences between Applications and Services/Libraries warrant two different workflow styles.

Both workflow styles use master as the stable branch. Feature branches come off master. An optional “integration” (or “develop”) branch may exist between master and feature branches, for example CI build tools might automatically merge integration changes back to master, but take care not to fall into the anti-pattern of using merges to mark state.

The workflow for Applications may use an optional stable branch where deployments are made from, this is used by projects that have not perfected continuous deployment. Here bug fix branches are taken from the stable branch, and forward ported to master. For applications practising continuous deployment a more GitHub approach may be taken where deployments from finished feature branches occur and after successfully running in production such feature branches are then merged to master.

The workflow for Services is based upon each release creating a new semantic version and the git tagging of it. Continuous deployment off master is encouraged but is limited to how compatible API and the client libraries are against the HEAD code in the master branch – code that is released and deployed must work together. Instead of the optional stable branch, optional versioned branches may exist. These are created lazy from the release tag when the need for a bug fix on any previous release arises, or when master code no longer provides compatibility to the released artifacts currently in use. The latter case highlights the change when deployments start to occur off the versioned branch instead of off master. Bug fix branches are taken from the versioned branch, and forward ported to master.

Similar to Services are Libraries. These are artifacts that have no server-side code. They are standalone code artifacts serving as compile-time dependencies to the platform. A Library is released, but never itself deployed to any environment. Libraries are void of any efforts towards continuous deployments but otherwise follow very similar workflow as Services – typically they give longer support to older versions and therefore have multiple release branches active.

How any team operates their workflow is up to them, free to experiment to see what is effective. At the end of the day as long as you understand the differences between merge and rebase then evolving from one workflow to another over time shouldn’t be a problem.

Infrastructure: Atlassian Stash

The introduction of Git was stalled for a year from our Ops team as there was no repository management software they were happy enough with to support (integration with existing services was important, particularly Crowd). Initially they were waiting on either Gitolite or Gitorius. Eventually someone suggested Stash from Atlassian and after a quick trial this was to be it. We’re using a number of Atlassian products already: Jira, Fisheye, Crucible, and Confluence; so the integration factor was good and so we’ve paid for a product that was at the time incredibly overpriced with next to nothing on its feature list. One feature the otherwise very naked Stash comes with is Projects, which provides a basic grouping of repositories. We’ve used this grouping to organise our repositories based on our architectural layers: “applications”, “services”, “libraries”, and “operations”. The idea is not to build fortresses with projects based on teams but to best please the outsider who is looking for some yet unknown codebase and only knows what type of codebase it is. We’re hoping that Atlassian adds labels and descriptions to repositories to further help organisation. Permissions is easy, full read and write access to everyone, we’ll learn quickest when we’re all free to make mistakes, it’s all under version control at the end of the day.

We’re still a cathedral

Git decentralises everything, but we’re not a real bazaar: our private code is our cathedral with typical enterprise trends like scrum and kanban in play; and so we have still the need to centralise a lot. Our list of users and roles we still want centralised, when people push to the master repository are all commits logged against known users or are we going to end up with multiple aliases for every developer? Or worse junk users like “localhost”? To tackle this we wrote a pre-push hook that authenticates usernames for all commits against Crowd. If a commit from an unknown user is encountered the push fails and the pusher needs to fix their history using this recipe before pushing again.

Releases can be made off any clone and obviously not be something we want. Released artifacts need to be permanent and unique, and deployed to our central maven repository. Maven’s release plugin fortunately tackles this for us as when you run mvn release:prepare or mvn release:branch it automatically pushes resulting changes upstream for you, as dictated by the scm details in the pom.xml

Migrating repositories

Our practice with subversion was to have everything in one large subversion repository, like how Apache does it. This approach worked best for us allowing projects and the code across projects to be freely moved around. With Git it makes more sense for each project to have its own repository, as moving files along with their history between repositories is easy.

Initial attempts of conversion were using svn2git as described here along with svndumpfilter3.

But a plugin in Stash came along called SubGit. It rocks! Converting individual projects from such a large subversion repository one at a time is easy. Remember to moderate the .gitattributes file afterwards, we found in most usecases it could be deleted.

Hurdles

Integration with our existing tools (bamboo, fisheye, jira) was easier when everything was in one subversion repository. Now with scores of git repositories it is rather cumbersome. Every new git repository has to be added manually into every other tool. We’re hoping that Atlassian comes to the rescue and provides some sort of automatic recognition of new and renamed repositories.

Renaming repositories in Stash is very easy, and should be encouraged in an agile world, but it breaks the integration with other tools. Every repository rename means going through other tools and manually updating them. Again we hope Atlassian comes to the rescue.

Binary files we were worried about as our largest codebase had many and was already slow on subversion due to it. Subversion also stores all xml files by default as binary and in a large spring based application with a long history this might have been a problem. We were ready to investigate solutions like git-annex. All test migrations though showed that it was not a problem, git clones of this large codebase were super fast, and considerably smaller (subversion 4.1G -> git 1.1G).

Adaptation

Towards the end of February we were lucky enough to have Tim Berglund, Brent Beer, and David Graham, from GitHub come and teach us Git. The first two days was a set course with 75 participants and covered

Git Fundamentals (staging, moving, copying, history, branching, merging, resetting, reverting),
Collaboration using GitHub (Push, pull, and fetch, Pull Requests Project Sites, Gists, Post-receive hooks), and
Advanced Git (Filter-Branch, Bisect, Rebase-onto, External merge/diff tools, Event Hooks, Refspec, .gitattributes).

The third day with the three GitHubbers was more of an open space with under twenty participants where we discussed various specifics to FINN’s adoption to Git, from continuous deployment (which GitHub excels at) to branching workflows.

No doubt about it this was one of the best, if not very best, courses held for FINN developers, and left everyone with a resounding drive to immediately switch all codebases over to Git.

Other documentation that’s encouraged for everyone to read/watch is

The entire Pro Git book,
Advanced Git video from Tim Berglund,
The Flow Of Change video from Google (covers engineering principles of branching codebases).

Tips and tricks for beginners…

To wrap it up here’s some of the tips and tricks we’ve documented for ourselves…

Cant push because HEAD is newer So you pull first… Then you go ahead and push which adds two commits into history: the original and a duplicate merge from you. You need to learn to do git rebase in such situations, better yet to do git --rebase pull. You can make the latter permanent behaviour with ` git config –global branch.master.rebase true git config –global branch.autosetuprebase always`

Colour please! git config --global color.ui auto

Did you really want to push commits on all your branches? This can trap people, they often expect push to be restricted to the current branch your on. It can be enforced to be this way with git config --global push.default tracking

Pretty log display Alias git lol to your preferred log format… Simple oneline log formatting: git config --global alias.lol "log --abbrev-commit --graph --decorate --all --pretty=oneline"

Oneline log formatting including committer’s name and relative times for each commit: git config --global alias.lol "log --abbrev-commit --graph --all --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset'"

Compact log formatting with full commit messages, iso timestamps, file history over renames, and mailmap usernames: git config --global alias.lol "log --follow --find-copies-harder --graph --abbrev=4 --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %Cgreen%ai %n %C(bold blue)%aN%Creset %B'"

Cherry-picking referencing the original commit When recording a cherry-pick commit, using the “-x” option appends the line “(cherry picked from commit …)” to the original commit message so to indicate which commit this change was cherry-picked from. For example git cherry-pick -x 3523dfc

Quick log to see what i’ve done in the last 24 hours? git config --global alias.standup "log --all --since yesterday --author git config –global user.email"

What files is this project got but ignoring? git config --global alias.ignored "ls-files --others --i --exclude-standard"

Wipe all uncommitted changes git config --global alias.wipe "reset --hard HEAD"

Edit and squash commits before pushing them git config --global alias.ready "rebase -i @{u}"

I wish I knew my consumers - Maven Reverse Dependency

2013-01-31T12:49:05+00:00

At FINN.no being a developer fixing bugs in a library is a breeze. Getting every user of your library to use the fix, however, is a different story. How to know who to notify? I mean, I know my library’s dependencies, but who “out there” has dependency to the component where I just fixed a bug? I wish. Enter maven-dependency-graph.

The idea was born on the plane back home from a Copenhagen hosted conference. Graph database. Download neo4j and start dabbling at a maven plugin. Flying time Copenhagen - Oslo was too short, all of a sudden.

From there, the idea slept for a couple of years. Until the need arose somewhere among the developers. With 100+ different applications running with common core services and libraries, everybody suddenly needed to know who depended on their code which had recently been bugfixed. So the old idea was dusted off and once more saw the light of day. We needed to upgrade the server installation and the API to neo4j - which took some time to grasp; but after some playing around, it became obvious and easy.

The idea was to have every project report its dependencies to a graph database, building the tree of dependencies on each commit. This constitutes one half of the plugin. Over time, all projects will have reported their dependencies, and from there on part two of the plugin comes into use. It will examine the reverse dependencies to the current maven project, and report all incoming dependencies to it in the maven log. Hey, presto! We now know who out there uses us! And even which version they are using, thanks to two different keys into the built-in lucene index engine.

The plugin is published on github @ Finn Technology’s account. Feel free! @gardleopard and @roarjoh

Usage examples

Dependencies to current maven project:

mvn no.finntech:dependency-mapper-maven-plugin:read
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building greenpages thrift-client 3.4.5-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- dependency-mapper-maven-plugin:1.0-SNAPSHOT:read (default-cli) @ commons-thrift-client ---
[INFO] Resolving reverse dependencies
[INFO] no.finntech.travel.supplier:supplier-client:1.2-SNAPSHOT -> no.finntech:commons-thrift-client:3.1.1
[INFO] no.finntech.cop:client:1.1-SNAPSHOT -> no.finntech:commons-thrift-client:3.1.1
[INFO] no.finntech.oppdrag-services:iad-model:2013.2-SNAPSHOT -> no.finntech:commons-thrift-client:3.4.3
[INFO] no.finntech:minfinn:2013.2-SNAPSHOT -> no.finntech:commons-thrift-client:3.4.3
[INFO] no.finntech:service-user:2013.2-SNAPSHOT -> no.finntech:commons-thrift-client:3.4.3
[INFO] no.finntech:service-oppdrag:2013.2-SNAPSHOT -> no.finntech:commons-thrift-client:3.4.3
[INFO] no.finntech:kernel:2013.2-SNAPSHOT -> no.finntech:commons-thrift-client:3.4.3
`
(umpteen lines skipped...)
`
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.957s
[INFO] Finished at: Thu Jan 31 09:50:19 CET 2013
[INFO] Final Memory: 9M/211M
[INFO] ------------------------------------------------------------------------

Usage of third party framework (using neo4j’s included admin interface):

StrataConf & Hadoop World 2012…

2012-11-09T13:23:06+00:00

A summary of this year’s Strataconf & Hadoop World. A fascinating and inspiring conference with use-cases on both sides of an ethical divide – proof that the technologies coming are game-changers in both our industry and in society. Along with some intimidating use-cases i’ve never seen such recruitment efforts at any conference before, from multi-nationals to the CIA. The need for developers and data scientists in Big Data is burning – the market for Apache Hadoop Market is expected to reach $14 billion by 2017.

Plenty of honesty towards the hype and the challenges involved too. A barcamp Big Data Controversies labelled it all as Big Noise and looked at ways through the hype. It presented balancing perspectives from a insurance company’s statistician who has dealt successfully with the problem of too much data for a decade and a hadoop techie who could provide much desired answers to previously impossible questions. Highlights from this barcamp were…

One should always use intelligent samples before ever committing to big data.
Unix tools can be used but they are not very fault tolerant.
You know when you’re storing too much denormalised data when you’re also getting high compression rates on it.
MapReduce isn’t everything as it can be replaced with indexing.
If you try to throw automated algorithms at problems without any human intervention you’re bound to bullshit.
Ops hate hadoop and this needs to change.
Respecting user privacy is important and requires a culture of honesty and common-sense within the company. But everyone needs to understand what’s illegal and why.

Noteworthy (10 minute) keynotes…

* The End of the Data Warehouse. They are monuments to the old way of doing things: pretty packaging but failing to deliver the business value. But Hadoop too is still flawed… Also a blog available.

Moneyball for New York City. How NYC council started combining datasets from different departments with surprising results.
The Composite Database, a focus on using big data for product development. To an application programmer the concept of a database is moving from one entity into a componential architecture.
Bringing the ‘So What’ to Big Data, a different keynote with a sell towards going to work for the CIA. Big data isn’t about data but changing and improving lives.
Cloud, Mobile and Big Data. Paul Kent, a witty speaker, talks about analytics in a new world. “At the end of the day, we are closer to the beginning than we are at end of this big data revolution… One radical change hadoop and m/r brings is now we push the work to the data, instead of pulling the data out.”

Noteworthy (30 minute) presentations…

* The Future – Hadoop-2. Hadoop YARN makes all functional programming algorithms possible, reducing the existing map reduce framework to just one of many user-land libraries. Many existing limitations are removed. Fault-tolerance is improved (namenode). 2x performance improvement on small jobs.

Designing Hadoop for the Enterprise Data Center. A joint talk from Cisco and Cloudera on hardware tuning to meet Hadoop’s serious demands. 10G networks help. dual-attached 1G networks is an alternative. More jobs in parallel will average out network bursts. Data-locality misses hurt network, consider above a 80% data-locality hitrate good.
How to Win Friends and Influence People. LinkedIn presents four of their big data products ∘ Year in Review. Most successful email ever – 20% response rate. ∘ Network Updates. ∘ Skills and Endorsements. A combination of propensity to know someone and the propensity to have the skill. ∘ People You May Know. Listing is easy, but ranking required combining many datasets.  All these products were written in PIG. Moving data around is the key problem. Kafka is used instead of scribe.
Designing for data-driven organisation. Many companies who think they are data-driven are in fact metrics-driven. It’s not the same thing. Metrics-driven companies often want interfaces with less data. Data-driven companies have data rich interfaces presenting holistic visualisations.
Visualizing Networks. The art of using the correct visualisation and layout. Be careful of our natural human trait to see visual implications from familiarity and proximity – we don’t always look at the legend. A lot of examples using the d3 javascript library.

The two training sessions I attended were Testing Hadoop, and Hadoop using Hive. Testing Hadoop. Presented by an old accomplice from the NetBeans Governance board, Tom Wheeler. He presented an interesting perspective on testing calling it another form of computer security: “a computer is secure if you can depend on it and its software to behave as you expect”. Otherwise i took home a number of key technologies to fill in the gaps between and around our current unit and single-node integration tests on our CountStatistics project: Apache MRUnit for m/r units, MiniMRCluster and MiniDFSCluster for multi-jvm integration cluster, and BigTop for ecosystem testing (pig, hive, etc). We also went through various ways to benchmark hadoop jobs using TeraSort, MRBench, NNBench, TestDFSIO, GridMix3, and SWIM. Lastly we went through a demo of the free product “Cloudera Manager” – a diagnostics UI similar to Cassandra’s OpsCenter.

Hadoop using Hive. Hive provides an SQL interface to Hadoop. It works out-of-the-box if you’re using HBase but with Cassandra as our underlying database we haven’t gotten around to installing it yet. The tutorial went through many live queries on a AWS EC2 cluster, exploring the specifics to STRUCTs in schemas, JOINs, UDFs, and serdes. This is a crucial interface to make it easier for others, particularly BI and FINN økosystem, to explore freely through our data. Pig isn’t the easiest way in for outsiders, but everyone knows enough SQL to get started. Fingers crossed we get Hive or Impala installed some time soon…

A number of meet-ups occurred during the evenings, one hosted at AppNexus, a company providing 10 billion real-time ads per day (with a stunning office). AppNexus does all their hadoop work using python, but they also putting focus on RESTful Open APIs like we do. The other meetup represented Cassandra by DataStax with plenty of free Cassandra beer. Latest benchmarks prove it to be the fastest distributed database around. I was hoping to see more Cassandra at strataconf – when someone mentions big data i think of Cassandra before Hadoop.

Otherwise this US election was on the news as the big data election

Leaving the Tower of Babel

2012-09-04T09:48:05+00:00

Here at FINN we use mostly Java for our day-to-day work, but we do have some applications and modules written in other languages. We currently have code in Java, Ruby, JavaScript, Objective-C and Scala, supporting things as diverse as testing frameworks and iOS-apps.

Recently there has been some discussion with regards to what we should be using for new projects, as each team of developers start suggesting that they should use something they think would be easier and more effective. The arguments against are based on how easy it would be to recruit new developers who know these new technologies, how well it would perform under our kinds of loads, and wheter the assumption that we can be more effective is actually true.

Our strategy is firm on this point, that experiments can be done, but for major applications serving the public, we should be using Java for the time being. We are however considering if it’s time to take a second look at our chosen languages, and see if we should expand our portfolio. This strategy is founded on a wish to reduce the size of our technology portfolio, so that we don’t have to spend time and effort to bring developers up to speed when they switch teams. There’s also a higher chance that new hires will be familiar with Java, while other technologies would often require a period of training when they first start.

During these discussions, we created a quick poll and sent it out on our internal discussionboard. The poll asked questions like “How long have you worked as a developer?”, “Which language do you use for your day-to-day work?”, “If you were free to chose, which language would you use for your next project?” and “Which programming languages do you know?”. Our definition of “know” was very open in this poll, lowering the bar to allow people who had maybe written a single example program, and understands a bit of code could check the box.

Out of the nearly 100 people that work with development, 56 answered. We can assume that the people who did answer, were the people who were interested in the topic, and might not be a representative selection, but the results are interesting none the less.

So.. what did we learn?

Experience

Being a poll made in the midst of a discussion, mostly for fun, this part of the poll wasn’t framed in a way that gives us much useful information. We can say that the average developer who answered has worked for around seven or eight years, but with what looks like a reasonably fair spread across all groups. Almost as many with a couple years experience, as there are people with 10 to 15 years.

Primary language in day-to-day work

Being a predominatly Java shop, most of the respondents were using Java for their daily work. 35 out of 56 were Java-developers. 11 respondents worked with JavaScript daily, while four worked with Scala, three with Ruby, two with Objective-C, and finally one database-developer who worked mostly in T-SQL (The SQL-variant used in Sybase).

Most popular choice if allowed to chose freely

The most interesting part of the poll, with many interesting insights. This is also the most loaded part, as the results could easily short-circuit any discussion about future choice in language.

The bad news (or good, depending on your point of view), is that there was no clear answer from this section. Keep in mind that around 40 people didn’t answer this poll, and could be assumed to be content with the current status-quo.

The biggest group was Java, not surprisingly. The surprising part is that it was only 16 people who would chose Java if they could chose freely. This is much lower than expected, but still make up the largest group. Of these, 13 people are already using Java today. 20 Java developers would like to use something else.

So, if Java was the largest group, at 16 people, what does that mean for the remainder of the results? Well, 10 people would have chosen Scala, while JavaScript, Ruby, Clojure and Groovy clock in at about four or five respondents wanting to use them. At the bottom of the pack we have Python and Objective-C with three and two respondents chosing them.

Six people were interested enough to answer the poll, but chose the non-commitant “I don’t care, as long as I’m making good stuff for our users” option.

The number of possible answers here was a rather large selection of popular and not-so-popular-but-quite-well-known languages, so the fact that the list only includes eight languages is a sign that it’s not a completely random selection. Still, quite a lot of discussion is needed before any one of those languages gets center stage.

A couple curious details found in this part of the poll includes the fact that the only people who would chose JavaScript are people who are already working in JavaScript every day. Groovy, Clojure, Objective-C and Scala have all managed to be chosen by a person who doesn’t actually know the langauge. There’s only three people who would switch to Java, if allowed to chose.

Which languages do we “know”?

As mentioned earlier, the bar for “knowing” a language was set quite low in this poll, to get more diversity in answers. This resulted in some interesting numbers.

Being primarily a Java-shop, it might not surprise anyone that a full 100% knows Java. A little over three fourths know JavaScript, while Ruby and T-SQL was known to about half. These are the main languages most of us have some sort of dealings with in our daily work, so that they score high is not surprising.

Next on the list is PHP and Python, with around 40% knowing them.

We run primarily Sybase for our databases, with a small number of MySQL and PostgreSQL servers for smaller applications and modules. We are trying to standardise on PostgreSQL, but this isn’t reflected in knowledge, with 37.5% knowing how to code MySQL-procedures, 35.7% knowing how to code for Oracle, and only 25% knowing their way around a PostgreSQL codebase.

Our operations-department have had a “vendetta” against Perl for several years, but there are still more people who knows Perl than there are people who knows Scala, with the score 18 to 15. Groovy has quite a following too, with 13 respondents knowing Groovy.

We have a small Apps team working with iOS-development, but Objective-C has a reach far outside that team, with 10 respondents knowing Objective-C.

Common Lisp is known to five respondents, while Clojure suprisingly was only known to seven respondents. As you can read elsewhere in this blog, we had a Clojure workshop at our Technology Day earlier this summer, and these results might be telling us something about the language or our teachers when it didn’t have a better ability to “stick” than this.

We also have people who know some Erlang, Smalltalk, Lua, Haskell, Scheme, Eiffel and ML/SML.

How many languages do we “know”?

On average, we know 6.8 languages each. The most knowledgeable person knows 16 languages, while the least knowledgeable knows only one. The high experience respondents, 16 years or more, have a higher average than the rest of us, with 9.4, while the rest of the “experience brackets” know somewhere between six and seven languages.

Several programmer-gurus seem to think you should stribe to teach yourself one new language every year, and it would seem atleast one of our respondents have been able to follow this advice. For those of us with less extra time, a less ambitious strategy might be more compatible, but that we should stribe to learn something new every once in a while seems to be good advice.

Now what?

As mentioned previously, this poll was not a serious attemt at gaining actionable insights. The results should be taken with a large helping of salt, and at most used as a basis for discussions. On the other hand, I know we will be discussing this topic going forward, because the fact that only 16 of 56 wanted to use Java is telling us it’s time to start the discussion. There are possibly 40 developers who would prefer Java, who didn’t respond, so it’s too early to draw conclusions, but we have a place to start.

There are other questions falling out of this too:

There are two developers who wants to work with Objective-C, and eight Java-developers who wants to work with Scala, so why do we have so few internal applicants when we have openings on the teams working with these languages?
What is it that makes developers want to work with languages they don’t even know?
Of the respondents, more than half the Java-developers wants to use something else. Why is that?

Foraging in the landscape of Big Data

2012-08-06T10:26:00+00:00

This is the first article describing FINN.no’s coming of age with Big Data. It’ll cover the challenges arising the need for it, the challenges in accomplishing new goals with new tools, and the challenges that remain ahead.

Big Data is for many just another vague and hyped up trend getting more than its far share of attention. The general definition, from Wikipedia, is big data covers the scenario where existing tools fail to process the increasing amount or dimensions of data. This can mean anything from:

α – the existing tools being poor (while large companies pour $$$ into scaling existing solutions up) or
β – the status quo requiring more data to be used, to
γ – a requirement for faster and more connected processing and analysis on existing data.

The latter two are also described as big data’s three V’s: Volume, Variety, Velocity. If the theoretical definition isn’t convincing you put it into context against some of today’s big data crunching use-cases…

online advertising combining content analysis with behavioural targeting,
biomedical’s DNA sequence tagging,
pharmaceutics’s metabolic network modelling,
health services detecting disease/virus spread via internet activity & patient records,
the financial industry ranging from credit scores at retail level to quant trading,
insurance companies crunching actuarial data,
US defence programs for offline (ADAMS) and online (CINDER) threat detection,
environmental research into climate change and atmospheric modelling, and
neuroscience research into mapping the human brain’s neural pathways.

On the other hand big data is definitely no silver bullet. It cannot give you answers to questions you haven’t yet formulated (pattern recognition aside), and so it doesn’t give one excuses to store overwhelming amounts of data where the potential value in it is still undefined. And it certainly won’t make analysis of existing data sets initially any easier. In this regard it’s less to do with the difficulty of achieving such tasks and more to do with the potential to solve what was previously impossible.

Often companies can choose their direction and the services they will provide, but in any competitive market where one fails to match the competitor’s offerings it can result in the fall of that company. Here Big Data earns its hype with many a CEO concerned into paying attention. And it probably gives many CEOs a headache as the possibilities it opens, albeit tempting or necessary, create significant challenges in and of themselves. The multiple vague dimensions of big data also allows the critics plenty of room to manoeuvre.

One can argue that to scale up: to buy more powerful machines or to buy more expensive software solves the problem (α). If all you have is this problem then sure it’s a satisfactory solution. But ask yourself are you successfully solving today’s problem while forgetting your future?

“If we look at the path, we do not see the sky.” – Native American saying

One can also argue away the need for such vasts amounts of data (β). Through various strategies: more aggressive normalisation of the data, storing data for shorter periods, or persisting data in more ways in different places; the size of each individual data set can be significantly reduced. Excessively normalising data has its benefits and is what one may do in the Lean development approach for any new feature or product. Indeed a simpler datamodel trickles through into a simpler application design, in turn leaving more content, more productive, pragmatic developers. Nothing to be scoffed at in the slightest. But in this context the Lean methodology isn’t about any one state or snapshot in time illustrating the simple and the minimalistic, rather it’s about evolution, the processes involved and their direction. Much like the KISS saying: it’s not about doing it simple stupid but about keeping it simple stupid. Here it questions to how does overly normalised data evolve as its product becomes successful and something more complicated over time. Anyone who has had to deal awkwardly with numerous and superfluous tables, joins, and indexes in a legacy application because it failed over time to continually improve its datamodel due to needs of compatibility knows what we’re talking about. There is another problem from such legacy applications that follow a datamodel centric design: the datamodel itself becomes a public API and the many consumers of it create this need of compatibility and the resulting inflexible datamodel. But this isn’t an underlying but rather overlapping problem as one loses oversight of the datamodel in its completeness and in a way represented optimally.

It also difficult to deny the amount of data we’re drowning in today. “90% of the data that exists today was created in the last two years.. the sheer volume of social media and mobile data streaming in daily, businesses expect to use big data to aggregate all of this data, extract information from it, and identify value to clients and consumers.. Data is no longer from our normalised datasets sitting in our traditional databases. We’re accessing broader, possibly external, sources of data like social media or learning to analyse new types of data like image, video, and audio..” – greenbookblog.org.

And one may also argue that existing business intelligence solutions (can) provide the analysis required from all already existing datasets (γ). It ignores a future of possibilities: take for example the research going into behavioural targeting giving glimpses into the challenges of modern marketing as events and trends spark, shift, and evolve through online social media with ever faster frequencies – just the tip of the iceberg when one thinks forward to being able to connect face and voice recognition to emotional pattern matching analysis. But it also defaults to the conservative opinion that business intelligence need be but a post-mortem analysis of events and trends so to provide the insights intended and required only for internal review. This notion that this large scale analysis is of sole benefit to company strategy must become the tell tale of companies failing to see how users and the likes of social media change dramatically what is possible in product development today.

The methodologies of innovation therefore change. Real time analysis of user behaviour plays a forefront role in decisive actions on what product features and interfaces will become successful. This is the potential to cut down the risk of product development’s internal guesswork for a product’s popularity at any given point in time. In turn this cuts down time-to-market bringing the products release date closer to its maximum popularity potential moment. Startup companies know that a success doesn’t come only from a clever designed product, there is a significant factor of luck involved in releasing the right product at the right time. Large, and very costly, marketing campaigns can only extend, or synthetically create, this potential moment by so much.

This latter point around the extents and performance of big data analysis and the differentiation it creates between business analysis versus richer, more spontaneous, product development and innovation creates for FINN an important and consequential factor to our forage into big data. Here at FINN a product owner recently said: “FINN is already developing its product largely built around the customer feedback and therefore achieving continuous innovation?”. Of course what he meant to say was “from numbers we choose to collect, we generate the statistics and reports we think are interesting, and from these statistics we freely interpret them to support the hypothesis we need for further product development…” We couldn’t be further from continuous innovation if it hitched a lift to the other side of Distortionville¹.

“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts” – Arthur Conan Doyle

FINN isn’t alone here, I’d say most established companies while trying to brand themselves as innovative are still struggling to break out of their existing models of product development based upon internal business analysis. No one said continuous innovation was easy, and there’s a lot of opinions out on this, but along with shorter deployment cycles i reckon there’s two keywords: truth and transparency. Tell your users the truth and watch how they respond. For example give them the statistics that show them their ads stop getting visits after a week, and then observe how they behave, to which solution do they flock to regain traffic to their ads. Don’t try and solve all their problems for them, rather try to enable them. You’ll probably make more money out of them, and by telling them the truth you’ve removed a vulnerability, or to what some fancy refers to as “a business secret”.

There’s also a potential problem with organisational silos. Large companies having invested properly in business intelligence and data warehousing will have assigned these roles of data collection, aggregation, and analysis, to a separate team or group of experts typically trained database administrators, statisticians, and traffic analysts. They are rarely the programmers, the programmers are on the front lines building the product. Such a split can parallel the sql vs nosql camps. This split against the programmers whom you rely on to make continuous innovation a reality can run the risk of stifling any adoption of big data. With the tools enabling big data the programmers can generate reports and numbers previously only capable from the business intelligence and data warehousing departments, and can serve them to your users at web response times – integrating such insights and intelligence into your product. Such new capabilities doesn’t supersede these traditional departments, rather it needs everyone accepting the new: working together to face new challenges with old wisdom. The programmers working on big data, even if tools and data become shared between these two organisational silos, cannot replace the needs of business intelligence any more than business intelligence can undertake big data’s potential. As data and data sources continue to increase year after year the job of asking the right questions, even knowing how to formulate the questions correctly, needs all hands on deck. Expecting your programmers to do it all might well swamped them into oblivion, but it isn’t just the enormity of new challenges involved, it’s that these challenges have an integral nature to them that programmers aren’t typically trained to tackle. Big Data can be used as an opportunity not only to introduce exciting new tools, paradigms, and potential into the company but as a way to help remove existing organisational silos.

The need for big data at FINN came from a combination of (α) and (γ). The statistics we show users for their ads had traditional been accumulated and stored in a sybase table. These statistics included everything from page views, “tip a friend” emails sent, clicks on promoted advertisement placements, ads marked as favourite, and whatever else you can imagine.

(α) FINN is a busy site, the busiest in Norway, and we display ~50 million ad pages each day. Like a lot of web applications we had a modern scalable presentation and logic tier based upon ten tomcat servers but just one not-so-scalable monster database sitting in the data tier. The sybase procedure responsible for writing to the statistics table ended up being our biggest thorn. It used 50% of the database’s write execution time, 20% of total procedure execution time, and the overall load it created on the database accounted for 30% of ad page performance. It was a problem we had lived with long enough that Operations knew quickly to turn off the procedure if the database showed any signs of trouble. Over one period of troublesome months Operations wrote a cronjob to turn off the procedure automatically during peak traffic hours – when ads were receiving the most traffic we had to stop counting altogether, embarrassing to say the least!

(γ) On top of this product owners in FINN had for years been requesting that we provide statistics on a day basis. The existing table had tinkered with this idea for some of the numbers that didn’t accumulate so high, eg “tip a friend emails”, but for page viewings this was completely out of the question – not even the accumulated totals were working properly.

At the time we were in the process of modularising the FINN web application. The time was right to turn statistics into something modern and modular. We wanted an asynchronous, fault-tolerance, linearly scaling, and durable solution. The new design uses the Command-Query Separation pattern by using two separate modules: one for counting and one for displaying statistics. The counting achieves asynchronousity, scalability, and durability by using Scribe. The backend persistence and statistics module achieves all goals by using Cassandra and Thrift. As an extension of the push-on-change model: the counting stores denormalised data and it is later normalised to the views the statistics module requires; we use MapReduce jobs within a Hadoop cluster.

The resulting project is kick-ass, let me say. Especially Cassandra, it is a truly amazing modern database: linear scalability, decentralised, elastic, fault-tolerant, durable; with a rich datamodel that provides often superior approaches to joins, grouping, and ordering than traditional sql. But we’ll spend more time describing the project in technical details in a later article.

A challenge we face now is broader adoption of the project and the technologies involved. Various departments: from developers to tech support; want to read the data, regardless if it is traditional or ‘big data’, and the habit was always to read it directly from production’s Sybase. And it’s a habit that’s important in fostering a data-driven culture within the company, without having to encourage datamodel centric designs. With our Big Data solution this hasn’t been so easy. Without this transparency to the data, developers, tech support, and product owners alike seem to be failing to initiate further involvement - to solve this, since our big data is stored in Cassandra, we’d love to see a read only web-based gui interface based off caqel. …to be continued…

References:

• DNA sequencing is ultimately a big data problem. • Obama’s 2012 Big Data initiatives • Beware The Hype Over Big Data Analytics • Big Data won’t solve your company’s problems • Big Data: Opportunity or Threat for Market Research? • O’Reilly MySQL CE 2010: Mark Atwood, “Guide to NoSQL, redux” • Lean Startup • FINN moving toward continuous innovation • Will data warehousing survive the advent of big data? • Why are Facebook, Digg, and Twitter so hard to scale? • Do You Really Need SQL to Do It All in Cassandra?

the ultimate view — Tiles-3

2012-07-25T13:48:52+00:00

A story of getting the View layer up and running quickly in Spring…

Since the original article, parts of the code has been accepted upstream, now available as part of the Tiles-3 release, so the article has been updated — it's all even simpler!

Based upon the Composite pattern and Convention over Configuration we'll pump steroids into
     a web application's view layer
          with four simple steps using Spring and Tiles-3
               to make organising large complex websites elegant with minimal of xml editing.

Contents of article… • Background • Step 0: Spring to Tiles Integration • Step 1: Wildcards • Step 2: The fallback pattern • Step 3: Definition includes • When the Composite pattern is superior • Conclusion

Background

At FINN.no we were redesigning our control and view layers. The architectural team had decided on Spring-Web as a framework for the control layer due to its flexibility and for providing us a simpler migration path. For the front end we were a little unclear. In a department of ~60 developers we knew that the popular vote would lead us towards SiteMesh. And we knew why – for practical purposes sitemesh gives the front end developer more flexibility and definitely less xml editing. But sitemesh has some serious shortcomings...

SiteMesh shortcomings:

from a design perspective the Decorator pattern can undermine the seperation MVC intends,
requires all possible html for a request in buffer requiring large amounts of memory
unable to flush the response before the response is complete,
requires more overall processing due to all the potentially included fragments,
does not guaranteed thread safety, and
does not provide any structure or organisation amongst jsps, making refactorings and other tricks awkward.

One of the alternatives we looked at was Apache Tiles. It follows the Composite Pattern, but within that allows one to take advantage of the Decorator pattern using a ViewPreparer. This meant it provided by default what we considered a superior design but if necessary could also do what SiteMesh was good at. It already had integration with Spring, and the way it worked it meant that once the Spring-Web controller code was executed, the Spring’s view resolver would pass the model onto Tiles letting it do the rest. This gave us a clear MVC separation and an encapsulation ensuring single thread safety within the view domain.

“Tiles has been indeed the most undervalued project in past decade. It was the most useful part of struts, but when the focus shifted away from struts, tiles was forgotten. Since then struts as been outpaced by spring and JSF, however tiles is still the easiest and most elegant way to organize a complex web site, and it works not only with struts, but with every current MVC technology.” – Nicolas Le Bas

Yet the best Tiles was going to give wasn’t realised until we started experimenting a little more…

Step 0: Spring to Tiles Integration

The first step is integrating Tiles and Spring together. For Tiles-3 it boils down to registering a ViewResolver and a TilesConfigurer in your spring-web configuration.

<bean id="viewResolver" class="org.springframework.web.servlet.view.tiles3.TilesViewResolver"/>  
<bean id="tilesConfigurer" class="org.springframework.web.servlet.view.tiles3.TilesConfigurer">  
<property name="tilesInitializer">  
<bean class="no.finntech.control.servlet.tiles.FinnTilesInitialiser"/>  
</property>  
</bean>

You need Spring-3.2 to get this to work, specifically "spring-webmvc". If you're using an older version of Spring then you can download the required classes separately and add them to your classpath. (There's a pom file also available.) The TilesConfigurer hooks in the "tilesInitializer" class. This is the way in Tiles-3 of providing the configuration. If you don't specify one it'll use by default CompleteAutoloadTilesInitializer which gives you a fully featured Tiles setup (you'll need tiles-extras.jar in the classpath). The setup here uses the FinnTilesInitialiser and FinnTilesContainerFactory classes to make possible further configurations. For now FinnTilesContainerFactory is an empty class extending BasicTilesContainerFactory, and FinnTilesInitialiser looks like

public class FinnTilesInitialiser extends AbstractTilesInitializer {

    public FinnTilesInitialiser() {}

    @Override
    protected AbstractTilesContainerFactory createContainerFactory(TilesApplicationContext context) {
        return new FinnTilesContainerFactory();
    }
}

Step 1: Wildcards

We are going to create a basic website with three simple pages: cat, cow, and dog. The jsps making up these pages are shown to the right. Composing these jsps together into their corresponding pages we encounter Tiles' most obvious downfall: every single jsp included must be declared in a definition in the tiles.xml file. Anyone that ever tried Tiles-1 knows this bad smell. Since Tiles-2 this can be avoided by using wildcards¹, and when hearing people talk about tiles it is often clear this dynamic composite paradigm hasn't yet superseded the old tiles prejudices.
Let's declare a definition and template as below and create some files and folders as shown to the right:

//--tiles.xml
//--  anything that doesn't start with a slash is considered a definition here.
<definition name="REGEXP:([^/].*)" template="/WEB-INF/tiles/template.jsp">
  <put-attribute name="meta" value="/WEB-INF/tiles/{1}/meta.jsp"/>
  <put-attribute name="header" value="/WEB-INF/tiles/{1}/header.jsp"/>
  <put-attribute name="body" value="/WEB-INF/tiles/{1}/body.jsp"/>
  <put-attribute name="footer" value="/WEB-INF/tiles/{1}/footer.jsp"/>
</definition>
//--template.jsp

<!DOCTYPE html>
<%@ prefix="tiles" taglib uri="http://tiles.apache.org/tags-tiles" %>
<html>
    <head>
        <tiles:insertAttribute name="meta"/>
    </head>
    <body>
        <div id="header"><tiles:insertAttribute name="header"/></div>
        <div id="body"><tiles:insertAttribute name="body"/></div>
        <div id="footer"><tiles:insertAttribute name="footer"/></div>
    </body>
</html>

Write a Spring controller to return definitions that match the names of the folders, that is the three definitions "cat", "dog", and "cow".

@Controller
public class MyWebsite{

    @RequestMapping("/cat")
    public ModelAndView viewCat(){
        Map modelMap = new HashMap();
        //...
        return new ModelAndView("cat", modelMap);
    }

    @RequestMapping("/dog")
    public ModelAndView viewDog(){
        Map modelMap = new HashMap();
        //...
        return new ModelAndView("dog", modelMap);
    }

    @RequestMapping("/cow")
    public ModelAndView viewCow(){
        Map modelMap = new HashMap();
        //...
        return new ModelAndView("cow", modelMap);
    }
}

Now with this setup we can keep adding new definitions and jsps without having to edit xml.
The Apache Tiles tutorial on wildcards explains that to extend the use of wildcards to also allow rich regular expressions one should use the CompleteAutoloadTilesContainerFactory. In this article we are building up our own FinnTilesInitialiser and FinnTilesContainerFactory configuration classes. To enable both wilcards and regular expressions, distinguished by the use of prefixes, we need to override the following method in FinnTilesContainerFactory…

    @Override
    protected <T> PatternDefinitionResolver<T> createPatternDefinitionResolver(Class<T> cls) {
        PrefixedPatternDefinitionResolver<T> r = new PrefixedPatternDefinitionResolver<T>();
        r.registerDefinitionPatternMatcherFactory(
                       "WILDCARD", new WildcardDefinitionPatternMatcherFactory());
        r.registerDefinitionPatternMatcherFactory(
                       "REGEXP", new RegexpDefinitionPatternMatcherFactory());
        return r;
    }

Troubleshooting: remember to include the tiles-extras dependency! Without it you'll get ...CompatibilityDigesterDefinitionsReader cannot be cast to ...Definition

Step 2: The fallback pattern

Not needing anymore to edit tiles.xml is nice, but as the number of definitions grows the number of duplicated jsps with also grow. For example maybe footers and headers are identical in nearly every definition. Here we'll use the OptionsRenderer to provide a desired fallback pattern. In the xml a fallback is notated with the syntax ${options[myopts]} where "myopts" is a list-attribute that references the options in preferential order. Introduce the "common" folder, as shown to the right, and the example turns into

//--tiles.xml
<definition name="REGEXP:([^/].*" template="/WEB-INF/tiles/template.jsp">
    <put-attribute name="meta" value="/WEB-INF/tiles/${options[myopts]}/meta.jsp"/>
    <put-attribute name="header" value="/WEB-INF/tiles/${options[myopts]}/header.jsp"/>
    <put-attribute name="body" value="/WEB-INF/tiles/${options[myopts]}/body.jsp"/>
    <put-attribute name="footer" value="/WEB-INF/tiles/${options[myopts]}/footer.jsp"/>


    <put-list-attribute name="myopts" cascade="true">
        <add-attribute value="{1}"/>
        <add-attribute value="common"/>
    </put-list-attribute>
</definition>

To configure this OptionsRenderer to work in your TilesContainerFactory you'll also need to do the following

public class FinnTilesContainerFactory extends BasicTilesContainerFactory {
    ...
    @Override
    protected Renderer createTemplateAttributeRenderer(BasicRendererFactory rendererFactory, ApplicationContext applicationContext, TilesContainer container, AttributeEvaluatorFactory attributeEvaluatorFactory) {

        Renderer original = super.createTemplateAttributeRenderer(rendererFactory, applicationContext, container, attributeEvaluatorFactory);
        OptionsRenderer optionsRenderer = new OptionsRenderer(applicationContext, original);
        return optionsRenderer;
    }
    ....
}

As the system developer writes new spring controllers to create completely new definitions the front end developer, with this new Convention over Configuration setup, only needs to create the new folder and jsps for fragments they wish to customise. Again there's no further xml editing.
No more xml editing and no duplicate JSPs.

For a large company this can help enforce UI standards by having control over the common folder — keeping an eye on UI overrides and customisations never become too outlandish with the standard look of the website. The front end developer can also be referencing this common folder for UI standards.

Step 3: Definition includes

Realistically these fragments: meta, header, body, and footer; won't be enough. You can quickly be using scores of fragments for columns within the body, banners and advertisements, analytics, seo meta data, css and javascript meta includes, etc, etc. With these different pages cat, dog, and cow it'll also be pretty certain that they'll have different "actions" applicable upon them. For example viewing, editing, and searching pages. Once again there'll be a lot of duplicate jsps. By being able to dynamically inject one tiles definitions into another, referred to here as "definition injection", this can all elegantly handed. First of all change the existing definition names into the form "action.category" making the controllers look like

    @Controller
    @RequestMapping("/cat")
    public class CatController{
        @RequestMapping("/cat/view")
        public ModelAndView viewCat(){ //...
            return new ModelAndView("view.cat", modelMap);
        }
        @RequestMapping("/cat/edit")
        public ModelAndView editCat(){ //...
            return new ModelAndView("edit.cat", modelMap);
        }
        @RequestMapping("/cat/search")
        public ModelAndView searchCat(){ //...
            return new ModelAndView("search.cat", modelMap);
        }
    }
    @Controller
    @RequestMapping("/dog")
    public class DogController{
        @RequestMapping("/dog/view")
        public ModelAndView viewDog(){/ /...
            return new ModelAndView("view.dog", modelMap);
        }
        @RequestMapping("/dog/edit")
        public ModelAndView editDog(){ //...
            return new ModelAndView("edit.dog", modelMap);
        }
        @RequestMapping("/dog/search")
        public ModelAndView searchDog(){ //...
            return new ModelAndView("search.dog", modelMap);
        }
    }
    @Controller
    @RequestMapping("/cow")
    public class CowController{
        @RequestMapping("/cow/view")
        public ModelAndView viewCow(){ //...
            return new ModelAndView("view.cow", modelMap);
        }
        @RequestMapping("/cow/edit")
        public ModelAndView editCow(){ //...
            return new ModelAndView("edit.cow", modelMap);
        }
        @RequestMapping("/cow/search")
        public ModelAndView searchCow(){ //...
            return new ModelAndView("search.cow", modelMap);
        }
    }

then change the tiles.xml to introduce the definition includes

//-- tiles.xml
<definition name="REGEXP:([^/.][^.]*)\.([^.]+)" template="/WEB-INF/tiles/template.jsp">
    <put-attribute name="meta" value="/WEB-INF/tiles/${options[myopts]}/meta.jsp"/>
    <put-attribute name="header" value="/WEB-INF/tiles/${options[myopts]}/header.jsp"/>
    <put-attribute name="body" value="/WEB-INF/tiles/${options[myopts]}/body.jsp"/>
    <put-attribute name="footer" value="/WEB-INF/tiles/${options[myopts]}/footer.jsp"/>

    <!-- definition injection performed by DefinitionInjectingContainerFactory.instantiateDefinitionFactory(..) -->
    <put-list-attribute name="definition-injection">
        <add-attribute value=".category.{2}" type="string"/>
        <add-attribute value=".action.{1}.{2}" type="string"/>
    </put-list-attribute>

    <put-list-attribute name="myopts" cascade="true">
        <add-attribute value="{2}"/>
        <add-attribute value="{1}"/>
        <add-attribute value="common"/>
    </put-list-attribute>
    <put-list-attribute name="myopts-view" cascade="true">
        <add-attribute value="{2}"/>
        <add-attribute value="{1}"/>
        <add-attribute value="view"/>
    </put-list-attribute>
    <put-list-attribute name="myopts-edit" cascade="true">
        <add-attribute value="{2}"/>
        <add-attribute value="{1}"/>
        <add-attribute value="edit"/>
    </put-list-attribute>
    <put-list-attribute name="myopts-dog" cascade="true">
        <add-attribute value="{2}"/>
        <add-attribute value="{1}"/>
        <add-attribute value="dog"/>
    </put-list-attribute>
    <put-list-attribute name="myopts-cat" cascade="true">
        <add-attribute value="{2}"/>
        <add-attribute value="{1}"/>
        <add-attribute value="cat"/>
    </put-list-attribute>
    <put-list-attribute name="myopts-cow" cascade="true">
        <add-attribute value="{2}"/>
        <add-attribute value="{1}"/>
        <add-attribute value="cow"/>
    </put-list-attribute>
</definition>

//-- tiles-core.xml
<definition name="REGEXP:\.action\.view\.([^.]+)">
    <!-- override attributes -->
    <put-attribute name="body" value="/WEB-INF/tiles/${options[myopts-view]}/view_body.jsp"/>
    <!-- search attributes -->
    <put-attribute name="view.content" value="/WEB-INF/tiles/${options[myopts-view]}/view_content.jsp"/>
    <put-attribute name="view.statistics" value="/WEB-INF/tiles/${options[myopts-view]}/view_statistics.jsp"/>
</definition>
<definition name="REGEXP:\.action\.edit\.([^.]+)">
    <!-- override attributes -->
    <put-attribute name="body" value="/WEB-INF/tiles/${options[myopts-edit]}/edit_body.jsp"/>
    <!-- edit attributes -->
    <put-attribute name="edit.form" value="/WEB-INF/tiles/${options[myopts-edit]}/edit_form.jsp"/>
    <put-attribute name="edit.status" value="/WEB-INF/tiles/${options[myopts-edit]}/edit_status.jsp"/>
</definition>
<definition name="REGEXP:\.action\.search\.([^.]+)">
    <!-- override attributes -->
    <put-attribute name="body" value="/WEB-INF/tiles/${options[myopts-search]}/search_body.jsp"/>
    <!-- search attributes -->
    <put-attribute name="search.form" value="/WEB-INF/tiles/${options[myopts-search]}/search_form.jsp"/>
    <put-attribute name="search.results" value="/WEB-INF/tiles/${options[myopts-search]}/search_results.jsp"/>
</definition>
//-- tiles-cat.xml
<definition name="REGEXP:\.category\.cat">
    <!-- cat attributes -->
    <put-attribute name="cat.extra_information" value="/WEB-INF/tiles/${options[myopts-cat]}/cat_extra_information.jsp"/>
    <put-attribute name="cat.feline_attributes" value="/WEB-INF/tiles/${options[myopts-cat]}/cat_feline_attributes.jsp"/>
</definition>
//-- tiles-dog.xml
<definition name="REGEXP:\.category\.dog">
    <!-- cat attributes -->
    <put-attribute name="dog.extra_information" value="/WEB-INF/tiles/${options[myopts-dog]}/dog_extra_information.jsp"/>
    <put-attribute name="dog.k9_attributes" value="/WEB-INF/tiles/${options[myopts-dog]}/dog_k9_attributes.jsp"/>
</definition>
//-- tiles-cow.xml
<definition name="REGEXP:\.category\.cow">
    <!-- cat attributes -->
    <put-attribute name="cow.extra_information" value="/WEB-INF/tiles/${options[myopts-cow]}/cow_extra_information.jsp"/>
    <put-attribute name="cow.farm" value="/WEB-INF/tiles/${options[myopts-cow]}/cow_farm.jsp"/>
</definition>

There's a lot of configuration here now, but the idea isn't to avoid configuration altogether but to avoid having to edit the configuration everytime a new jsp or page is created. That is we are putting effort here into creating our convention so that further configuration isn't required. Now there's a clear separation between each category definition and each action definition. For example such separation helps various development roles work in parallel: front end developers can concentrate on categorical designs while system developers often work initially with actions jsps to get them functionally working before passing them off to the front end developers. Such separation of concerns will show benefits in more ways that just this example. The following DefinitionsFactory makes this all come together

public class FinnUnresolvingLocaleDefinitionsFactoryImpl extends UnresolvingLocaleDefinitionsFactory {

    private static final String DEF_INJECTION = "definition-injection";

    public FinnUnresolvingLocaleDefinitionsFactoryImpl() {}

    // this method may return null
    @Override
    public Definition getDefinition(String name, TilesRequestContext tilesContext) {
        Definition def = null;
        // WEB-INF is a pretty clear indicator it is not a definition
        if(!name.startsWith("/WEB-INF/")){
            def = super.getDefinition(name, tilesContext);
            if(null != def){
                def = new Definition(def); // use a safe copy
                Attribute defList = def.getLocalAttribute(DEF_INJECTION); // explicit injected definitions
                if(null != defList && defList instanceof ListAttribute){
                    for(Attribute inject : (List) defList.getValue()){
                        injectDefinition((String) inject.getValue(), def, tilesContext);
                    }
                }
            }
        }
        return def;
    }
    private void injectDefinition(String fromName, Definition to, TilesRequestContext cxt) {
        assert null != fromName : "Definition not found " + fromName;
        Definition from = getDefinition(fromName, cxt);
        if (null != from) {
            if (null != from.getLocalAttributeNames()) {
                for (String attrName : from.getLocalAttributeNames()) {
                    to.putAttribute(attrName, from.getLocalAttribute(attrName), true);
                }
            }
        }
    }
}

To plug this DefinitionsFactory in make sure to return it from your TilesContainerFactory

public class FinnTilesContainerFactory extends BasicTilesContainerFactory {
    ...
    @Override
    protected UnresolvingLocaleDefinitionsFactory instantiateDefinitionsFactory(TilesApplicationContext applicationContext, TilesRequestContextFactory contextFactory, LocaleResolver resolver) {
        return new FinnUnresolvingLocaleDefinitionsFactoryImpl();
    }
    ...
}

When the Composite pattern is superior

It's nonsense to think that any one pattern is better. Both the Composite pattern and the Decorator pattern have their strengths and weaknesses, even after we have progressed the Composite on to being highly dynamic and automated. They do, and achieve, quite different things and hence each should be used where applicable...

The Decorator pattern allows front end code and design to be injected as we process. When a decision is known during the request processing the code can immediately build a decorator. The context here only holds all the built decorators. And at the end of the request lifecycle the page is assembled by putting together all these decorators. Decorators can also be built upon each other, or stacked, and this can be useful when the composition of the page is completely loose.

The Composite pattern presumes the page to be a "composite" made up from components, where each component is free to be itself a "composite". The pattern allows more control of the composite's heirarchy: as the delegation is top-down as opposed to the Decorator pattern whom's is bottom-up. The composite pattern works well when the operations you need to perform on each object is limited, that is it isn't a problem that the operations you may perform on any one object is the lowest common demominator of operations that you can perform on every object.

When you use jsp includes whether you like it or not you have a page heirarchy. If your jsps typically have a template with fragments you are already working within the composite pattern's paradigm: the template being the "composite" and the fragments being the components. Each fragment, each JSP, has a context that holds the model map, or variables, in various scopes, and within this context they are self-sufficient. The only operation required upon them by others is inclusion. Applying the composite pattern here means treating each fragment as a self-sufficient object, the only thing that can happen to it is it will be included. This shows how we can design to reduce complexity and encourage there to be only one unified context. It also shows how we can maintain a top-down control of the page's heirarchy, something necessary in a MVC design where the control layer wants to hand off a finished model map, ie one fixed and unified context, that the view layer is free to build itself off. In contrast when we choose the Decorator pattern we can run foul of letting code run in parallel to the MVC pattern.

Conclusion

Front end developers, the system developers, and the architects need to work together. With the front end today often centered around loose coupling design: javascript's lack of type safety, ajax's separation from the server, and jQuery's liberation of the dom; it is all too easy to rebel against any form of structure or organisation for fear of being tied into the formalities of the strict-typed java world. Neither should the fear of giving up control of one's craftsmanship mean keeping logic and control in the View layer, this stuff belongs in the Control layer: and this means the system developers need to work harder to get the front end developers collaborating in the control layer, or the roles of system and front end developers need more overlap within teams. Everything has entropy, the larger the organisation, the larger the codebase, the less you fight the entropy, the quicker you end up with a pile of spaghetti in your lap. These four steps show you how Tiles-3 and the Composite pattern can be ramped up on steroids to give front end and systems developers what they want in a way that encourages them to work together.

References
  1. Tiles-3
  2. Composite pattern
  3. Decorator pattern
  4. Spring Web
  5. Integrating Tiles and Spring
  6. Initial discussion regarding Tiles definition delegation[Gone]
  7. Spring 3.2 with Tiles-3 :: Tiles-3 support

Going all out with the Strap-on Project

2012-06-23T18:32:28+00:00

In January we started on an ambitious project called “the Strap-on project”. The infantile name aside this project is about rewriting our entire front-end tier using Object Oriented CSS (OOCSS), written by Nicole Sulivan, as the secret sauce to achieving the desired results.

Why OOCSS?

At FINN we have teams grouped by business units and all of them have developers who contribute with code on our site. This is provided us with a head ache with CSS, because there are no clear idioms for how to best write CSS. In our case this had cause teams to create their own “islands” of CSS by using # trails, train wreck selector classes and !important-wars. Teams had duplicate CSS for the same layout and we had a hard time keeping the user experience consistent across different parts of our service.

We had way too much CSS code in total and we sent loads of CSS on each page request with just a small percentage actually being used. This made page load slow and rendering slow. Page rendering speed is equal to money, so we had to change something.

OOCSS - making CSS coding a thing of the past

OOCSS is a framework on which to build your own. It provides with a base set of modules and concepts which is essentials to building stuff in HTML with CSS. Grids, modules, lines, etc. These base modules provides an abstraction on top of CSS which makes authoring of CSS a thing of the past. In order to layout basic pages all we do is to use the building blocks and put together what ever you want. Basic boring stuff such as clearing and the box-model is no longer anything you need to worry about, it’s taken care of. This enables developers to focus upon fulfilling business requirements instead of battling CSS differences in browser time and time again.

Why not LESS, Compass or stuff like that?

The CSS language abstractions that exist out there are all very cool projects and make a whole lot of sense to use for a lot of projects. However, in our case choosing one of these tools right now would not provide us with some of the benefits we are looking for.

At FINN we have a problem that we are duplicating CSS across our development teams. This makes creating a consistent user experience and making global changes harder than it should be. Using an abstraction would help us hide some these problems, but would not solve the underlying issue which is that we create too much code. OOCSS and its rigorous set of rules helps reduce the amount of code drastically and provides rigid rules which prevents duplication across teams. What we are looking for are:

Improve rendering speed
Improve development speed
Easier to provide a consistent user experience

A touch of Bootstrap too

The engineers over at Twitter has created an amazing framework, Bootstrap, which is hugely popular all over the world. We have indeed paid close attention to how they have done and we are very much inspired by their work. However, going all out and just adopting Bootstrap was not an option. We feel it is too bloated and it does not provide the speed and performance benefits OOCSS gives. Having said that, a lot of our setup with forms and form elements is influenced by how Bootstrap does things.

Half way, how does it look?

In short,iIt looks pretty darn impressive! We have removed more than twenty CSS files and reduced the amount of CSS code lines with more than thirty thousand (this does say quite a bit of the mess we where in, I know). We have migrated the motorized vehicles, jobs, real estate sections and the front page.

Before After

Number of CSS files

130

Lines of CSS code

32 789

2 927

Lines of section specific CSS

727

These are pretty impressive results and very much in line with what Nicole is talking about in her presentations about OOCSS and performance.

is it all singing, all dancing?

Of course not! The responsive bit is something we are looking at reworking. Team Oppdrag has created one way of solving this and Team Reise another. For the Strap-on project we will probably end up with something in between. There are some things you need to take into consideration when choosing an approach: How much control do you have over the markup being written? Or to put it in another way, how many people are working with the same code? The more people, the harder it is to apply very strict rules as people will be “tourists” in the code base and might have a hard time getting it.

Another big challenge is to provide enough support and training to help everyone utilize the OOCSS framework so we can continue to reap benefits from what we have done, but that is probably some other blog post.

Profiling and debugging view templates

2012-06-23T10:31:21+00:00

Ever needed to profile the tree of JSPs rendered server-side? Most companies do and I’ve seen elaborate and rather convoluted ways to do so.

With Tiles-3 you can use the PublisherRenderer to profile and debug not just the tree of JSPs but the full tree of all and any view templates rendered whether they be JSP, velocity, freemarker, or mustache.

At FINN all web pages print such a tree at the bottom of the page. This helps us see what templates were involved in the rendering of that page, and which templates are slow to render.

]

We also embed into the html source wrapping comments like

...template output...

The code please

To do this register and then attach your own listener to the PublisherRenderer. For example in your TilesContainerFactory (the class you extend to setup and configure Tiles) add to the methd createTemplateAttributeRenderer something like:

    @Override
    protected Renderer createTemplateAttributeRenderer(BasicRendererFactory rendererFactory, ApplicationContext applicationContext, TilesContainer container, AttributeEvaluatorFactory attributeEvaluatorFactory) {

        Renderer renderer = super.createTemplateAttributeRenderer(rendererFactory, applicationContext, container, attributeEvaluatorFactory);
        PublisherRenderer publisherRenderer = new PublisherRenderer(renderer);
        publisherRenderer.addListener(new MyListener());
        return publisherRenderer;
    }

Then implement your own listener, this implementation does just the wrapping comments with profiling information…

class MyListener implements PublisherRenderer.RendererListener {

    @Override
    public void start(String template, Request request) throws IOException {
        boolean first = null == request.getContext("request").get("started");
        if (!first) {
            // first check avoids writing before a template's doctype tag
            request.getPrintWriter().println("\n");
            startStopWatch(request);
        } else {
            request.getContext("request").put("started", Boolean.TRUE);
        }
    }

    @Override
    public void end(String template, Request request) throws IOException {
        Long time = stopStopWatch(request);
        if(null != time){
            request.getPrintWriter().println("\n");
        }
    }

    private void startStopWatch(Request request){
        Deque<stopwatch> stack = request.getContext("request").get("stack");
        if (null == stack) {
            stack = new ArrayDeque<stopwatch>();
            request.getContext("request").put("stack", stack);
        }
        StopWatch watch = new StopWatch();
        stack.push(watch);
        watch.start();
    }

    private Long stopStopWatch(Request request){
        Deque<stopwatch> stack = request.getContext("request").get("stack");
        return 0 < stack.size() ? stack.pop().getTime() : null;
    }

It’s quick to see the possibilities for simple and complex profiling open up here as well as being agnostic to the language of each particular template used. Learn more about Tiles-3 here.

Technology Day 2012

2012-06-12T08:43:19+00:00

One of the ways of getting inspired and educated about technology here at FINN is through our “Tech-Dag” (which means something along the lines of Day of Technology). This year we have a packed program which consists of workshops and a bunch of lightning talks.

Workshops

Clojure workshop

Alternative languages on the Java Virtual Machine is a hot topic all over and at FINN we are exploring our options as well. Therefor we have invited Johannes Brodwall, Ivar Nilsen and Anders Karlsen from Steria to do a introduction to Clojure workshop.

FINN Way of Innovation

Our Lean-navigators have been working hard to help our development teams make conscious decisions and to make sure they make the right priorities to maximize customer value. This workshop will touch upon some of the basic aspects and give the participants a way of giving input to the process.

Lightning talks

Lightning talks are 10 minutes long and tightly timed. We hold them after lunch, starting at 1330 – 1500 pm. The topics are broad ranging, and not limited to programming at all.

About love (… and ask yourself “why do we work 8 hours a day”)
Fragmentation of the Android and why it is not a problem for us
Testcomplete database code before check - is it possible?
Elastic Search
Geolocation with MongoDB
Traditional version control systems vs distributed version control systems (VCS -> DVCS).
Eat your own dogfood
Semantic web
Large IT projects should fail!

Følg oss på Hash tag #techdagen

When do you know, how much testing is enough?

2012-05-21T11:56:43+00:00

Let me begin with an example, from every days scenario at work. You are at the end of the sprint and have some errors reported. You think that the quality of code and tests are good enough for release. Based on this the reported bugs the failures are minor and won’t have big impact after fixing them. But the reality may be different, that you really are on opposite ends of the quality. The code and test quality is low, and the reported tests are few or no bugs. Your worst case scenario could be that deployment/release will go very bad and can be very expensive in a technical or business nature.

It’s important to evaluate the possibility of finding more defects based on results you have gathered so far. The key is to analyze your rapport thoroughly because:

If you have just found a defect, this is a signal to keep testing. It may seem counter-intuitive but in general the more defects you find the more likely it is that there are additional defects.
If you have only exercised a small portion of the overall functionality and have found defects, then this is a signal to continue testing.
If you have been testing a particular piece of functionality for a while and are not finding any new defects, then this is a signal for you to stop testing. How much of the system’s functionality have you tested?
If there are significant features that are mostly or entirely not tested, then you will likely not be prepared to recommend it is good to go.
False confidence is a very real danger. This is especially true for developers. I have seen or heard of many who have an unrealistically high level of confidence that their code works, in some cases without any testing whatsoever. As a specific example, I have heard developers state that if their code compiles, it is good enough to be promoted to acceptance testing.

Testing by definition is comparing an expected result to an observed result. So it should be easy to answer the question how much testing is enough? But the answer could be based on number of factors such as:

Identifying risks early in your project on technical and business aspect.
The project constraints such as time or budget. If a team decides to enlarge the scope of a project the time will become larger as well along with the cost. If the time constraint is tighter the scope may be reduced but the cost will remain high. If the team decides to tighten the budget, the scope will become smaller but the time will increase. The time constraint deals with the time necessary to finish a project. To successfully complete a project, the time constraint should be comprised of a schedule. You should have a specific schedule related to the time that it will take you to finish the project.
Measure the complexity and risk of new features and how much of the product they impact.
Measure bugs found during the release and bugs found after the release and in what areas of the product.

By prioritizing risks early it will help to determine how much testing is considered to be enough. Once you have considered the risk, you should find the goal of testing. Once you have obtained a sufficiently high level of confidence in different levels of test such as Unit testing, Integration testing, System testing, Acceptance testing and Regression testing you are done testing.

It is possible to do enough testing but determining the how much is enough is difficult. Simply doing what is planned is not sufficient since it leaves the question as to how much should be planned? What is enough testing can only be confirmed by evaluating the results of testing? If lots of defects are found with a set of planned tests it is likely that more tests will be required to assure that the required level software quality is achieved. On the other hand if very few defects are found with the planned set of tests, then no more tests will be required.

Testing should provide information to the stakeholders of the system, so that they can make an informed decision about whether to release a system into production or to customers. Testers are not responsible for making that decision, they are responsible for providing the information so that the decision can be made in the light of good information.

So going back to my question, testing is done when its objectives have been achieved and more specifically, you are done testing when:

You are unlikely to find additional defects.
You have a sufficiently high level of confidence that the software is ready to be released.

Vivek Sharma Test Manager at FINN.no

Putting a mustache on Tiles-3

2012-05-15T18:58:12+00:00

We’re proud to see a contribution from one of our developers end up in the Tiles-3 release!

The front-end architecture of FINN.no is evolving to be a lot more advanced and a lot more work is being done by client-side scripts. In order to maintain first time rendering speeds and to prevent duplicating template-code we needed something which allowed us to reuse templates both client- and server-side. This is where mustache templates have come into play. We could’ve gone ahead and done a large template framework review, like others have done, but we instead opted to just solve the problem with the technology we already had.

Morten Lied Johansen’s contribution allows Tiles-3 to render mustache templates. Existing jsp templates can be rewritten into mustache without having to touch surrounding templates or code!

The code please

To get Tiles-3 to do this include the tiles-request-mustache library and configure your TilesContainerFactory like

    protected void registerAttributeRenderers(...) {
        MustacheRenderer mustacheRenderer = new MustacheRenderer();
        mustacheRenderer.setAcceptPattern(Pattern.compile(".*.mustache"));
        rendererFactory.registerRenderer("mustache", mustacheRenderer);
        ...
    }
    protected Renderer createTemplateAttributeRenderer(...) {
        final ChainedDelegateRenderer chainedRenderer = new ChainedDelegateRenderer();
        chainedRenderer.addAttributeRenderer(rendererFactory.getRenderer("mustache"));
        ...
    }

then you’re free to replace existing tiles attributes like

<put-attribute name="my_template" value="/WEB-INF/my_template.jsp"></put-attribute>

with stuff like

<put-attribute name="my_template" value="/my_template.mustache"></put-attribute>

Good stuff FINN!

QA role is dead but development and testing together provides quality for FINN.no

2012-05-07T07:29:20+00:00

When a developer is finished with a development issue, it must be tested. People often think the goal of testing should be to find all the errors. Well, it’s not entirely realistic. Actually, it’s impossible! Despite the title of “quality insurance” testers can never really ensure level of quality, nor can they be expected to find all the errors. The real goal of testing should be to improve the software. As at the end of the day we want to minimize the cost of fixing critical bugs after a release, minimize cost of handling customer complaints and pushing a patch in the production. So it is import to understand that Quality Assurance is a process, not a department.

Today there are basically two different components of QA role in FINN. One of which is “QA in a team” and the other one is of a “team without the QA”. The QA role today is often a safety net for many developers and teams. The role involves manual testing in the sprint, regression testing and writing Cucumber tests. In addition this role has some administrative tasks such as being contact person for the Release Manager, participating in the release status meeting, filling out the checklist of early / late shift, contact for Tech Support (E-journal issues), participating in status meetings at Call Center after release, and verifying the patch to production. Everything that’s listed so far is managed also by teams without QA. What we have seen so far, is that a team without QA delivers the same quality as the team with QA in developing an FINN application.

Then comes the question, how do they manage to do it?

They have the same responsibilities as the “QA in a team”, but they have different practices on how they manage the tasks (some have rolling QA role, others have a fixed arrangement but they are NOT dependent on one individual person).

If we want to deliver a good quality code, testing must become a central pillar of the development process. When I say the role of QA is dead, but development and testing together provides quality for FINN, by that I mean that the whole team should focus on unit testing, integration testing, system testing and functional testing (as part of the pipeline). This will build and provide quality from top to bottom in the product stack.

Despite back-end test automation, front end problems with web applications are always discovered by human testers, including issues related to browser compatibility or CSS flaws. This can be covered by focusing on exploratory testing, which is also done in a team with no QA.

The QA testers should rather concentrate on the planning of tests within their teams. They should focus on three things.

Attributes: Describes the high level concept testing are meant to ensure performance testing, usable testing, secure testing, accessible testing and so forth.
Components: Define the major code chunks that comprise the product. These are classes, module names and features of the application.
Capabilities: Describes user actions and activities.

We know it very well that we cannot test everything so there is no pointing in investing too much time in documenting everything either? And we know as we start testing things the schedule, the requirements, the architecture etc are going to change. But the greater benefit is that whole team takes the responsibility for the quality, quicker feedback, tests documenting the code and not the least satisfied customers.

Written by Vivek Sharma Test Manager at FINN.no

Spreading love and good vibrations on GitHub

2011-11-21T19:02:40+00:00

We have finally made an effort to put some of our repos out on GitHub under the finn-no organization. It is not exactly filled with stuff right now, but we aim to put a lot more of our libraries and tools out here.

Currently there are a few repositories out there:

eventHub
Awesome Board[Gone]

eventHub

This is just a tiny piece of code which enables you to build loosely coupled components or applications by utilizing the publish-subscribe pattern. The eventHub is just an object which publishes event to those to registered event listeners functions.

We have written one-page JS applications using only the hub besides jQuery for DOM manipulation. It scales pretty well and it is pretty much what you’d need to create an application. Once your application grows to be quite large you might want to add some other architectural components, but for small to medium sized projects it is brilliant. It is used for the dashboard part for the Awesome Board[Gone].

Awesome Board

The Awesome Board[Gone] is the application which has featured in two articles about visualizing quality. We have shown it a few times to different audiences and due to request for making it available we decided it was a good thing to share it on GitHub. There are some pieces missing from the version we run internally, but have patience the rest might still make it to a repo near you.

Enjoy! Feel free to drop us a line in the comments section or follow us on Twitter.

We rocked out at the Smidig conference!

2011-11-19T12:31:11+00:00

The annual Smidig conference in Oslo, Norway was held last week and as usual people from FINN.no made a big contribution to the conference. We hade three lightening talks during the two day conference and there were about ten attendants in total. This community event brings together newbies and veterans of agile software development for two days of lightning talks and open space sessions. We all had a great time at the conference and are very much looking forward to next years conference.

Our three lightning talks spanned from a Norwegian version of over to a talk showing how we transformed our customer service team and the last one about how you need to measure in order to achieve the full effect of agile processes. You can check out the videos below, but be warned they are all in Norwegian.

Transforming our customer services team

Jo Anders Heir held a talk about how we have worked to transform our customer service team from being a shield preventing customer feedback from reaching product organization into a megafon which blasts information throughout the organization. It shows how simple software and processes can help you make sure you work on what really matters for you users and customers.

Measure KPI’s, iterate and release. The three things you need to do to succeseed with agile

Arve Søreide talked about three things you need to do in order to achieve the desired effects from using agile methods. You need to mease Key Performance Indicators while you iterate and release frequently. Only then can you harvest the big benefits from using agile methods.

An end to negativity - Norwegian style

This talk was originally held by Chris William at JSConf EU 2011 and you should check out the original version of An End To Negativity.

A big thanks from all of us at FINN.no to the crew who’s hard work made this years conference a huge success. See you all again next year!

Make real estate brokers happy with hard boiled business cards

2011-10-14T08:21:02+00:00

FINN.no has recently released new pages for real estate ads. Larger images, broker business cards and tabs for additional information are some of the new features on this page. A long process of in-depth users interviews, workshops with real estate brokers, user testing of design sketches and several sprints of development, we are proud to finally see the new pages on www.finn.no.

Pick a nice home from this list to view the ads Cut the general crap - show me the business cards!

What’s in it for the user?

Larger images and the possibility to navigate through images without leaving the page were the first thing that was noticed, and liked, during our user testing sessions. For a full size view of 900 pixel images we have still kept a link to our old image viewing page. However, after release we have got feedback from many users who failed to find this link and complain that images are smaller. There is still work to be done to improve this.

The information about the home is divided into primary and secondary information. The prime information, like price, area, address and viewings, is put at the top of the page, so the user won’t need to scroll to find it. More detailed information, like facilities, plot size and descriptions is located below the prime information and the image viewing module.

Secondary information is integrated by tabs. Fakta (facts) is the default tab, containing information about the estate from the broker. Prisstatistikk (price statistics) is highlights from Eiendomspulsen, which displays information like average price per square meter in the area and prices for previous sales of the actual estate. Neighborhood profile is an additional service brokers may add to their prospects with information about nearby services, schools and demographics.

What’s in it for the broker?

Realestate brokers have requested more options for building brand awareness in the prospect. To meet this request our designers suggested the brokers may design their own business cards for their contact information and to use their profile colors at some other elements on the page. They may also offer their services after the last image in the image gallery and in a tab next to Price statistics and Neighborhood profile. Interviews with users indicate that information from the broker won’t be regarded as ordinary banner ads as long as they provide information that is relevant for users who are in the process of changing their home. Krogsveen (a major realestate agent in Norway) has taken good advantage of this banner position by listing properties they have recently sold in the same area.

How to make the business cards

There are three ways to display the broker’s contact information. Basic prospects have a simple listing of name, position, phone and fax numbers along with a company logo. Brokers who are using expanded prospects, which are prospects with the broker’s own set of banners, have their contact information styled as a business card. Brokers may customize these cards with their own css, background image and a picture of the broker.

Some prospects have two brokers listed, but to save space only one broker is visible at the time. In expanded prospects business cards will swap when the user click a looping arrow or the card in the back. In basic prospects the name of the other broker slide in when clicking the “more contact persons” link. Jquery animation is used to swap the cards and CSS transitions is used for the sliding of contacts in basic prospects.

The markup is the same for all levels of contact information, supporting the hcard microformat, while there are three layers of css. Styling of the contact information on basic prospects is in the main css file. Default business cards are styled by using a seperate css file which adds the business card look and overrides the default css where necessary. Customized business cards are styled by a third layer of css, which overrides the classes from the default card to add a background image and their company’s colors and fonts.

By the time of release we had 13 different customized business cards on our prospects.

Contact information for a basic prospect

HTML code for a business card looks like this:

    <address id="flip_0" class="vcard flip  backgroundimage">
      <a href="(url)" class="orglogo" target="_blank"><img src="(url)" alt="Utleiemegleren Sinsen" border="0" class="logo"></a>
      <span class="fn n brokerName">Karianne Johannessen</span>
      <span class="ul">
        <span class="li title brokerTitle">Supermegler</span>
        <span class="li brokerPhoneHolder">
          <dl class="tel brokerPhone">
            <dt class="type hideText">Work</dt>
            <dt class="displayText">Telefon</dt>
            <dd class="value">22 79 66 26</dd>
          </dl>
        </span>
        <span class="li brokerMobilHolder">
          <dl class="tel brokerMobil">
            <dt class="type hideText">Cell</dt>
            <dt class="displayText">Mobil</dt>
            <dd class="value">48133956</dd>
          </dl>
        </span>
        <span class="li brokerFaxHolder">
          <dl class="tel brokerFax">
            <dt class="type hideText">Fax</dt>
            <dt class="type">Faks</dt>
            <dd class="value">22796601</dd>
          </dl>
        </span>
        <span class="li brokerEmail">
          <a data-click-mt="mailto" href="(url)" class="click-track">Send epost</a>
        </span>
      </span>
      <div class="clearall"></div>
    </address>

This is the basic css for contact information:

    .contact {overflow: hidden; width: 330px;}
    .contact .cardWrapper {position: relative; width: 700px;}
    .contact address {-moz-transition: left 0.7s ease-in-out;-webkit-transition: left 0.7s ease-in-out; -o-transition: left 0.7s ease-in-out; -ms-transition: left 0.7s ease-in-out; transition: left 0.7s ease-in-out;
        background-color:#FFFFFF; display:block; float:left; font-style:normal; position:relative; width:330px;}
    .contact .hideText {display: none;}
    .contact .orglogo {float: right;}
    .contact .brokerName {display:block; font-size:12px; font-weight:normal; margin:0;}
    .contact .li {clear: left; display: block; margin: 0; padding: 0;}
    .contact dt {float: left; padding-right: 0.5em;}
    .contact dd {float: left;}
    .contact .moreContacts {float:left; width:330px; text-align:right; margin-bottom:20px; }
    .contact address {left:0;}
    .contact address.backCard {position:relative; left:360px;}
    .contact address +  address {left:-330px;}
    .contact address + address.backCard {left:30px}

Default styling of a business card

This is the second layer of css which forms a basic business card:

    #flip_0{z-index:1; top:0; left:0;}
    #flip_1{z-index:0; top:20px; left:15px;}
    .contact .flip.backCard {cursor:pointer; background:#f3f3f3; display:block; transform: rotate(3deg);-o-transform: rotate(3deg);
        -ms-transform: rotate(3deg); -moz-transform: rotate(3deg); -webkit-transform: rotate(3deg);}
    .contact .flip.backCard:hover {background:#ffffff;}
    .contact .cardWrapper {position:relative; min-height:195px; width:auto;}
    .contact .spacer {height:25px; display:block; clear:both;}
    .contact .flip{position:absolute; background-color:white;}
    .contact address {box-shadow: 2px 2px 8px 0 #cccccc; padding:10px; width:300px; height:150px; border: #eeeeee solid 1px; float:none; overflow:hidden;
        -moz-transition: none;-webkit-transition: none;-o-transition: none;transition: none;-ms-transition: none;}
    .contact address .brokerPhoto {float:left; max-height:100px; max-width:75px; margin: 15px 10px 0 0; display:block;}
    .contact .brokerName {font-size:16px; font-weight:normal; margin-top:15px}
    .contact img.brokerPhoto~ul {max-width:190px}
    .contact .orglogo img {position:absolute; bottom:10px; right:10px;max-height:100px; max-width:100px; }
    .contact .moreContacts {display:none;}
    .contact img.moreContacts  {float:right; position:absolute; top:0; right:-12px; display:block; cursor:pointer; width:14px; height:22px}
    .contact .ul {float: left; display:block;}

Customized business card

This third layer of css forms the broker’s customized business card:

    .contact * {color:#46464a; font-size:11px;}
    .contact .orglogo {display:none;}
    .contact .brokerPhoto {margin:11px 20px 0 11px !important;}
    .contact .brokerName {font-weight:bold; font-family:"Lucida Sans","Lucida Sans Unicode","Lucida Grande",sans-serif; margin:8px 10px 0 0; font-size:12px;}
    .contact .brokerTitle {margin:0 0 5px 0;}
    .contact .brokerPhone dt {font-weight:bold;}
    .contact .brokerMobil dt {font-weight:bold;}
    .contact .brokerFax dt {font-weight:bold;}
    .contact .brokerEmail a {padding-top:5px; display:inline-block;}
    .contact .brokerEmail a, .contact .homePage a {color:#46464a;}
    .contact .homePage {margin-top:0;}
    .contact .backgroundLogoLink {position:absolute; left:0; bottom:0; width:320px; height:25px; display:block;}

The feedback

Three weeks after the release we have recieved 1060 feedbacks, 630 positive, 374 negative and 56 indifferent. This is actually better feedback than we are used to after releasing such a radical redesign. Our biggest challenge is to make it easier to view images in full size. For other layout issues it seems like people simply need some time to get used to it. I have seen no complaints from users who dislikes the business cards, which was one of our biggest concerns before the release.

Our analyzing tools tell us that the average viewing time on this page before the release was 33 seconds. After the release of the new design the viewing time has increased to 1 minute and 15 seconds, at the expense of the image viewing page, which page views has dropped rapidly. The number of page views of the ad page has also dropped, probably because of less navigation to and from the image viewing page. However, the number of visits on the ad page seems to be stable.

How to get a job at FINN.no

2011-09-02T08:08:46+00:00

If you’re a student or just someone who is looking for a job you might be wondering: “How could I get to work for FINN.no?”. One guy found an answer and is now one of the key developers on our iOS applications. We did an advert for summer interns last year and he was one of the people who came in to do some proof of concept work on iOS. The end result was an iPad application that never hit the AppStore, but for one of the participant this internship was the first step towards landing a job as a developer at FINN.no.

What made you apply for the summer internship at FINN.no?

It all started with my university teacher pointed out that FINN.no had an open internship during the summer. At first I did not want to apply, because I wanted to have vacation all summer (Like all other informatics students). However, I ended up with applying and just a few hours later I got a call; the internship was mine! The main reason to apply for the internship was that I was finishing my master’s degree and was lacking work experience in the field of informatics. Secondly, I did not know that much about FINN.no and I could always need the extra money :)

What did you learn during that summer?

I learned that FINN.no is not just a big website, it is also a big company. And there are not just developers at FINN.no, but also a range of different jobs (Like finance, marketing, economics and so on). When taking about the technical aspects I learned how to develop for the iPhone/iPad. Furthermore, I learned that to get things working you are depended on a range of different “teams” to develop what you need. And this is not always trivial, because all “teams” have their own focus areas and problems. So you never work independent!

What was it like to be a summer intern at FINN.no?

It was both interesting and rich on experiences. The one thing I want to point out is how the atmosphere and the peoples at FINN.no are. The atmosphere is comfortable and “chill” and the people are welcoming and easy to talk with. They want to teach you what they know and want to learn about what you know.

How did you do about landing the position at FINN.no?

Firstly, I work hard during the summer and tried my best to present what I could do. Before I left FINN.no that summer I was told that if I needed a part time work, I should just ask and they would see what they could give me. After the summer internship was completed I went back to school for a few months. Some time before Christmas I sent my supervisor at FINN.no a e-mail and ask if it was possible to get a 50% position at the company the last 6 months of my masters degree. 1. January I started with working part time at FINN.no and before I completed my degree I was presented with a full time position at FINN.no.

What have you been doing since you came to work for FINN.no?

I have been continuing learning objective-c and iOS. Furthermore have I learned more about designing projects, completing them and how to work with other development teams (To get our hands on the right APIs we needed in the iOS app). One example is the new iOS app, FINN Torget annonse innleggning”, that FINN.no released a week ago.

What is the difference between working with school assignments and working at a company?

The biggest difference must be that a school assignment is something that will never be used by any persons. So you just complete it because you must, and not because you want to. In a company as big as FINN.no your applications will be used by a lot of peoples. You must always perform your best to please all of the users. Because of this you want to stay on the edge of new technology and learn more.

What’s your advice for other young developers out there wanting to work for the coolest company in Norway?

You must work hard and make them see you in the crowd. If a new internship opens up do not think twice just apply. Before you know it, you are placed in my position to answer a few questions in a interview like this one.

Anything you want to add?

I love how they work at FINN.no, a lot of very smart people and it feels like peoples at FINN.no don’t stress that much! And peoples at FINN.no do know how to create a party ;)

It has never been about the standers, while it has always been about the standards

2011-08-26T06:50:51+00:00

I was introduced to the World Wide Web back in 1995 while being a student at Molde College, Norway. What fascinated me the most was that I could learn how to create stuff for the web by just viewing the source code of other web sites. This blew my mind and since then I have built a career working with web development.

Creating web sites and web-based applications for me has always been about trying to adopt the latest technological advances in browsers, as soon as they were available. Naturally, this has lead to me reaching a few dead ends (IE’s Data Binding and Netscape JavaScript style sheets), but mostly it has enabled me to create some pretty cool things.

From DHTML to an atomic winter

In the days of the .com era, the community for creating web based applications was buzzing. We were cranking out all kinds of DHTML magic. Every site out there was jumping on the band wagon without any hesitation. One of the classics is online retailer Boo.com which went all out with new technology without really succeeding. Naturally we had to deal with older browsers then too, but we did not let that get in the way of using the latest standards being adopted in browsers.

The burst of the .com bubble created something of an atomic winter as far as web based user interface development goes. A lot of people lost their jobs and the rest of us took refuge in server side development.

A clean slate and a new dawn

Thankfully all the bad stuff was swiftly cleaned up with the introduction of the AJAX approach by Jesse James Garret in 2005. All the interface developers came out from their hiding places and they sure had been busy during that atomic winter. Out popped all kinds of amazing things like Dojo Toolkit, Yahoo! UI Library and JQuery. Once more the web moved forward at a rapid pace thanks to the newly found fame due to the AJAX hype. JavaScript was getting recognition as an actual language and CSS was being implemented in a more serious manner than before.

We all implemented our applications and sites using HTML, JavaScript and CSS2 and felt good about it. However it is worth noting that CSS2 was not a finished specification and no browser has yet to implement the entire CSS2 specification. Still, we have used loads of what is in CSS2 for years.

HTML5 - a new generation of web developers

Lulling in the wake of AJAX a new generation of web developers entered the scene. The WHATWG decided to take matters into their own hands and do what the W3C seemed incapable of, creating something which made creating web based applications with HTML easier. With the new hype of HTML5 we started to get all kinds of, what I thought, strange questions on blogs, Twitter and in industry media. “When can I start using HTML5?”, “When will it be ready?”, “I am so disappointed, all this hype and I can not use it”.

All these questions struck me as very odd and I was beginning to think there was something wrong with me, had I missed something? I had never heard these kinds of questions before regarding CSS, JavaScript or HTML. Sure, we had to take into consideration that not all browsers supported everything in the same manner. But that was what we as web developer did, right? We create web based applications using the best technology available to us meanwhile making sure older less capable browsers could view it to. This is what got my career started and what I have spent a lot of time doing as a living.

This new generation of developers seemed to be very focused upon only applying technology which where in standards that where finished. To me this sounded weird as we have never once before had any hesitation as to adopt new technology when it was available to us. When the XmlHttpRequest object was missing in browsers, what did we do? We used the old trick of the hidden IFrame. Back when Opera’s JavaScript support was horrendous (luckily this is no longer the case) what did we do? We made sure our stuff worked without the use of JavaScript. If something was not a final specification or something was not implemented in all browsers we implemented fallback solutions or provided alternate experiences.

Our bread and butter

Creating web applications using the latest technologies available to us should be our bread and butter as web developers. We should not wait around for the W3C to put some stamp on some documents. Browser vendors drive innovation, not standards bodies and therefor we should apply the technology being made available to us as quickly as we can, while making sure those with less capable browsers also can access our service or site. HTML5 does not make this any harder than before and we should not use the lack of a finalized HTML5 specification hold us back.

Web Standards are everything

You might be tempted into thinking that web standards are not that important, after all the browser vendors drive innovation. Innovation is done by the vendors, but the standards are extremely important as a way of making sure the web stays open and does not become fragmented. There are always numerous organizations trying to create their own special web, but thanks to the great work of the standard bodies the web has stayed open and will continue to do so.

The web standards (such as The Web Standards Project and WHATWG) community has really blossomed in the years after the introduction of AJAX and developers are way more conscious about the importance open standards. Without the work of the people in various working groups and standard bodies we would not be where we are today. Web technology is being used in more innovative ways than ever before. Not only restricted to making content or applications for the web, but also on the server side or to create native mobile applications.

In summary the web standards are not important, but still they are essential for us to continue to enjoy all the amazing service and applications created with open web technology.

This article was published in the August issue of the NDC Magazine from Programutvikling AS

Putting numbers that goes with the face we put on quality

2011-07-04T06:50:39+00:00

In the post Putting a face on quality we talked about how we wanted to bring our end users closer to our developers by visualizing tweets about our service. This was only one of many tasks we performed in order to get a better feel on how our work effects our users. In this post we’ll show you what we did in order to put numbers on end user quality by working with out customer service team.

Up until now our customer service team had a hard time visualizing the effect our code had on end users. They where publishing numbers on number of incoming requests, but these numbers where on a weekly basis at best.

Research in the wild

Therefor we set out with the goal of making all support requests visible in real time for the entire organization. Out hypothesis was that this would provide everyone on our teams from developers, product owners and sales staff with a clear view on what kind of quality they deliver.

In order to figure out what our application should look like we set out to see what the reality was and sat with our customer service team to see how they worked. But they had no system in place for logging requests and what area of our service it was about. All we had was the statistics from our support ticket system and call center. These systems did not provide enough detailed information, therefor our Lean task force team encouraged them to start logging requests on paper then put it all into a set of Excel spread sheets. Needless to say this required quite a bit of effort from the people one the customer service team and it was not a solution for the long term.

When deciding how to approach creating a application for logging requests we looked at how the analog logging process was performed. Realizing that a request could be for more than one problem area and effect multiple teams we realized that the paper layout was in fact ideal. We created a quick prototype in HTML and had the users test it. This prototype had no persistence and did only show how the process of logging a request would look and feel. We didn’t create anything else as this would’ve been waste if we where not to proceed with the application. By not building anything besides what was absolutely necessary we where able do deliver quickly (3-4 days) and we had no inventory which would become waste if our prototype didn’t match what our end users required. For simultaneous testing of the prototype we compared against an off the shelf product which used a wizard style approach to logging requests. Both prototypes was tested by the customer service team and the results where clear: the flexibility of the paper layout performed way better as it was quick and non-intrusive.

The penalty of success

With the success of the prototype we were now expected to deliver a solution which actually persisted the data and had a set of reports which would help visualize the requests coming in from our customers.

The application has to be delivered to the team actually responsible for all our back office applications and so this put some requirements on which technologies to use. We develop most of our services with Java so this would be the choice for the back end. On our mobile web solution m.finn.no we use RestEasy to provide a REST interface for our one-page JavaScript front end, therefor this would be the choice for our application as well. Along with Spring/Spring JDBC and a MySQL database this was our back end. We specifically opted for conservative technology choices as we wanted to hand over to be as smooth as possible. On the client side we opted to do a one page application written in JavaScript with the help of jQuery and a custom made publish-subscribe library along with Mustache for templating in our JS code.

Your friendly neighborhood log application

The only requirement from the customer was that it had to be effortless to log entries into our system. “There should be no reason not to log entries into the system”. We adopted what we call a “angstfri brukeropplevelse”, which translates into something like an anguish-free user experience. Everything else was pretty much up to us to decide. The grid based layout inherited from the analog log was a success. Therefor all we added was keyboard navigation and a cross hair behavior which provided the user with an easier way of seeing which column they where in to help avoid invalid log entries. We specifically avoided any kind of validation or other kinds of business rules as this would only make the threshold for logging a bit higher and this was not acceptable.

Having added a back-end and enhanced the user experience we did an initial test as soon as possible to get feedback on reports. A couple of tries later we where in production.

Quality ~= Visibility + Skills

Why did our architects set out to create an application for our customer service team you may ask? We focus upon enhancing the quality of what we deliver to our users and customers. Making sure everyone in our organization knows exactly how our work affect our users is one of the most important ways to make sure quality is always at the forefront in everything we do. This is why we used a couple of weeks to create this application. To quote some of the engineers at Facebook: “we want to bring your mother close to our developers”. We want everyone to feel that their mom is their users and who wants their mom to experience bugs or errors?

In order to make things visible we needed to iterate over our previous solution which visualized tweets about us on Twitter. We did a redesign and integrated some key figures into the application and the end result became really awesome, hence the name Awesome Board II. There is still one more piece of the plan for visualizing quality and it will be discussed in a later post. However you can view it on some of our pages already.

Graded browser support version 1.1

2011-06-27T13:41:23+00:00

We have reworked our graded browser support matrix which was released last year. Following the Laws of Simplicity we reduced the complexity in order to make it easier for everyone at our company to understand. We cut a lot of the version number into targeting latest for browsers which are frequently updated. There is also a support matrix for mobile phones which is targeting our new mobile web service m.finn.no.

In addition to updating the versions we have also made the decision to include tablets in the support matrix for our desktop verison of our service. This is because we are currently not looking at creating specialized versions for tablets and our service works just fine on the tablets currently hitting our site. Currently almost all of our trafic is coming from iPad’s, therefor Andoid powered devices are not supported.

Hosting our first ever hackaton

2011-06-24T07:09:41+00:00

Inspired by seeing how much fun and how much innovation is going on at hackatons all over the world we finally go round to hosting one ourselves. The first ever hackaton event at FINN.no was hosted on the 22-23. of june. At FINN.no we are accustomed to working in cross functional teams, so it was only natural that we invited the entire company to attend our hackaton. In order not to scare people with a non-technical background we had to change the name, so we created what we call FINNovasjonsdagen (which translates into FINNovationday). We decided to keep this one open for only employees since this was our first shot at this.

What happened?

In total there where 43 attendants spread out across 13 teams, where about half of the attendants had the opportunity to stay the entire time. After the 24 hours we invited the entire company to come and vote on which team had produced the coolest thing.

Check out this time laps video which was shot with a web cam:

The winners of the hackaton was awarded the price of 25 thousand NOKs and bragging rights until the next event. Team Meh created a global search for all of FINN.no fully integrated and almost ready to be rolled out into production. This has been something our users and employees have been supporting for years and to see that a team of three was able to crank it out in 24 hours was quite an accomplishment.

Check out the blog[Gone] from the event for more pictures and more details about all the things created (English translation of the blog).

Dependency Injection with constructors?

2011-05-12T23:01:00+00:00

The debate whether to use constructors, setters, fields, or interfaces for dependency injection is often heated and opinionated. Should you have a preference?

The argument for Constructor Injection

We had a consultant working with us reminding us to take a preference towards Constructor injection. Indeed we had a large code base using predominantly setter injection because in the past that is what the Spring community recommended.

The arguments for constructor injection goes like:

Dependencies are declared public, providing clarity in the wiring of Dependency Inversion,
Safe construction, what must be initialised must be called,
Immutability, fields can be declared final, and
Clear indication of complexity through numbers of constructor parameters.

And that Setter injection can be used when needed for cyclic dependencies, optional and re-assignable dependencies, to support multiple/complicated variations of construction, or to free up the constructor for polymorphism purposes.

Being a big fan of Inversion of Control but not overly of Dependency Injection frameworks something smelt wrong to me. Yet solely within the debate of constructor versus setter injection i don’t disagree that constructor injection has the advantage. Having been using Spring’s dependency injection through annotation a little recently and building a favouritism towards field injection I was happy to get the chance to ponder it over, to learn and to be taught new things. What was it i was missing? Is there a bigger picture?

API vs Implementation

If there is a bigger picture it has to be around the Dependency Inversion argument since this is known to be potentially complex. The point here of using constructor injection is that 1) through a public declaration and injection of dependencies we build an explicit graph showing the dependency inversion throughout the application, and 2) even if the application is wired magically by a framework such injection must still be done in the same way without the framework (eg when writing tests). The latter (2) is interesting in that the requirement on “dependency injection” is too also inverted, that the framework providing dependency injection is removed from the architectural design and becomes solely a implementation detail. But it is the graph in (1) that becomes an important facet in the following analysis.

With this dependency graph in mind what does happen when we bring into the picture a desire to distinguish between API and implementation design…

The DI graph being requested to be clarified by using constructor injection will fall into one of two categories: → ‘implementation-specific’, an interface defines the public API and the DI is held private by the constructor in the implementation class, → ‘API-specific’ when the class has no interface. Here everything public is a fully exposed api. There is no implementation-protected visibility here for injectable constructors.

By introducing the constraint of only ever using constructor based injection: in the pursuit of a clarified dependency graph; you remove or make more difficult the ability to publicly distinguish between API and Implementation design.

This distinction between API and implementation is important in being able to create the simple API. The previous blog “using the constretto configuration factory” is a co-incidental example of this. I think the work in Constretto has an excellent implementation design to it, but this particular issue raised frustrations that the API was not as simple as it could have been. Indeed: to obtain the “simplest api”; Constretto (intentionally or not) promotes the use of Spring’s injection, a loose coupling that can be compared to reflection. It may be that our usage of Constretto’s API, where we wanted to isolate groups of properties, was not what the author originally intended but this only re-enforces the need for designing the simplest possible API.

Therefore it is important to sometimes have all dependency injection completely hidden in the implementation. A clean elegant API must take precedence over a clean elegant implementation. And to achieve this one must first make that distinction between API and Implementation design.

Taking this further we can introduce the distinction between API and SPI. Here a good practice is to stick to using final classes for API and interfaces for SPI. By the same argument as above SPI can’t use constructor injection because they don’t have constructors.

Inversion-of-Control vs Dependency-Injection

What about the difference between IoC and DI. They are overlapping concepts: the subtlety between the “the contexts” and “the dependencies” rarely emphasised enough. (Java EE 6 has tried to address the distinction between contexts and dependencies at the implementation level with the CDI spec.) The difference between the two, nuanced as it may be, can help illustrate that the DI graph in any application deserves attention in multiple dimensions.

Drawing an application’s architecture up as a graph where the vertical axis represents the request stack: that which is typically categorised into architectural layers view, control, and model/services; and the horizontal axis representing the broadness of each architectural layer, then it can be demonstrated that: → IoC generally forms the passing and layering of contexts downwards.

→ The api-specific DI is fulfilling the layer of such contexts, and these contexts can be dependencies directly or helper classes holding such dependencies. Such dependencies must therefore be initially defined high up in the stack.

→ The DI that is implementation-specific is at most only visible inside each architectural layer and is the DI that is represented horizontally on the graph. Possibly still within the definition of IoC it can also be considered a “wiring of collaborating components”. The need for clarity in the dependency graph isn’t as critical and so applications here often tend towards Service Locators, Factories, and Injectable Singletons. On the other hand many of the existing Service Locator implementations have been poor enough to push people towards (and possibly it was an instigator for the initial implementations of) dependency injection.

→ Constructor injection works easily horizontally, especially when instantiation of objects is under one’s ability, but has potential hurdles when working vertically down through the graph. Sticking to constructor injection horizontally can also greatly help when the wiring of an application is difficult, by ensuring at the construction of each object dependency injection has been successful. Missing setter, field, interface injection and Service Locators may not report an error until actually used in runtime.

A simple illustration of difficulty with vertical constructor injection is looking at these helper contexts and how they may be layering contexts through delegation rather than repetitive instantiation, a pattern more applicable for an application with a deep narrow graph. This exemplifies a pattern that has often relied on proxy classes.

Another illustration is when having to instantiate the initial context at the very top of the request/application stack it involves instantiating all the implementation of dependencies used in contexts down through the stack, this is when dependency inversion explodes - the case where the IoC becomes up-front and explicit, and the encapsulation of implementation is lost through an unnecessary leak of abstractions. A problem paralleling to this is trying to apply checked exceptions up through the request stack: one answer is that we need different checked exceptions per architectural layer (another answer is anchored exceptions). With dependencies we would eventuate with requiring different dependency types per architectural layer and this could lead to dependencies types from inner domains needing to be declared from the outer domains. Here we can instead declare resource loaders in the initial context and then letting each architectural layer build from scratch its own context with dependencies constructed from configuration. But this comes the full circle in coming back to a design similar to a service locator. Something similar has happened with annotations in that by bringing Convention over Configuration to DI what was once loose wiring with xml has become the magic of the convention and begins too to resemble the service locator or naming lookups.

For a legacy application this likely becomes all too much: the declaring of all dependencies required throughout all these contexts; and so relying on a little louse-coupling-magic (be it reflection or spring injection) is our answer out. Indeed this seems to be one of the reasons spring dependency injection was introduced into FINN. And so we’ve become less worried about the type of injection used…

Broad vs Deep Applications

FINN.no is generally a broad application with a shallow contextual stack. Here is the traditional view-control-model design and the services inside the model layer typically interact directly with the data stores and maybe interact with one or two peer services.

Focusing on the interfaces to the services we see there is a huge amount of public api available to the controller layer and very little in defined contexts except a few parameters, or maybe the whole parameter map, and the current user object. There is therefore very little inversion of control in our contexts, it is often just parameterisation. (Why we often use interfaces to define service APIs is interesting since we usually have no intention for client code to be supplying their own implementations, it is definitely not SPIs that are being published. Such interfaces are used as a poor-man’s simplification of the API declaration of public methods within the final classes. Albeit these interfaces do make it easy to make stubs and mocks for tests.)

In this design the implementation details of service-layer dependencies is rarely passed down through contexts but rather hard baked into the application. And in a product like FINN it probably always will be hard baked in. Hard baked here doesn’t mean it can’t be changed or mocked for testing, but that it is not a dynamic component, it is not contextual, and so does not belong in the architectural design of the application.

In such a broad architectural layer i can see two problems in trying to obtain a perfect DI graph:

→ cyclic dependencies: bad but forgiven when existing as peers within a group. In this case constructor injection fails. We can define one as the lesser or auxiliary service and fall-back to the setter/field injection just for it, but if they are real equal peers this could be a bullet-in-the-foot approach and using field injection for both with documentation might be the better approach.

→ central dependencies: these are the “core” dependencies used throughout the bulk of the services, the database connection, resource loaders, etc. If we enforce these to be injected via constructors then we in turn are enforcing a global-store of them. Such a global store would typically be implemented as a factory or singleton. Then what is the point of injection? Worse yet is that this could encourage us to start passing the spring application context around through all our services. A service locator may better serve our purpose…

Hopefully by now you’ve guessed that we really should be more interested in modularisation of the code. Breaking up this very broad services layer into appropriate groups is an easier and more productive first step to take. And during this task we have found discovering and visualising the DI graph is not the problem. Untangling it is. Constructor injection can be used to prevent these tangles, but so can tools like maven reporting and sonar. This shows the the DI graph is actually more easily visualised through the class’s import statements than through constructor parameters.

With modularisation we can minimise contexts, isolate dependency chains, publish contextual inversion of control into APIs, declare interface-injection for SPIs, and move dependency injection into wired constructors.

Back to Constructor injection

So it’s true that constructor injection goes beyond just DI in being able to provide some IoC. But it alone can not satisfy Inversion of Control in any application unless you are willing to overlook API and SPI design. DI is not a subset or union of IoC: it has uses horizontally and in loose-coupling configuration; and IoC is not a subset or union of DI: to insinuate such would mean IoC can only be implemented using spring beans leading to an application of only spring beans and singletons. In the latter case IoC will often become forgotten outside the application’s realm of DI.

Constructor injection is especially valid when it’s desired for code to be used via both spring-injection and manual injection, and it does make test code more natural java code. But imagine manually injecting every spring bean in a oversized legacy broad-stack application using constructor injection without the spring framework - is this really a possibility, let’s be serious? What you would likely end up with is one massive factory with initialisation code constructing all services instead of the spring xml, and looking up using this factory every request. What’s the point here? This isn’t where IoC is supposed to take us.

If code is being moved towards a distributed and modular architecture you should pay be aware on how it clashes with the DI fan club.

If code is in development and you are uncertain if the dependency should be obtained through a service locator or declared public giving dependency inversion, and in the spirit of lean think it smart to not yet make the decision, then using field injection can be the practical solution.

And just maybe you are not looking to push the Dependency Inversion out into the API and because you think of Spring’s ApplicationContext (or BeanFactory) as your application’s Service Locator, you use field injection as a method to automate service locator lookups.

For the majority of developers for the majority of the time you will be writing new code, not caring about dependency injection trashing inversion of control, wanting lots of easy to write tests, and not be worrying about API design so it’s ok to have a healthy preference towards constructor injection…

Keep questioning everything… …by remaining focused on what is required from the code at hand we can be pragmatic in a world full of rules and recommendations. This isn’t about laziness or permitting poor code, but about being the idealist: the person that knows the middle way between the pragmatist and ideologue. By knowing: when what can, and for how long, be dropped; we can incrementally evolve complex code towards a modular design in a sensible, sustainable, and practical way. In turn this means the programmer gets the chance to catch breath and remember paramount to their work is the people: those that will develop against the design and those end-users of the product.

References:

Martin Fowler gives a great introduction into all this at martinfowler.com/articles/injection.html.
Taking taking this further Jaroslav Tulach presents an interesting view on Dependency Injection in relations to API design at wiki.apidesign.org/index.php?title=Dependency_Injection
Another read that investigates IoC : Constructor Injection vs Setter Injection - Shaun Smith
What’s the difference between Dependency Injection and Inversion of Control

Credits: A large and healthy dose of credit must go to Kaare Nilsen for being a sparring partner in the discussion that lead up to this article.

Putting a face on quality

2011-05-09T12:44:05+00:00

The topic of ensuring quality in the services we provide is a much debated topic in our industry [1]. It tends to focus upon things such as testing and how to best automate it to try and reach a state of zero defects. These initiatives are great and we should all pursue them in our daily work. These debates tend to be very focused upon the technical side and how to prevent the intrusion of defects in our build cycle. A result of the technical focus is that you become abstracted from the real issues at hand and it becomes yet another ideological debate (or even worse yet another pissing contest between believers and non-believers).

At FINN.no we try to always view things from an end-user perspective and technical quality is no different. So what does technical quality mean for our end users and do they really care?

Making dreams come true

A service such as FINN.no is much more than just a site where you view classified ads. It is a service where everyday people make some of their dreams come true. They buy the house they have been dreaming of in order to start a family. It is a place where you buy things to keep your kids safe and comfortable. So in order to answer the previous question, do they care about quality? Naturally. Down time on a Sunday at FINN.no prevents people from buying the house, the car or the boat of their dreams.

You get the picture right? Quality of service means that we deliver on our promisse of being a marketplace you can rely on to make some of your dreams come true. Behind the discussions about unit-test coverage there is a family not getting their house, car or boat if you mess things up. When we debate whether to write tests up-front or do waterfall planning there is someone out there not getting their car sold. We get so caught up in technology some times that we actually think that it matters. In order to try and get more in touch with how our quality affects our users we realized we had to do something. When growing from a small company to a larger company you often end up lossing track of these things and we needed to correct this.

Continuous improvements a.k.a. Lean

At FINN.no we have some people work solely with teaching us continuous improvement. They try to help every team or department to learn the Lean way of working and resolving issues. By spreading their knowledge across the company they start a lot of cross-team-department-initiatives which would not happen without them.

Bulding quality into your product is one of the essential things towards creating an amazing user experience, but the problem is how do you do this on a regular basis? There are numerous tools available for all kinds of testing and quality analysis of code and things like that. These tools are all good and you should use them. However, they all fail to bring you the one insight which is the most valueable yo your business: what does your users actually think about what you do.

Our customer service and architectural teams have both been working to try and establish a way of working which ensure continuous improvement in what we do. This has resulted in an initiative to try and visualize how our level of quality effects our users and customers.

FINNback or Tweet Board

One easy way of putting a face on quality was to utilize the Twitter Streaming API which enables us to see what people think about our service. Thanks to the brilliant engineers at Twitter this was easily accomplished in just a few hours and we had a node.js application which displayed tweets about us on monitors/TVs in our cantina and in our reception. Thanks to the awesomeness of the Twitter Streaming API we are able to pull both the profile picture, the profile background image and the number of followers into our application in order to give a more human feel to the tweets coming in. This was critical as it makes the tweets real as they come from real live people who we effect with our work. In the sample screenshot of the FINNback application we get a confirmation of something we are painfully aware of, we should have an app.

Doing this Twitter feed visualization is just a first step and we are working hard towards creating more real-time feedback visualization in order to make sure every one of our employees knows exactly how our users feel about what we do. We are confident that this is a great way of getting focus on technical quality and to make sure we work even harder to provide a service which makes our users dreams come true.

XSS protection: who's responsibility?

2011-04-08T07:09:39+00:00

In a multi-tier application who can be responsible for XSS protection? Must security belong to a dedicated team…or can it be a shared responsibility? Today XSS protection is typically tackled by front end developers. Let’s challenge the status quo.

New Applications vs Legacy Applications

For protection against Stored XSS many applications have the luxury of ensuring any text input from the user, or from a CMS system, is made clean and safe before being written to database. Given a clean database the only XSS protection required is around request values, for example values from url parameters, cookies, and form data.

But some applications, especially legacy applications, are in a different boat. Databases typically have lots of existing data in many different formats and tables so it’s often no longer feasible to focus on protecting data on its way into the system. In this situation it is the front end developers that pay the price for the poor quality of backend data and are are left to protect everything. This often results in a napalm-the-whole-forest style of xss protection where every single variable written out in the front end templates goes through some equivalent of

<c:out value="${someText}"/>

This makes sense but… if you don’t have control is your only option to be so paranoid?

A Messed up World

To illustrate the problem let’s create a simple example by defining the following service

  interface AdvertisementService{
    Advertisement getAdvertisement(long id);
  }

  interface Advertisement{
    /** returns plain text title */
    String getTitle();
    /** return description, which may contain html if isHtmlEnabled() returns true */
    String getDescription();
    /** indicates the description is html */
    boolean isDescriptionHtml();
  }

The web application, having already fetched an advertisement in the control tier, somewhere would have a view template looking something like

    <div>
        <h1><c:out value="${advertisement.title}"/></h1>
        <p>
            <c:out value="${advertisement.description}"
                      escapeXml="${!advertisement.descriptionHtml}"/>
        </p>
    </div>

Here we add another dimension: simple escaping with c:out won’t work if you actually want to write html (and what is the safety and quality of such html data).

When this service is used by different applications each with their own view templates, and maybe also exposed through web services, you end up no doubt protecting it over many times, system developers in the web services, and front end developers in each of the presentation layers… likely there will be confusion over the safety and quality of data coming from this service, and of course everyone will be doing it differently so nothing will be consistent.

Is there a better way?
Can we achieve a consistent XSS protection in such an environment?
How many developers need to know about XSS and about the origin of the data being served?

In the above code if we can guarantee that the service always returns safe data then we can simplify it by removing the isDecriptionHtml() method. The same code and view template would become

  interface AdvertisementService{
    Advertisement getAdvertisement(long id);
  }

  /** All fields xss protected */
  interface Advertisement{
    /** returns plain text title */
    String getTitle();
    /** return description, which may contain safe html */
    String getDescription();
  }



    <div>
        <h1>${advertisement.title}</h1>
        <p>${advertisement.description}</p>
    </div>

By introducing one constraint: that all data is xss protected in the services tier; we have provided a simpler and more solid service API, and allowed all applications to have simpler, more concise, more readable, view templates.

Solutions

Having a clean database all non-html data can be escaped as it comes in. Take advantage of Apache’s StringEscapeUtils.escapeHtml(..) from commons-lang library. For incoming html one can take advantage of a rich html manipulation tool, eg like JSoap, to clean, normalise, and standardise it.

With legacy or foreign data, especially those applications with exposed service architecture and/or multiple front ends, a different approach is best: ensure nothing unsafe ever comes out of the services tier. For the html data the services will often be filtering many snippets of html over and over again, so this needs to be fast and a heavy html manipulation library like JSoap isn’t appropriate any more. A suitable library is xss-html-filter, a port from libfilter. It is fast and has an easy API

String safeHtml = new HTMLInputFilter().filter( "some html" );

If we do this it means xss protection is not duplicated, but rather made clear who is responsible for what, c:out becomes just junk in verbosity and performance* , service APIs become simpler, view templates look the same for both new and legacy applications, system developers become responsible for protection of database data and this creates a natural incentive for them to clean up existing data and ensure new data comes in safe,

No matter what an inescapable fact is all front ends developers must have a concrete understanding that any value fresh from the request is prone to Reflected XSS and there’s nobody but them that can be responsible for protecting these values.

XSS protection has become a basic knowledge requirement for all developers. Like much to do about security … it’s always only as strong as its weakest link.

At FINN.no because we take security seriously, and because we know we are only human and we need some room for the occasional mistake, for each release we run security audits. These include using WatchCom reports, tools like Acunetix, and custom tests using Cucumber.

Multiple versions of Chrome on OS X

2011-04-07T13:37:45+00:00

As a professional web developer you should use the bleeding version of Google Chrome, so you’re prepared what your users will see a few weeks ahead. But the dev version tends to be a bit more buggy (naturally), and you have to test that your page works with both the beta and the stable version. Google Chrome has a command-line parameter to specify another profile, but it’s a bit tricky to add command-line parameters to a Mac application. Yesterday I found a blog-post by Duo Consulting. They have made a nice little script that will generate an app with the profile you specify. I modified it a bit to allow running different versions of Google Chrome as well as a profile for each of them.

To make the install process even easier for others I’ve zipped the generated applications (they just contain a script).

Disclaimer: Each version will have it’s own profile, so you have to set up each one from scratch with bookmarks, plugins and everything else. As Scott mentions in the comments, you can use the sync feature in Chrome to keep the installations in sync.

First download each version and rename the original Google Chrome apps to “Google Chrome Stable”, “Google Chrome Beta” and “Google Chrome Dev” and put them in Applications under your user folder. I.e.: /Users/gregers/Applications/
Unzip Chrome-Trio.zip(File removed) and put the apps in /Applications
Don’t select any of them as default browser, since that will use the default profile instead of the custom. Instead start Safari -> Preferences -> General -> Default web browser -> Select… -> Go to /Applications and choose the version of Chrome you want as default :)

If you’re interested in the script I used to make the wrapper-apps. You can find it here: http://pastebin.com/gPSdNi40

The Great Leap - getting serious with JavaScript at FINN.no

2011-04-01T11:45:39+00:00

FINN.no are blessed to be based in the same country as one of the brightest young stars when it comes to testing in JavaScript and applying Test Driven Development: Christian Johansen (the poster above was used to promote the event). He works as a developer at Gitorious during the day time, but he also cranks out frameworks like SinonJS and he has even published a book on how to do Test Driven Development in JavaScript.

Driving your design with tests

We first had an amazing live coding session which he had previously done at FrontTrends. The task was to create a type-ahead widget which sent requests to a server if the delay was more than 50 millies. Driving the design of an HTML widget with tests is just as awesome as doing the same thing with JUint on the back-end. You get a nice set of really simple objects which provide you with a set of tools to create the functionality you want. This is very different from the one-object-with-everything kind of code you see a lot of when you do not focus upon design before you code. You can accomplish good clean code without

Refactor some legacy code with tests

We have just recently ported parts of our platform related to advanced search to a new Java framework, but we did not port much of the JavaScript code. Only thing we have done is to prevent it from flooding the global scope and extract the JS code into a separate file. Christian gave us the challenge of trying to test this code. This was by no means an easy task. The code was not written with testing in mind. It has some weird coding errors which for some reason does not produce errors in the user interface. Stuff like a function which one would expect to return a boolean, but when we tried to assertTrue on the result of a validation it failed and we couldn’t see why.

function validateForm() {
    var valid = true;
    $("input.number").each(function(){
        valid &= validateInput($(this));
    });
    return valid;
    }
...
    if (validateForm()) { ....}

That was until we noticed the bitwise and operator used to indicate valid, which returns 0 or 1. Not true or false which you might expect since the valid variable is initialized with true.

Two wrongs make a right?

To proceed with testing this advanced search field we had to assert whether the red border was on or not. This was the only thing visible from outside of the object. We hit a bit of a snag when it came to the css-function in jQuery, which it turns out does not return a color if you pass in just border. However if you pass in border-color-left you get the result rgb(204, 0, 0).

function validateInput(element) {
    if (val.length > 0 && val.search(/^\d+$/)) {
        element.css({border:"1px solid #CC0000"});
        element.focus(function() {
            element.css({border:"1px solid #BBBBCC"});
        });
        return false;
    }
    return true;
}

These are just a few of the thing we needed to figure out in order to add a test harness to some of our legacy code. It is very very hard work and you really need some cojones to start doing it. It was truly inspiring to have Christian over and we just have no excuse anymore for not writing tests for our JavaScript code. This is a huge leap forward in terms of getting serious with our JavaScript code.

Style sheet for touch devices

2011-03-10T13:05:53+00:00

It’s easy to include a style sheet for small-screen mobile devices, but what if you specifically want to target touch devices. Then you also need to include devices with larger screens. The iPad has a resolution of 1024x768, but many desktop users also have this resolution, so you can’t just increase the max-device-width in your CSS media query. I’ve tried to find a better solution to the problem, but none seems to exist yet.

So the best solution I’ve found is to use JavaScript to detect if the browser has touch events. It’s a bit dangerous because Firefox and Chrome have support for touch, so you can’t be sure they don’t expose the touch events to devices without a touch screen. Fortunately this has already happened in Google Chrome. The team decided to disable the touch events so the detection developers already used would continue to work.

Paul Irish has researched a bit for Modernizr to find out what detection methods work best. You can of course use Modernizr, but if all you need is this test it’s not necessary to include the whole library. All you really need is this:

'ontouchstart' in window

At first I did the test in the initialization code on document.ready, but then the style sheet will load too late. I definitely don’t like document.write in external scripts because then you can’t load them asynchronously in an easy manner. However, now it’s actually useful, since we want the style sheet to be written out and loaded before the page is visible.

<script type="text/javascript">
    if("ontouchstart" in window) {
        document.write('<link rel="stylesheet" type="text/css" media="only screen" href="/css/touch.css">');
    }
</script>

Anyone have a better way?

Getting Geeky in Hønefoss

2011-03-03T09:35:37+00:00

FINN recently created a position called Technical Leader which is going to be our champions of technical excellence. One of the objects for this two-day field trip was to get everyone in the same room and share ideas and challenges from a technological point of view.

Quality does not cost … it pays

Most companies talk about how they work hard to “produce quality”, but most companies also fail to follow through and make quality a focus for the entire organization. Our first session was about defining what quality means for different stakeholders such as: end users, our CEO, operations and product owners. These discussions lead us to a first draft of a checklist which will be used before each new release.

No known bugs
Tested in every environment
Customer support is informed of important changes

Walk’n talk for good mental health

The term walk and talk is both a technique for making movies, but it is also a therapeutical technique. At FINN we practice this by randomly pairing two people who both have prepared a challenge they have in their daily work and then they walk and talk for about an hour. This was the first time this group did this exercise and it was a huge success. It was great to get out and talk about stuff to someone you do not get to talk to very often during the work day. As a team building exercise I think this is a great tool which is highly recommended!

The challenge of company Open Space sessions

During day two we had three open space sessions which was buzzing with excitement. Doing open spaces at conferences it is really inspiring and you learn stuff which you can bring back to your organization. Having open space sessions for just company employees has a different dynamic and can really be something to get your spirits down. When you do this inside a company you build up excitement during the sessions and along with the excitement comes expectations that “somebody needs to make this happen”. There is nothing in “open space theory” about how you can easily get more out of these session than just a lot of nice thoughts. This would resemble putting up a suggestions box and then do nothing with any of the suggestions. What we did was to get some people together after the session to see what actions needed to be taken after the sessions. Kind of like a retrospective where we looked at what came out of the open spaces. It would be interesting to hear what others do with their internal open space sessions.

Developing for mobile is punk rock!

2011-02-01T19:08:24+00:00

The title of this post is of course a blatant lie. Mobile development is nothing but pain from here until eternity (no, not really). But, luckily the brave developers at FINN.no aren’t scared of a little pain and therefor we decided to do a mobile development trippel header gig to showcase some of the different approaches to mobile developement.Due to some kids falling ille we had to reduce the show to be a double header. It did not end up to make a difference as the guys doing the talks totally kicked ass!

Oh and the picture on the left is a flyer we put up everywhere to market the event at the FINN.no headquarters. Finnfrontend is a name for one of our communities of practice dedicated to front-end development. Unfortunately it was reduced to a double header and we missed out on getting the low down on iOS development of native apps.

Mobile web on m.finn.no

Frank and Sven Inge from Team Mobile at FINN.no presented the beta for m.finn.no. As announced previously on labs.finn.no[Gone] the beta for our mobile web service is now available for testing and feedback. We are very excited about this application and we feel that it takes mobile web to a new level here in Norway.

Frank presented some of the plans going forward with the beta (which I am not going to tell here). He also shared some of the experiences and pains with developing for mobile devices. Naturally there are quite a few challenges with compatibility and providing a consistent look and feel across multiple platforms. This has been the bread and butter for web developers for years and it is no surprise that it is hard on mobile devices.

The Team also talked about some of the design decisions they have made and why they ended up solving a challenge a certain way. Choosing to display all information and letting the user scroll as opposed to applying the show/hide-style functionality we are used to from the desktop web seems like a really good choice.

The mobile web solution looks awesome and we can not wait to see the final version! I can only say that it definitely will ship before the 17th of May!

App development with HTML+CSS and Phonegap

Audun Kjelstrup from Nordaker shared his experiences with mobile app development using HTML+CSS+JavaScript for multiple platforms using Phonegap. Drawing on his past experiences with developing the Dolly Dimple pizza configurator for iPhone and the NHO Conference app for iOS, Android and Symbian he covered some of the challenges with this kind of development.

Audun recommended going for a web based app if you wanted a user experience which differed a lot form what comes out of the iOS UIKit. This was mainly due to the cost of development and the work required to create custom experiences such as the Dolly Dimple pizza configurator. He also pointed out that web based apps will always struggle to have the exact same snappyness and scroll smoothness as a native application. Creating a web based app for multiple platforms is hard work (no real surprise there) and having to deal with older Symbian based phones was a lot of pain (no real surprise there either), so you should consider what platform you target carefully. The slides from this presentations can be downloaded from the Cocoa Heads Oslo Meetup where it was held some days ago.

This was an awesome session with two great talks which highlighted some of the challenges with creating applications for mobile devices across multiple platforms. Thanks to Audun and Team Mobile!

Using the Constretto Configuration Factory

2011-01-19T18:07:02+00:00

Development at FINN.no was quick to jump on the Constretto library. It’s a great addition to any application making it easy to handle different environments and profiles with just the one build.

When using Constretto at FINN.no one problem we found with the API is that it creates very verbose client code when the same ini or properties file is accessed throughout various class files.

For example a repetition of the following happens

public static final String CONSTANT_A = new ConstrettoBuilder()
    .createIniFileConfigurationStore()
    .addResource(new DefaultResourceLoader().getResource("classpath:file.ini"))
    .done()
    .getConfiguration()
    .evaluateToString("CONSTANT_A");

public static final String CONSTANT_B = new ConstrettoBuilder()
    .createIniFileConfigurationStore()
    .addResource(new DefaultResourceLoader().getResource("classpath:file.ini"))
    .done()
    .getConfiguration()
    .evaluateToString("CONSTANT_B");

One way around this is to use spring injection. This makes nice concise code but comes with two flaws: the constants can no longer be constant (they have to be member fields), and you have to lump all your properties together into one big “globals” store (days of f77 comes to mind). A third problem with this approach is that you’re passing Constretto’s context through Spring’s context, we don’t really want chained contexts but direct access to individual contexts if possible.

So looking for a more elegant solution to all this we wrote a simple factory, that has now been accepted upsteam, that provides the necessary convenience methods so the constants can be declared short, sweet, and simple

import static org.constretto.util.StaticlyCachedConfiguration.config;

public static final String CONSTANT_A = config("classpath:file.ini").evaluateToString("CONSTANT_A");
public static final String CONSTANT_B = config("classpath:file.ini").evaluateToString("CONSTANT_B");

This also caches the ConstrettoConfigurations improving your application’s startup performance.

This was taken from Constretto’s issue CC-14

Graded browser support at FINN.no

2011-01-07T14:16:09+00:00

Inspired by the kick-ass developers of Yahoo! and their work on Graded Browser Support (GBS) we at FINN.no decided to adopt graded browser support as a way to communicate what level of support we have for different combinations of browsers and platforms.

Creating the support matrix

FINN is the largest site in Norway when it comes to traffic and we have a good framework for statistics. In order to create the support matrix we took the statistics for the most popular browsers and platform combinations and put them in a grid. This gave us a matrix showing us which browser/platform combinations we need to consider from a business perspective. Although this was a good start, we also needed to figure out what level of support we should provide each combination. Numbers and usage alone does not provide a good enough basis upon setting support levels.

The cost of support

Having read the early drafts of the awesome book Secrets of a JavaScript ninja by John Resig we decided to follow his approach on creating a GBS matrix and perform a cost-benefit-analysis. We did was to do a quick survey among our developers in order to figure out what the costs where for supporting different user agents on certain platforms. The results are displayed in the chart below and shows our subjective opinions on how much effort is required to support a certain browser. Note that this chart will not look the same for everyone. It is will vary based on the skills of your developers, what browsers they work in, etc, etc.

Support levels

Our levels of support are slightly more simple than those of Yahoo! and we only provide three levels:

A-support: no visual or functional errors, all errors are reported and all features should be tested for each release.
B-support: no functional errors, core features should be tested for each release, visual errors are only reported with a low priority
C-support: no functional errors in core features, all other errors are not reported.

This is how we created our GDS matrix and we hope this will inspire you to do the same for your shop. We are going to do a similar exercise when it comes to mobile, but this is still work in progress.

The matrix is available on labs.finn.no (the url is no longer available).

the ultimate view

2010-11-04T18:11:57+00:00

This article has been rewritten for Tiles-3.
tech.finn.no/the-ultimate-view-tiles-3/

A story of getting the View layer up and running quickly in Spring, using Tiles with wildcards, fallbacks, and definition includes, to use the Composite pattern and Convention over Configuration providing a minimal ongoing xml changes.

Summary

From the architect’s perspective you see Apache Tiles as rare exotic Italian marble sheets laid out exquisitely, while the same architect will see SiteMesh as the steel wiring stuck inside the concrete slab. The beauty of the tiles is always admired and a key component in creating an eye-catching surrounding.

Summary
Background
Step 0: Spring to Tiles Integration
Step 1: Wildcards
Step 2: The fallback pattern
Step 3: Definition includes
When the Composite pattern is superior
Conclusion

Background

At FINN.no we were redesigning our control and view layers. We, being the architectural team of six, had already decided on Spring-Web as a framework for the control layer due to its flexibility and a design for us providing better, simpler, migration path. For the front end we were a little unclear. In a department of ~60 developers we knew that the popular vote would lead us towards SiteMesh. And we knew why for practical purposes sitemesh gives the front end developer more flexibility and less xml editing. But we knew sitemesh has some serious shortcomings…

SiteMesh shortcomings:

from a design perspective the Decorator pattern doesn’t combine with MVC as elegantly as the Composite pattern does
requires to hold all possible html for a request in buffer requiring large amounts of memory
unable to flush the response before the response is complete
requires more overall processing due to the processing of all the potentially included fragments
does not guaranteed thread safety
does not provide any structure or organisation amongst jsps, making refactorings and other tricks awkward

One of the alternatives we looked at was Apache Tiles. It follows the Composite Pattern, but within that allows one to take advantage of the Decorator pattern using a ViewPreparer (URL not working anymore). This meant it provided by default what we considered a superior design but if necessary could also do what SiteMesh was good at. It already had integration with Spring, and the way it worked it meant that once the Spring-Web controller code was executed, the Spring’s view resolver would pass the ball onto Tiles letting it do the rest. This gave us a clear MVC separation and an encapsulation ensuring single thread safety within the view domain.

Yet the most valuable benefit Tiles was going to offer wasn't realised until we [started experimenting a little more...](/2010/11/04/the-ultimate-view/2/)

Step 0: Spring to Tiles Integration

The first step was integrating [Tiles and Spring](http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/view.html#view-tiles) together. There exists tutorials on this already, it boils down to registering both a ViewResolver and a TilesConfigurer in your spring-web configuration.

    <bean id="viewResolver" class="org.springframework.web.servlet.view.tiles2.TilesViewResolver">
	    <property name="order" value="1"/>
	    <property name="viewClass" value="org.springframework.web.servlet.view.tiles2.TilesView"/>
    </bean>
    <bean id="tilesConfigurer" class="org.springframework.web.servlet.view.tiles2.TilesConfigurer">
	    <property name="tilesInitializer">
		    <bean class="no.finntech.control.servlet.tiles.FinnTilesInitialiser"/>
	    </property>
    </bean>

The TilesConfigurer hooks into the "tilesInitializer" class that you must supply yourself. This is the new way in Tiles-2 of providing the [configuration](http://tiles.apache.org/config-reference.html). Our FinnTilesInitialiser looks like

    public class FinnTilesInitialiser extends AbstractTilesInitializer {
        public FinnTilesInitialiser() {}

        @Override
        protected AbstractTilesContainerFactory createContainerFactory(TilesApplicationContext context) {
            return new FinnTilesContainerFactory();
        }
    }

Step 1: Wildcards

Here we hit Tiles’ most obvious downfall: every single jsp included must be declared in a definition in the tiles.xml file. Anyone that ever tried Tiles-1 knows this bad smell.

Tiles-2 though provides a way to avoid it by using wildcards, and when i hear people talking about tiles it’s clear this Tiles-2 “dynamic composite” paradigm hasn’t really caught on.

Let's declare a definition and template like (and create some files and folders as shown to the right):


//--tiles.xml
    <definition name="REGEXP:(.+)" template="/WEB-INF/tiles/template.jsp">
	    <put-attribute name="meta" value="/WEB-INF/tiles/{1}/meta.jsp"/>
	    <put-attribute name="header" value="/WEB-INF/tiles/{1}/header.jsp"/>
	    <put-attribute name="body" value="/WEB-INF/tiles/{1}/body.jsp"/>
	    <put-attribute name="footer" value="/WEB-INF/tiles/{1}/footer.jsp"/>
    </definition>






 //--template.jsp
    <!DOCTYPE html>
    <%@ prefix="tiles" taglib uri="http://tiles.apache.org/tags-tiles" %>
    <html>
    <head>
    <tiles:insertAttribute name="meta"/>
    </head>
    <body>
    <div id="header"><tiles:insertAttribute name="header"/></div>
    <div id="body"><tiles:insertAttribute name="body"/></div>
    <div id="footer"><tiles:insertAttribute name="footer"/></div>
    </body>
    </html>

Now our Spring controller can return definitions that match the names of the folder containing meta, header, body, and footer.

Now we that we have three ready to go definitions “cat”, “dog”, and “cow”. Spring controllers can now use them like this

    @Controller
    public class MyWebsite{
        @RequestMapping("/cat")
        public ModelAndView viewCat(){
            Map modelMap = new HashMap();
            //...
            return new ModelAndView("cat", modelMap);
        }
        @RequestMapping("/dog")
        public ModelAndView viewDog(){
            Map modelMap = new HashMap();
            //...
            return new ModelAndView("dog", modelMap);
        }
        @RequestMapping("/cow")
        public ModelAndView viewCow(){
            Map modelMap = new HashMap();
            //...
            return new ModelAndView("cow", modelMap);
        }
    }

What’s brilliant here is that we can keep adding new definitions and matching jsps in new folders without ever having to edit the xml.

Step 2: The fallback pattern

Not having to edit tiles.xml anymore is nice, but as the number of definitions grows the number of duplicated jsps with also grow. For example maybe footer and header are identical in nearly every definition.

Here we looked into writing a custom AttributeRenderer to give us such a “fallback pattern”.

In the xml the fallbacks are represented within square braces [] where each fallback option follows a pipe character “

”. And they are ordered in preference.

Now we introduce a “common” folder for jsps, and our example would turn into

//--tiles.xml
    <definition name="REGEXP:(.+)" template="/WEB-INF/tiles/template.jsp">
	    <put-attribute name="meta" value="/WEB-INF/tiles/[{1}|common]/meta.jsp"/>
	    <put-attribute name="header" value="/WEB-INF/tiles/[{1}|common]/header.jsp"/>
	    <put-attribute name="body" value="/WEB-INF/tiles/[{1}|common]/body.jsp"/>
	    <put-attribute name="footer" value="/WEB-INF/tiles/[{1}|common]/footer.jsp"/>
    </definition>

The AttributeRenderer, at least a simplified version of it, looks like

public class TemplateAttributeRenderer extends org.apache.tiles.renderer.impl.TemplateAttributeRenderer {

        private static final Pattern OPTIONAL_PATTERN
                    = Pattern.compile(Pattern.quote("[") + ".+" + Pattern.quote("]"));

        private static final String OPTION_SEPARATOR = Pattern.quote("|");

        @Override
        public void write(Object obj, Attribute attribute, TilesRequestContext request) throws IOException {

            Matcher matcher = null != obj && obj instanceof String
                    ? OPTIONAL_PATTERN.matcher((String) obj)
                    : null;

            if (null != matcher && matcher.find()) {
                PrintWriter writer = request.getPrintWriter();
                String match = matcher.group();
                String[] choices = match.substring(1, match.length() - 1).split(OPTION_SEPARATOR);
                for (int i = 0; i < choices.length && !done; ++i) {
                    String template = ((String) obj).replaceFirst(Pattern.quote(match), choices[i]);
                        try {
                            if (null != applicationContext.getResource(template)) {
                                super.write(template, attribute, request);
                            }
                        } catch (FileNotFoundException ex) {
                            if(ex.getMessage().contains(template)){
                                LOG.trace(ex.getMessage()); // expected outcome. continue loop.

                            }else{
                              throw ex; // comes from an inner templateAttribute.render(..) so throw on
                            }
                        }
                }
            } else {
                super.write(obj, attribute, request);
            }
        }
    }

To plug this AttributeRenderer make sure your TilesContainerFactory returns it

    public class FinnTilesContainerFactory extends BasicTilesContainerFactory {
        ...
        @Override
        protected AttributeRenderer createTemplateAttributeRenderer(
                BasicRendererFactory rendererFactory,
                TilesApplicationContext applicationContext,
                TilesRequestContextFactory contextFactory,
                TilesContainer container,
                AttributeEvaluatorFactory attributeEvaluatorFactory) {

            TemplateAttributeRenderer templateRenderer = new TemplateAttributeRenderer();
            templateRenderer.setApplicationContext(applicationContext);
            templateRenderer.setRequestContextFactory(contextFactory);
            templateRenderer.setAttributeEvaluatorFactory(attributeEvaluatorFactory);
            return templateRenderer;
        }
        ....
    }

As the system developers write new spring controllers creating brand new definitions the front end developers, with this new “Convention over Configuration” Tiles setup, only need to create the new folder and jsps for fragments they wish to customise. No more xml editing and no duplicate JSPs.

For a large company this could help enforce UI standards because they have control over the common/ folder - and they can keep an eye that the overrides, the customisations, never become too outlandish, or too out of line, with the standard look of the website. In turn the front end developers can also look back at the common/ folder to see what the expected default should look like.

Step 3: Definition includes

For a large website these fragments: meta, header, body, and footer; won’t be enough. A real website can quickly be using scores of fragments for columns within the body, banners and advertisements, analytics, seo meta data, css and javascript meta includes, etc, etc.

And adding another touch of reality is knowing that these different pages: cat, dog, and cow; for any real website would likely have different “actions”. For example view a dog, edit a dog, and search for a dog. Again there’ll be a lot of duplicate jsps.

By being able to dynamically inject one tiles definitions into another, a la “definition injection”, this was all elegantly handed. Here we’re stepping into the Decorator pattern’s paradigm, needing to decorate our definitions by extending Tiles’ DefinitionFactory. It would also make sense here to take advantage of Tiles’ ViewPreparer, but we weren’t so clear on the ViewPreparer implementation required, and overriding the DefintionFactory is one accepted way to configure Tiles now.

First of all we change the existing definition names into the form “action.category” making the controller(s) now look like

    @Controller
    @RequestMapping("/cat")
    public class CatController{
        @RequestMapping("/cat/view")
        public ModelAndView viewCat(){ //...
            return new ModelAndView("view.cat", modelMap);
        }
        @RequestMapping("/cat/edit")
        public ModelAndView editCat(){ //...
            return new ModelAndView("edit.cat", modelMap);
        }
        @RequestMapping("/cat/search")
        public ModelAndView searchCat(){ //...
            return new ModelAndView("search.cat", modelMap);
        }
    }
    @Controller
    @RequestMapping("/dog")
    public class DogController{
        @RequestMapping("/dog/view")
        public ModelAndView viewDog(){/ /...
            return new ModelAndView("view.dog", modelMap);
        }
        @RequestMapping("/dog/edit")
        public ModelAndView editDog(){ //...
            return new ModelAndView("edit.dog", modelMap);
        }
        @RequestMapping("/dog/search")
        public ModelAndView searchDog(){ //...
            return new ModelAndView("search.dog", modelMap);
        }
    }
    @Controller
    @RequestMapping("/cow")
    public class CowController{
        @RequestMapping("/cow/view")
        public ModelAndView viewCow(){ //...
            return new ModelAndView("view.cow", modelMap);
        }
        @RequestMapping("/cow/edit")
        public ModelAndView editCow(){ //...
            return new ModelAndView("edit.cow", modelMap);
        }
        @RequestMapping("/cow/search")
        public ModelAndView searchCow(){ //...
            return new ModelAndView("search.cow", modelMap);
        }
    }

then change the tiles.xml to introduce the definition includes

    //-- tiles.xml
    <definition name="REGEXP:([^.]+)\.([^.]+)" template="/WEB-INF/tiles/template.jsp">
        <put-attribute name="meta" value="/WEB-INF/tiles/[{2}|{1}|common]/meta.jsp"/>
        <put-attribute name="header" value="/WEB-INF/tiles/[{2}|{1}|common]/header.jsp"/>
        <put-attribute name="body" value="/WEB-INF/tiles/[{2}|{1}|common]/body.jsp"/>
        <put-attribute name="footer" value="/WEB-INF/tiles/[{2}|{1}|common]/footer.jsp"/>

        <!-- definition injection performed by DefinitionInjectingContainerFactory.instantiateDefinitionFactory(..) -->
        <put-list-attribute name="definition-injection">
            <add-attribute value=".category.{2}" type="string"/>
            <add-attribute value=".action.{1}.{2}" type="string"/>
        </put-list-attribute>
    </definition>

    //-- tiles-core.xml
    <definition name="REGEXP:\.action\.view\.([^.]+)">
        <!-- override attributes -->
        <put-attribute name="body" value="/WEB-INF/tiles/[{2}|{1}|view]/view_body.jsp"/>
        <!-- search attributes -->
        <put-attribute name="view.content" value="/WEB-INF/tiles/[{2}|{1}|view]/view_content.jsp"/>
        <put-attribute name="view.statistics" value="/WEB-INF/tiles/[{2}|{1}|view]/view_statistics.jsp"/>
    </definition>
    <definition name="REGEXP:\.action\.edit\.([^.]+)">
        <!-- override attributes -->
        <put-attribute name="body" value="/WEB-INF/tiles/[{2}|{1}|edit]/edit_body.jsp"/>
        <!-- edit attributes -->
        <put-attribute name="edit.form" value="/WEB-INF/tiles/[{2}|{1}|view]/edit_form.jsp"/>
        <put-attribute name="edit.status" value="/WEB-INF/tiles/[{2}|{1}|view]/edit_status.jsp"/>
    </definition>
    <definition name="REGEXP:\.action\.search\.([^.]+)">
        <!-- override attributes -->
        <put-attribute name="body" value="/WEB-INF/tiles/[{2}|{1}|search]/search_body.jsp"/>
        <!-- search attributes -->
        <put-attribute name="search.form" value="/WEB-INF/tiles/[{2}|{1}|search]/search_form.jsp"/>
        <put-attribute name="search.results" value="/WEB-INF/tiles/[{2}|{1}|search]/search_results.jsp"/>
    </definition>
    //-- tiles-cat.xml
    <definition name="REGEXP:\.category\.cat">
        <!-- cat attributes -->
        <put-attribute name="cat.extra_information" value="/WEB-INF/tiles/[{2}|{1}|cat]/cat_extra_information.jsp"/>
        <put-attribute name="cat.feline_attributes" value="/WEB-INF/tiles/[{2}|{1}|cat/cat_feline_attributes.jsp"/>
    </definition>
    //-- tiles-dog.xml
    <definition name="REGEXP:\.category\.dog">
        <!-- cat attributes -->
        <put-attribute name="dog.extra_information" value="/WEB-INF/tiles/[{2}|{1}|dog]/dog_extra_information.jsp"/>
        <put-attribute name="dog.k9_attributes" value="/WEB-INF/tiles/[{2}|{1}|dog/dog_k9_attributes.jsp"/>
    </definition>
    //-- tiles-cow.xml
    <definition name="REGEXP:\.category\.cow">
        <!-- cat attributes -->
        <put-attribute name="cow.extra_information" value="/WEB-INF/tiles/[{2}|{1}|cow]/cow_extra_information.jsp"/>
        <put-attribute name="cow.farm" value="/WEB-INF/tiles/[{2}|{1}|cow/cow_farm.jsp"/>
    </definition>

There’s a lot of configuration here now, but the idea isn’t to avoid configuration altogether but to avoid having to edit the configuration everytime a new jsp or page is created.

Take note there’s a clear separation now between each category definition and each action definition. For example such separation helps various development roles work in parallel: front end developers can concentrate on categorical designs while system developers often work initially with actions jsps to get them functionally working before passing them off to the front end developers. But of course such separation of concerns will show benefits in many more ways that just this example.

The following DefinitionsFactory makes this all come together

    public class FinnUnresolvingLocaleDefinitionsFactoryImpl extends UnresolvingLocaleDefinitionsFactory {

        private static final String DEF_INJECTION = "definition-injection";

        public FinnUnresolvingLocaleDefinitionsFactoryImpl() {}

        // this method can return null
        // injected definitions are expected to have "." prefix
        @Override
        public Definition getDefinition(String name, TilesRequestContext tilesContext) {
            Definition def = null;
            // WEB-INF is a pretty clear indicator it is not a definition
            if(!name.startsWith("/WEB-INF/")){
                def = super.getDefinition(name, tilesContext);
                if(null != def){
                    // use a safe copy
                    def = new Definition(def);
                    // explicit injected definitions
                    Attribute defList = def.getLocalAttribute(DEF_INJECTION);
                    if(null != defList && defList instanceof ListAttribute){
                        List list = (List) defList.getValue();
                        for(Attribute inject : list){
                            injectDefinition((String) inject.getValue(), def, tilesContext);
                        }
                    }
                }
            }
            return def;
        }

        private void injectDefinition(String fromName, Definition to, TilesRequestContext cxt) {
            assert null != fromName : "Definition not found " + fromName;
            assert fromName.startsWith(".") : "Injected definitions must have \".\" prefix: " + fromName;
            Definition from = getDefinition(fromName, cxt);
            if (null != from) {
                if (null != from.getLocalAttributeNames()) {
                    for (String attrName : from.getLocalAttributeNames()) {
                        to.putAttribute(attrName, from.getLocalAttribute(attrName), true);
                    }
                }
            }
        }
    }

To plug this DefinitionsFactory in again make sure to return it from your TilesContainerFactory

    public class FinnTilesContainerFactory extends BasicTilesContainerFactory {
        ...
        @Override
        protected UnresolvingLocaleDefinitionsFactory instantiateDefinitionsFactory(
                final TilesApplicationContext applicationContext,
                final TilesRequestContextFactory contextFactory,
                final LocaleResolver resolver) {

            return new FinnUnresolvingLocaleDefinitionsFactoryImpl();
        }
        ...
    }

When the Composite pattern is superior

It’s nonsense to think that any one pattern is better. Both the Composite pattern and the Decorator pattern have their strengths and weaknesses, even after we have progressed the Composite on to being highly dynamic and automated. They do, and achieve, quite different things and hence each should be used where applicable…

The Composite pattern presumes the page to be a “composite” made up from components, where each component is free to be itself a “composite”. The pattern allows more control of the composite’s heirarchy: as the delegation is top-down as opposed to the Decorator pattern whom’s is bottom-up. The composite pattern works well when the operations you need to perform on each object is limited, that is it isn’t a problem that the operations you may perform on any one object is the lowest common demominator of operations that you can perform on every object.

This shows how we can design to reduce complexity and encourage there to be only one unified context.

It also shows how we can maintain a top-down control of the page’s heirarchy, something necessary in a MVC design where the control layer wants to hand off a finished model map, ie a fixed context, that the view layer is free to build itself off. In contrast when we choose the Decorator pattern we can run foul of letting code run in parallel to the MVC pattern.

Conclusion

Front end developers, the system developers, and the architects need to work together. With the front end today often centered around loose coupling design: javascript’s lack of type safety, ajax’s separation from the server, and jQuery’s liberation of the dom; it is all too easy to rebel against any form of structure or organisation for fear of being tied into the formalities of the strict-typed java world.

Neither should the fear of giving up control of one’s craftsmanship mean keeping logic and control in the View layer, this stuff belongs in the Control layer: and this means the system developers need to work harder to get the front end developers collaborating in the control layer. What’s too easily forgotten is that everything has entropy, and the larger the organisation, the larger the codebase, the less you fight the entropy, the quicker you end up with just a pile of spaghetti in your lap.

The four steps gone through in this article show you how Tiles-2 and the Composite pattern can be ramped up on steroids to give front end and systems developers what they want hopefully in a way that encourages them to work together.

References

Tiles-2
Composite pattern
Decorator pattern
Spring Web
Integrating Tiles and Spring
Initial discussion regarding Tiles definition delegation (URL not available)
Spring >3.0.1 required to work with Tiles-2.2+ :: TilesConfigurer does not … SPR-6097

Design template

2010-01-01T13:37:00+00:00

Headings

The largest heading (an <h1> tag)

The second largest heading (an <h2> tag)

The 3rd largest heading (an <h3> tag)

The 4th largest heading (an <h4> tag)

The 5th largest heading (an <h5> tag)

The 6th largest heading (an <h6> tag)

Paragraph

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam mattis odio id quam semper, eu scelerisque quam condimentum. Curabitur eu arcu feugiat, vehicula mi ac, mollis lacus. In eu fringilla erat. Vestibulum purus enim, ornare elementum sollicitudin vel, lacinia eu neque. Sed dolor ex, accumsan a egestas et, facilisis et mi. Cras ullamcorper blandit lacus, a posuere arcu tristique ut. Curabitur feugiat placerat aliquam. Fusce sit amet nisl a lectus pellentesque faucibus ut ac massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec augue urna, lacinia sed maximus pulvinar, tempus eu massa. Vestibulum malesuada nunc ut tortor congue finibus ut vel urna. Phasellus quis arcu quis purus feugiat pulvinar condimentum eget felis. Donec tincidunt euismod auctor. Phasellus congue elementum faucibus. In ex velit, elementum et elementum et, accumsan hendrerit felis. Fusce lobortis bibendum leo eget condimentum.

Nunc scelerisque, sem quis rutrum scelerisque, lorem mauris scelerisque elit, ut tempus diam tellus at lectus. Donec nec hendrerit sem. Suspendisse eleifend iaculis lorem, nec ultrices nibh convallis sit amet. Ut risus erat, vestibulum vel placerat in, pulvinar nec ex. Ut nec cursus quam, eget pulvinar odio. Interdum et malesuada fames ac ante ipsum primis in faucibus. Pellentesque pretium mauris at ornare accumsan. Donec vel commodo magna. Mauris sed augue tortor. Etiam eget sem quis libero feugiat malesuada. Integer tempor, ligula vel vestibulum sagittis, risus ipsum tempus ipsum, ut maximus eros erat ut augue. Mauris rutrum aliquet est dictum efficitur. Vivamus vestibulum orci massa, ac dignissim mauris lacinia nec. Aliquam volutpat nisl nec ipsum lobortis pellentesque facilisis convallis nulla.

Vivamus sed interdum nibh, sit amet mollis lorem. Sed ultrices, magna id convallis consectetur, justo nunc commodo justo, at dapibus purus nibh sed mauris. Phasellus vulputate interdum aliquet. Donec a nibh id massa placerat ullamcorper. Vestibulum facilisis leo nulla, vitae porttitor lectus facilisis ac. Nulla facilisi. Aenean sagittis lorem id ante elementum dignissim. Vestibulum eu libero mattis, blandit risus eu, consectetur dolor. Aenean tincidunt nisi lacus, nec sodales est sodales at. Ut efficitur dignissim mauris a ullamcorper. Maecenas eget mi quis diam consequat placerat. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Aenean sed leo at risus feugiat mattis. Integer eget porta leo, at accumsan metus. Aliquam dictum metus sit amet diam tristique pellentesque. Sed facilisis viverra nisi, eget mattis nunc commodo a.

Blockquotes

In the words of Abraham Lincoln:

Pardon my french

Styling text

This text will be italic

This text will be bold

Lists

Unordered lists

Item
Item
Item
Item
Item
Item

Orderd lists

Item 1
Item 2
Item 3

Nested lists

Item 1
1. A corollary to the above item.
2. Yet another point to consider.
Item 2
- A corollary that does not need to be ordered.
- This is indented four spaces, because it’s two spaces further than the item above.
- You might want to consider making a new list.
Item 3

Links

Visit Finn!

HR

Images

Video

Embed:

This is a cool tweet:

Practical library for doing feature toggles in Java/JVM: https://t.co/XWX2CGxubE by @FINN_tech (via @rubendw)
— Thomas F. Nicolaisen (@tfnico) December 3, 2014

Tables

Tables	Are	Cool
col 3 is	right-aligned	Yey
col 2 is	centered	Yow
zebra stripes	are neat	Yaw

Markdown	Less	Pretty
Still	`renders`	nicely
1	2	3

Long list

Domain	Number registered	Percent of total
hotmail.com	1106084	28.06%
gmail.com	599293	15.20%
online.no	444897	11.28%
live.no	96052	2.44%
yahoo.no	94029	2.39%
yahoo.com	75772	1.92%
hotmail.no	67602	1.71%
c2i.net	67080	1.70%
broadpark.no	48232	1.22%

Code

var s = "JavaScript syntax highlighting";
alert(s);

s = "Python syntax highlighting"
print s

Code block with line numbers

1
2
3
yaml: true
linenos: true
But let's throw in a <b>tag</b>.

require 'redcarpet'
markdown = Redcarpet.new("Hello World!")
puts markdown.to_html

<!-- core libraries -->  
org.apache.logging.log4j:log4j-api:2.0.2  
org.apache.logging.log4j:log4j-core:2.0.2  
com.lmax:disruptor:3.2.1  

<!-- commons-logging -->  
org.apache.logging.log4j:log4j-jcl:2.0.2  
org.apache.logging.log4j:log4j-1.2-api:2.0.2  

<!-- slf4j -->  
org.slf4j:slf4j-api:1.7.7  

<!-- JUL routed through slf4j -->  
org.apache.logging.log4j:log4j-slf4j-impl:2.0.2  

<!-- override old log4j with an empty jarfile -->  
<!-- incase it gets re-introduced transitively -->  
log4j:log4j:2-empty  

<!-- logstash -->  
net.logstash.log4j2:log4j2-logstash-jsonevent-layout:3.0.0-finn-2

Emoji

:+1:

Mentions

@Andersos

FINN Technology

Haskell at FINN.no

Favorites

Haskell

Experiences

Check it out

Preparing your product for machine learning

What kind of data does a machine learning algorithm like?

Is the data numerical?

Is the data categorical?

UI - Making categorical/numerical inputs work

Can we process the data to make it numerical?

Good to go?

A few caveats

In conclusion

5 learnings from Hack Days in FINN

Personalized search with a custom Solr plugin

Background

A new solution

Solr SearchComponent

Requirements

SearchComponent.prepare

SearchComponent.process

Response times

Search examples

Product results

Summary

Åpen fagkveld hjemme hos FINN

Presentasjoner

Åpen fagkveld 28. februar kl 16-20 (for de som jobber med teknologi)

Åpen studentkveld 7. mars kl 16-20 (for studenter)

Browser statistics December 2017

Browser statistics, December 2017 at FINN.no#

How many visitors use a desktop or laptop?

Browsers and devices

Operating systems

FINN wins Universal Design Award

House and Work for Everyone

Increases Competitiveness

Far Ahead of the Competition

Other blog posts on accessibility in FINN.no:

Akka workshop at Finn.no

Deep NLP-based Recommenders at Finn.no

Using the model in recommendations

Further work

The summer interns of 2017

The starting point

Project 1: Tilbakemelding (Feedback)

Technical solution

Results UX Design

Recommendations

Project 2: Redesign ad dashboard

Technical solution

Results UX design

Recommendations

Final thoughts

From Hadoop and Cassandra to Kafka Streams

Some context

Existing solution

Replacement solution

Streams in use

Requirements

Architecture

Data quality

Avro and Schema Registry

Streams filtering

Dealing with duplicates

Limiting the disk usage

Migrating existing data

Integration testing

Reprocessing

Error diagnostic

Summary

Flappy Cat

Why make a Flappy Cat game?

How we made it

WAI-ARIA and its true impact on assistive technologies

1. Introduction

1.1. Generalize screen readers

1.2. The main issues with ARIA