FINN tech blog

tech blog

Archive: ‘Interface development’

We love NPM!


What? Why is FINN.no donating to scale NPM? I thought you guys were a pure Java shop? It is true, we used to be a pure Java-shop. However over the past three years we have adopted new technologies to solve specific problems. We have used Ruby and Cucumber for some time to make a platform for continuos delivery and it has worked out beautifully! Our front-end developers have been forced to deal with out dated and not suitable tools for doing their job. This is largely due to the fact that all innovation when it comes to front-end development does not happen in the Java community. Most of the exciting tools are written in Node and this has become a frustration and a challenge for us.

In the past year FINN have been gradually making a transition away from using only Java-based tools for front-end development and towards a NodeJS powered tool set. We are now at a point were we are on the brink of rolling this out for our projects. Having worked with Node for a while we have learned to appreciate the Node ecosystem which is NPM. Being part of such a vibrant ecosystem of modules makes the transition easier and it also inspires us to become better at giving back. Therefore we are trying to give back to NPM, when we can.

When the scale NPM campaign was launched it was obvious that this was something we wanted to be apart of. It is an investment in our own happiness in a sense, as NPM is becoming a very important part for our technology portfolio.

Nodeify all the things

So were is it that we use Node in our technology stack today? Earlier this year we moved away from JsTestDriver in favor of Karma-runner. This meant that we needed to create a trojan horse containing the goodness of Node/NPM into existing Java projects without causing too many problems for developers with no knowledge of Node. A part of this scheme was the frontend-maven-plugin, which enables us to have control of which Node projects use and allows developers without Node previously installed to build projects and run tests without having to learn anything about Node.

Currently we are working towards removing the need for using Maven to build and deploy pure JavaScript projects using Node. The end result is of course to have more web applications built using Node. Today we have just two which are running in a production environment.

Low-fi Coffe Surveilence

There is a Norwegian start-up called AppearIn which enables anyone to do online video conferencing with only a web browser. They utilize WebRTC to empower users to create rooms where they can hang out, do meetings, etc. We had the pleasure of having Ingrid form Appear In over for a visit and she suggested we’d contribute to their blog about use cases for appear in. That’s how we became use case #7 on their blog. Naturally this should’ve been an arduino powered thing with lot’s of sophisticated technology, but we decided we’d just get something simple working first and then make it better later on.

This is a really exciting platform as it empowers developers to have video/audio communication with little effort. Dealing with WebRTC, which is still a moving specification, can be quite challenging to implement properly. There’s numerous corner cases and differences between implementations.

Update: Through the wonders of the internet, it turns out that we have brought the webcam thing full circle. A user on Hacker News posted a comment which linked to a wikiepedia article about the first webcam. It turns out the use case for it was exactly the same as we had.

the ultimate view — Tiles-3

A story of getting the View layer up and running quickly in Spring…

Since the original article, parts of the code has been accepted upstream, now available as part of the Tiles-3 release, so the article has been updated — it’s all even simpler!


Based upon the Composite pattern and Convention over Configuration we’ll pump steroids into
   a web application’s view layer
      with four simple steps using Spring and Tiles-3
         to make organising large complex websites elegant with minimal of xml editing.



 




Background

At FINN.no we were redesigning our control and view layers. The architectural team had decided on Spring-Web as a framework for the control layer due to its flexibility and for providing us a simpler migration path. For the front end we were a little unclear. In a department of ~60 developers we knew that the popular vote would lead us towards SiteMesh. And we knew why – for practical purposes sitemesh gives the front end developer more flexibility and definitely less xml editing.
But sitemesh has some serious shortcomings…

SiteMesh shortcomings:
  • from a design perspective the Decorator pattern can undermine the seperation MVC intends,
  • requires all possible html for a request in buffer requiring large amounts of memory
  • unable to flush the response before the response is complete,
  • requires more overall processing due to all the potentially included fragments,
  • does not guaranteed thread safety, and
  • does not provide any structure or organisation amongst jsps, making refactorings and other tricks awkward.

One of the alternatives we looked at was Apache Tiles. It follows the Composite Pattern, but within that allows one to take advantage of the Decorator pattern using a ViewPreparer. This meant it provided by default what we considered a superior design but if necessary could also do what SiteMesh was good at. It already had integration with Spring, and the way it worked it meant that once the Spring-Web controller code was executed, the Spring’s view resolver would pass the model onto Tiles letting it do the rest. This gave us a clear MVC separation and an encapsulation ensuring single thread safety within the view domain.

“Tiles has been indeed the most undervalued project in past decade. It was the most useful part of struts, but when the focus shifted away from struts, tiles was forgotten. Since then struts as been outpaced by spring and JSF, however tiles is still the easiest and most elegant way to organize a complex web site, and it works not only with struts, but with every current MVC technology.” – Nicolas Le Bas

Yet the best Tiles was going to give wasn’t realised until we started experimenting a little more…

Going all out with the Strap-on Project

In January we started on an ambitious project called “the Strap-on project”. The infantile name aside this project is about rewriting our entire front-end tier using Object Oriented CSS (OOCSS), written by Nicole Sulivan, as the secret sauce to achieving the desired results.

Why OOCSS?

At FINN we have teams grouped by business units and all of them have developers who contribute with code on our site. This is provided us with a head ache with CSS, because there are no clear idioms for how to best write CSS. In our case this had cause teams to create their own “islands” of CSS by using # trails, train wreck selector classes and !important-wars. Teams had duplicate CSS for the same layout and we had a hard time keeping the user experience consistent across different parts of our service.

We had way too much CSS code in total and we sent loads of CSS on each page request with just a small percentage actually being used. This made page load slow and rendering slow. Page rendering speed is equal to money, so we had to change something.

OOCSS – making CSS coding a thing of the past

OOCSS is a framework on which to build your own. It provides with a base set of modules and concepts which is essentials to building stuff in HTML with CSS. Grids, modules, lines, etc. These base modules provides an abstraction on top of CSS which makes authoring of CSS a thing of the past. In order to layout basic pages all we do is to use the building blocks and put together what ever you want. Basic boring stuff such as clearing and the box-model is no longer anything you need to worry about, it’s taken care of. This enables developers to focus upon fulfilling business requirements instead of battling CSS differences in browser time and time again.

Why not LESS, Compass or stuff like that?

The CSS language abstractions that exist out there are all very cool projects and make a whole lot of sense to use for a lot of projects. However, in our case choosing one of these tools right now would not provide us with some of the benefits we are looking for.

At FINN we have a problem that we are duplicating CSS across our development teams. This makes creating a consistent user experience and making global changes harder than it should be. Using an abstraction would help us hide some these problems, but would not solve the underlying issue which is that we create too much code. OOCSS and its rigorous set of rules helps reduce the amount of code drastically and provides rigid rules which prevents duplication across teams. What we are looking for are:

  • Improve rendering speed
  • Improve development speed
  • Easier to provide a consistent user experience

A touch of Bootstrap too

The engineers over at Twitter has created an amazing framework, Bootstrap, which is hugely popular all over the world. We have indeed paid close attention to how they have done and we are very much inspired by their work. However, going all out and just adopting Bootstrap was not an option. We feel it is too bloated and it does not provide the speed and performance benefits OOCSS gives.
Having said that, a lot of our setup with forms and form elements is influenced by how Bootstrap does things.

Half way, how does it look?

In short,iIt looks pretty darn impressive! We have removed more than twenty CSS files and reduced the amount of CSS code lines with more than thirty thousand (this does say quite a bit of the mess we where in, I know). We have migrated the motorized vehicles, jobs, real estate sections and the front page.

  Before After
Number of CSS files 130 38
Lines of CSS code 32 789 2 927
Lines of section specific CSS 727 89

These are pretty impressive results and very much in line with what Nicole is talking about in her presentations about OOCSS and performance.

is it all singing, all dancing?

Of course not! The responsive bit is something we are looking at reworking. Team Oppdrag has created one way of solving this and Team Reise another. For the Strap-on project we will probably end up with something in between. There are some things you need to take into consideration when choosing an approach:
How much control do you have over the markup being written? Or to put it in another way, how many people are working with the same code? The more people, the harder it is to apply very strict rules as people will be “tourists” in the code base and might have a hard time getting it.

Another big challenge is to provide enough support and training to help everyone utilize the OOCSS framework so we can continue to reap benefits from what we have done, but that is probably some other blog post

Profiling and debugging view templates



Ever needed to profile the tree of JSPs rendered server-side?
  Most companies do and I’ve seen elaborate and rather convoluted ways to do so.

With Tiles-3 you can use the PublisherRenderer to profile and debug not just the tree of JSPs but the full tree of all and any view templates rendered whether they be JSP, velocity, freemarker, or mustache.

At FINN all web pages print such a tree at the bottom of the page. This helps us see what templates were involved in the rendering of that page, and which templates are slow to render.



We also embed into the html source wrapping comments like

<!-- start: frontpage_geoUserData.jsp -->
...template output...
<!-- end: frontpage_geoUserData.jsp :it took: 2ms-->


The code please

To do this register and then attach your own listener to the PublisherRenderer. For example in your TilesContainerFactory (the class you extend to setup and configure Tiles) add to the methd createTemplateAttributeRenderer something like:

    @Override
    protected Renderer createTemplateAttributeRenderer(BasicRendererFactory rendererFactory, ApplicationContext applicationContext, TilesContainer container, AttributeEvaluatorFactory attributeEvaluatorFactory) {
 
        Renderer renderer = super.createTemplateAttributeRenderer(rendererFactory, applicationContext, container, attributeEvaluatorFactory);
        PublisherRenderer publisherRenderer = new PublisherRenderer(renderer);
        publisherRenderer.addListener(new MyListener());
        return publisherRenderer;
    }

Then implement your own listener, this implementation does just the wrapping comments with profiling information…

class MyListener implements PublisherRenderer.RendererListener {
 
    @Override
    public void start(String template, Request request) throws IOException {
        boolean first = null == request.getContext("request").get("started");
        if (!first) {
            // first check avoids writing before a template's doctype tag
            request.getPrintWriter().println("\n<!-- start: " + template + " -->");
            startStopWatch(request);
        } else {
            request.getContext("request").put("started", Boolean.TRUE);
        }
    }
 
    @Override
    public void end(String template, Request request) throws IOException {
        Long time = stopStopWatch(request);
        if(null != time){
            request.getPrintWriter().println("\n<!-- end: " + template 
                                         + " :it took: " + time + "ms -->");
        }
    }
 
    private void startStopWatch(Request request){
        Deque<StopWatch> stack = request.getContext("request").get("stack");
        if (null == stack) {
            stack = new ArrayDeque<StopWatch>();
            request.getContext("request").put("stack", stack);
        }
        StopWatch watch = new StopWatch();
        stack.push(watch);
        watch.start();
    }
 
    private Long stopStopWatch(Request request){
        Deque<StopWatch> stack = request.getContext("request").get("stack");
        return 0 < stack.size() ? stack.pop().getTime() : null;
    }

It’s quick to see the possibilities for simple and complex profiling open up here as well as being agnostic to the language of each particular template used. Learn more about Tiles-3 here.

Putting a mustache on Tiles-3

We’re proud to see a contribution from one of our developers end up in the Tiles-3 release!

The front-end architecture of FINN.no is evolving to be a lot more advanced and a lot more work is being done by client-side scripts. In order to maintain first time rendering speeds and to prevent duplicating template-code we needed something which allowed us to reuse templates both client- and server-side. This is where mustache templates have come into play. We could’ve gone ahead and done a large template framework review, like others have done, but we instead opted to just solve the problem with the technology we already had.

Morten Lied Johansen’s contribution allows Tiles-3 to render mustache templates. Existing jsp templates can be rewritten into mustache without having to touch surrounding templates or code!

The code please

To get Tiles-3 to do this include the tiles-request-mustache library and configure your TilesContainerFactory like

    protected void registerAttributeRenderers(...) {
        MustacheRenderer mustacheRenderer = new MustacheRenderer();
        mustacheRenderer.setAcceptPattern(Pattern.compile(".*.mustache"));
        rendererFactory.registerRenderer("mustache", mustacheRenderer);
        ...
    }
    protected Renderer createTemplateAttributeRenderer(...) {
        final ChainedDelegateRenderer chainedRenderer = new ChainedDelegateRenderer();
        chainedRenderer.addAttributeRenderer(rendererFactory.getRenderer("mustache"));
        ...
    }

then you’re free to replace existing tiles attributes like

<put-attribute name="my_template" value="/WEB-INF/my_template.jsp"/>

with stuff like

<put-attribute name="my_template" value="/my_template.mustache"/>

Good stuff FINN!

Make real estate brokers happy with hard boiled business cards

FINN.no has recently released new pages for real estate ads. Larger images, broker business cards and tabs for additional information are some of the new features on this page. A long process of in-depth users interviews, workshops with real estate brokers, user testing of design sketches and several sprints of development, we are proud to finally see the new pages on www.finn.no.

Pick a nice home from this list to view the ads
Cut the general crap – show me the business cards!

What’s in it for the user?

Larger images and the possibility to navigate through images without leaving the page were the first thing that was noticed, and liked, during our user testing sessions. For a full size view of 900 pixel images we have still kept a link to our old image viewing page. However, after release we have got feedback from many users who failed to find this link and complain that images are smaller. There is still work to be done to improve this.

The information about the home is divided into primary and secondary information. The prime information, like price, area, address and viewings, is put at the top of the page, so the user won’t need to scroll to find it. More detailed information, like facilities, plot size and descriptions is located below the prime information and the image viewing module.

Secondary information is integrated by tabs. Fakta (facts) is the default tab, containing information about the estate from the broker. Prisstatistikk (price statistics) is highlights from Eiendomspulsen, which displays information like average price per square meter in the area and prices for previous sales of the actual estate. Neighborhood profile is an additional service brokers may add to their prospects with information about nearby services, schools and demographics.

What’s in it for the broker?

Realestate brokers have requested more options for building brand awareness in the prospect. To meet this request our designers suggested the brokers may design their own business cards for their contact information and to use their profile colors at some other elements on the page. They may also offer their services after the last image in the image gallery and in a tab next to Price statistics and Neighborhood profile. Interviews with users indicate that information from the broker won’t be regarded as ordinary banner ads as long as they provide information that is relevant for users who are in the process of changing their home. Krogsveen (a major realestate agent in Norway) has taken good advantage of this banner position by listing properties they have recently sold in the same area.

How to make the business cards

There are three ways to display the broker’s contact information. Basic prospects have a simple listing of name, position, phone and fax numbers along with a company logo. Brokers who are using expanded prospects, which are prospects with the broker’s own set of banners, have their contact information styled as a business card. Brokers may customize these cards with their own css, background image and a picture of the broker.

Some prospects have two brokers listed, but to save space only one broker is visible at the time. In expanded prospects business cards will swap when the user click a looping arrow or the card in the back. In basic prospects the name of the other broker slide in when clicking the “more contact persons” link. Jquery animation is used to swap the cards and CSS transitions is used for the sliding of contacts in basic prospects.

The markup is the same for all levels of contact information, supporting the hcard microformat, while there are three layers of css. Styling of the contact information on basic prospects is in the main css file. Default business cards are styled by using a seperate css file which adds the business card look and overrides the default css where necessary. Customized business cards are styled by a third layer of css, which overrides the classes from the default card to add a background image and their company’s colors and fonts.

By the time of release we had 13 different customized business cards on our prospects.

Contact information for a basic prospect

HTML code for a business card looks like this:

<address id="flip_0" class="vcard flip  backgroundimage">
  <a class="orglogo" target="_blank" href="(url)"><img border="0" alt="Utleiemegleren Sinsen" src="(url)" class="logo"></a>
  <span class="fn n brokerName">Karianne Johannessen</span>
  <span class="ul">
    <span class="li title brokerTitle">Supermegler</span>
    <span class="li brokerPhoneHolder">
      <dl class="tel brokerPhone">
        <dt class="type hideText">Work</dt>
        <dt class="displayText">Telefon</dt>
        <dd class="value">22 79 66 26</dd>
      </dl>
    </span>
    <span class="li brokerMobilHolder">
      <dl class="tel brokerMobil">
        <dt class="type hideText">Cell</dt>
        <dt class="displayText">Mobil</dt>
        <dd class="value">48133956</dd>
      </dl>
    </span>
    <span class="li brokerFaxHolder">
      <dl class="tel brokerFax">
        <dt class="type hideText">Fax</dt>
        <dt class="type">Faks</dt>
        <dd class="value">22796601</dd>
      </dl>
    </span>
    <span class="li brokerEmail">
      <a class="click-track" data-click-mt="mailto" href="(url)">Send epost</a>
    </span>
  </span>
  <div class="clearall"></div>
</address>

This is the basic css for contact information:

.contact {overflow: hidden; width: 330px;}
.contact .cardWrapper {position: relative; width: 700px;}
.contact address {-moz-transition: left 0.7s ease-in-out;-webkit-transition: left 0.7s ease-in-out; -o-transition: left 0.7s ease-in-out; -ms-transition: left 0.7s ease-in-out; transition: left 0.7s ease-in-out;
    background-color:#FFFFFF; display:block; float:left; font-style:normal; position:relative; width:330px;}
.contact .hideText {display: none;}
.contact .orglogo {float: right;}
.contact .brokerName {display:block; font-size:12px; font-weight:normal; margin:0;}
.contact .li {clear: left; display: block; margin: 0; padding: 0;}
.contact dt {float: left; padding-right: 0.5em;}
.contact dd {float: left;}
.contact .moreContacts {float:left; width:330px; text-align:right; margin-bottom:20px; }
.contact address {left:0;}
.contact address.backCard {position:relative; left:360px;}
.contact address +  address {left:-330px;}
.contact address + address.backCard {left:30px}

Default styling of a business card

This is the second layer of css which forms a basic business card:

#flip_0{z-index:1; top:0; left:0;}
#flip_1{z-index:0; top:20px; left:15px;}
.contact .flip.backCard {cursor:pointer; background:#f3f3f3; display:block; transform: rotate(3deg);-o-transform: rotate(3deg);
    -ms-transform: rotate(3deg); -moz-transform: rotate(3deg); -webkit-transform: rotate(3deg);}
.contact .flip.backCard:hover {background:#ffffff;}
.contact .cardWrapper {position:relative; min-height:195px; width:auto;}
.contact .spacer {height:25px; display:block; clear:both;}
.contact .flip{position:absolute; background-color:white;}
.contact address {box-shadow: 2px 2px 8px 0 #cccccc; padding:10px; width:300px; height:150px; border: #eeeeee solid 1px; float:none; overflow:hidden;
    -moz-transition: none;-webkit-transition: none;-o-transition: none;transition: none;-ms-transition: none;}
.contact address .brokerPhoto {float:left; max-height:100px; max-width:75px; margin: 15px 10px 0 0; display:block;}
.contact .brokerName {font-size:16px; font-weight:normal; margin-top:15px}
.contact img.brokerPhoto~ul {max-width:190px}
.contact .orglogo img {position:absolute; bottom:10px; right:10px;max-height:100px; max-width:100px; }
.contact .moreContacts {display:none;}
.contact img.moreContacts  {float:right; position:absolute; top:0; right:-12px; display:block; cursor:pointer; width:14px; height:22px}
.contact .ul {float: left; display:block;}

Customized business card

This third layer of css forms the broker’s  customized business card:

.contact * {color:#46464a; font-size:11px;}
.contact .orglogo {display:none;}
.contact .brokerPhoto {margin:11px 20px 0 11px !important;}
.contact .brokerName {font-weight:bold; font-family:"Lucida Sans","Lucida Sans Unicode","Lucida Grande",sans-serif; margin:8px 10px 0 0; font-size:12px;}
.contact .brokerTitle {margin:0 0 5px 0;}
.contact .brokerPhone dt {font-weight:bold;}
.contact .brokerMobil dt {font-weight:bold;}
.contact .brokerFax dt {font-weight:bold;}
.contact .brokerEmail a {padding-top:5px; display:inline-block;}
.contact .brokerEmail a, .contact .homePage a {color:#46464a;}
.contact .homePage {margin-top:0;}
.contact .backgroundLogoLink {position:absolute; left:0; bottom:0; width:320px; height:25px; display:block;}

Read more about hard boiled business cards and the hcard microformat
Dig deeply into hardboiled web design

The feedback

Three weeks after the release we have recieved 1060 feedbacks, 630 positive, 374 negative and 56 indifferent. This is actually better feedback than we are used to after releasing such a radical redesign. Our biggest challenge is to make it easier to view images in full size. For other layout issues it seems like people simply need some time to get used to it. I have seen no complaints from users who dislikes the business cards, which was one of our biggest concerns before the release.

Our analyzing tools tell us that the average viewing time on this page before the release was 33 seconds. After the release of the new design the viewing time has increased to 1 minute and 15 seconds, at the expense of the image viewing page, which page views has dropped rapidly. The number of page views of the ad page has also dropped, probably because of less navigation to and from the image viewing page. However, the number of visits on the ad page seems to be stable.

It has never been about the standers, while it has always been about the standards

Web Standards tag cloud

I was introduced to the World Wide Web back in 1995 while being a student at Molde College, Norway. What fascinated me the most was that I could learn how to create stuff for the web by just viewing the source code of other web sites. This blew my mind and since then I have built a career working with web development.

Creating web sites and web-based applications for me has always been about trying to adopt the latest technological advances in browsers, as soon as they were available. Naturally, this has lead to me reaching a few dead ends (IE’s Data Binding and Netscape JavaScript style sheets), but mostly it has enabled me to create some pretty cool things.

From DHTML to an atomic winter

In the days of the .com era, the community for creating web based applications was buzzing. We were cranking out all kinds of DHTML magic. Every site out there was jumping on the band wagon without any hesitation. One of the classics is online retailer Boo.com which went all out with new technology without really succeeding. Naturally we had to deal with older browsers then too, but we did not let that get in the way of using the latest standards being adopted in browsers.

The burst of the .com bubble created something of an atomic winter as far as web based user interface development goes. A lot of people lost their jobs and the rest of us took refuge in server side development.

A clean slate and a new dawn

Thankfully all the bad stuff was swiftly cleaned up with the introduction of the AJAX approach by Jesse James Garret in 2005. All the interface developers came out from their hiding places and they sure had been busy during that atomic winter. Out popped all kinds of amazing things like Dojo Toolkit, Yahoo! UI Library and JQuery. Once more the web moved forward at a rapid pace thanks to the newly found fame due to the AJAX hype. JavaScript was getting recognition as an actual language and CSS was being implemented in a more serious manner than before.

We all implemented our applications and sites using HTML, JavaScript and CSS2 and felt good about it. However it is worth noting that CSS2 was not a finished specification and no browser has yet to implement the entire CSS2 specification. Still, we have used loads of what is in CSS2 for years.

HTML5 – a new generation of web developers

Lulling in the wake of AJAX a new generation of web developers entered the scene. The WHATWG decided to take matters into their own hands and do what the W3C seemed incapable of, creating something which made creating web based applications with HTML easier. With the new hype of HTML5 we started to get all kinds of, what I thought, strange questions on blogs, Twitter and in industry media. “When can I start using HTML5?”, “When will it be ready?”, “I am so disappointed, all this hype and I can not use it”.

All these questions struck me as very odd and I was beginning to think there was something wrong with me, had I missed something? I had never heard these kinds of questions before regarding CSS, JavaScript or HTML. Sure, we had to take into consideration that not all browsers supported everything in the same manner. But that was what we as web developer did, right? We create web based applications using the best technology available to us meanwhile making sure older less capable browsers could view it to. This is what got my career started and what I have spent a lot of time doing as a living.

This new generation of developers seemed to be very focused upon only applying technology which where in standards that where finished. To me this sounded weird as we have never once before had any hesitation as to adopt new technology when it was available to us. When the XmlHttpRequest object was missing in browsers, what did we do? We used the old trick of the hidden IFrame. Back when Opera’s JavaScript support was horrendous (luckily this is no longer the case) what did we do? We made sure our stuff worked without the use of JavaScript. If something was not a final specification or something was not implemented in all browsers we implemented fallback solutions or provided alternate experiences.

Our bread and butter

Creating web applications using the latest technologies available to us should be our bread and butter as web developers. We should not wait around for the W3C to put some stamp on some documents. Browser vendors drive innovation, not standards bodies and therefor we should apply the technology being made available to us as quickly as we can, while making sure those with less capable browsers also can access our service or site. HTML5 does not make this any harder than before and we should not use the lack of a finalized HTML5 specification hold us back.

Web Standards are everything

You might be tempted into thinking that web standards are not that important, after all the browser vendors drive innovation. Innovation is done by the vendors, but the standards are extremely important as a way of making sure the web stays open and does not become fragmented. There are always numerous organizations trying to create their own special web, but thanks to the great work of the standard bodies the web has stayed open and will continue to do so.

The web standards (such as The Web Standards Project and WHATWG) community has really blossomed in the years after the introduction of AJAX and developers are way more conscious about the importance open standards. Without the work of the people in various working groups and standard bodies we would not be where we are today. Web technology is being used in more innovative ways than ever before. Not only restricted to making content or applications for the web, but also on the server side or to create native mobile applications.

In summary the web standards are not important, but still they are essential for us to continue to enjoy all the amazing service and applications created with open web technology.

This article was published in the August issue of the NDC Magazine from Programutvikling AS

Graded browser support version 1.1

We have reworked our graded browser support matrix which was released last year. Following the Laws of Simplicity we reduced the complexity in order to make it easier for everyone at our company to understand. We cut a lot of the version number into targeting latest for browsers which are frequently updated. There is also a support matrix for mobile phones which is targeting our new mobile web service m.finn.no.

In addition to updating the versions we have also made the decision to include tablets in the support matrix for our desktop verison of our service. This is because we are currently not looking at creating specialized versions for tablets and our service works just fine on the tablets currently hitting our site. Currently almost all of our trafic is coming from iPad’s, therefor Andoid powered devices are not supported.

XSS protection: who’s responsibility?

In a multi-tier application who can be responsible for XSS protection?
Must security belong to a dedicated team…or can it be a shared responsibility?
Today XSS protection is typically tackled by front end developers.
 Let’s challenge the status quo.  


New Applications vs Legacy Applications

For protection against Stored XSS many applications have the luxury of ensuring any text input from the user, or from a CMS system, is made clean and safe before being written to database. Given a clean database the only XSS protection required is around request values, for example values from url parameters, cookies, and form data.


Image by The Planet via Flickr -- http://www.flickr.com/photos/26388913@N05/4879419700

But some applications, especially legacy applications, are in a different boat. Databases typically have lots of existing data in many different formats and tables so it’s often no longer feasible to focus on protecting data on its way into the system. In this situation it is the front end developers that pay the price for the poor quality of backend data and are are left to protect everything. This often results in a napalm-the-whole-forest style of xss protection where every single variable written out in the front end templates goes through some equivalent of

            <c:out value="${someText}"/>

This makes sense but…   if you don’t have control is your only option to be so paranoid?

A Messed up World

To illustrate the problem let’s create a simple example by defining the following service

  interface AdvertisementService{
    Advertisement getAdvertisement(long id);
  }
 
  interface Advertisement{
    /** returns plain text title */
    String getTitle();
    /** return description, which may contain html if isHtmlEnabled() returns true */
    String getDescription();
    /** indicates the description is html */
    boolean isDescriptionHtml();
  }

The web application, having already fetched an advertisement in the control tier, somewhere would have a view template looking something like

    <div>
        <h1><c:out value="${advertisement.title}"/></h1>
        <p>
            <c:out value="${advertisement.description}" 
                      escapeXml="${!advertisement.descriptionHtml}"/>
        </p>
    </div>
Here we add another dimension: simple escaping with c:out won’t work if you actually want to write html (and what is the safety and quality of such html data).

When this service is used by different applications each with their own view templates, and maybe also exposed through web services, you end up no doubt protecting it over many times, system developers in the web services, and front end developers in each of the presentation layers… likely there will be confusion over the safety and quality of data coming from this service, and of course everyone will be doing it differently so nothing will be consistent.

  • Is there a better way?
  • Can we achieve a consistent XSS protection in such an environment?
  • How many developers need to know about XSS and about the origin of the data being served?

In the above code if we can guarantee that the service _always_ returns safe data then we can simplify it by removing the isDecriptionHtml() method. The same code and view template would become

  interface AdvertisementService{
    Advertisement getAdvertisement(long id);
  }
 
  /** All fields xss protected */
  interface Advertisement{
    /** returns plain text title */
    String getTitle();
    /** return description, which may contain safe html */
    String getDescription();
  }
    <div>
        <h1>${advertisement.title}</h1>
        <p>${advertisement.description}</p>
    </div>

By introducing one constraint: that all data is xss protected in the services tier; we have provided a simpler and more solid service API, and allowed all applications to have simpler, more concise, more readable, view templates.

Solutions

Having a clean database all non-html data can be escaped as it comes in. Take advantage of Apache’s StringEscapeUtils.escapeHtml(..) from commons-lang library. For incoming html one can take advantage of a rich html manipulation tool, eg like JSoap, to clean, normalise, and standardise it.

With legacy or foreign data, especially those applications with exposed service architecture and/or multiple front ends, a different approach is best: ensure nothing unsafe ever comes out of the services tier. For the html data the services will often be filtering many snippets of html over and over again, so this needs to be fast and a heavy html manipulation library like JSoap isn’t appropriate any more.
A suitable library is xss-html-filter, a port from libfilter. It is fast and has an easy API

String safeHtml = new HTMLInputFilter().filter( "some html" );

If we do this it means
   xss protection is not duplicated, but rather made clear who is responsible for what,
   c:out becomes just junk in verbosity and performance* ,
   service APIs become simpler,
   view templates look the same for both new and legacy applications,
   system developers become responsible for protection of database data and this creates a natural incentive for them to clean up existing data and ensure new data comes in safe,

* No matter what an inescapable fact is all front ends developers must have a concrete understanding that any value fresh from the request is prone to Reflected XSS and there’s nobody but them that can be responsible for protecting these values.

  XSS protection has become a basic knowledge requirement for all developers.
    Like much to do about security … it’s always only as strong as its weakest link.

At FINN.no because we take security seriously, and because we know we are only human and we need some room for the occasional mistake, for each release we run security audits. These include using WatchCom reports, tools like Acunetix, and custom tests using Cucumber.

Fork me on GitHub casus telefon telefon dinleme casus telefon telefon dinleme casus telefon telefon dinleme casus telefon telefon dinleme casus telefon telefon dinleme