RoundCube 0.7.1 login failed

Note to readers: RoundCube 0.7.1 uses the Iloha IMAP Library. RoundCube 0.7.2 does not, so this particular problem may already be fixed.

In the past I’ve had to enable PLAIN IMAP authentication because older versions of M$ Outlook didn’t support CRAM-MD5 authentication. This means that my IMAP server advertises CRAM-MD5 as well as PLAIN as possible authentication methods.

All my clients who wanted to login using PLAIN cannot login using CRAM-MD5 because their passwords have to be stored in a different format.

A problem occurs when you set RoundCube’s “imap_auth_type” to “check”, because it picks CRAM-MD5 over PLAIN every time. I don’t want this to happen because some of my clients cannot login using CRAM-MD5. Ideally RoundCube should be following RFC3501:

If an AUTHENTICATE command fails with a NO response, the client MAY try another authentication mechanism by issuing another AUTHENTICATE command. It MAY also attempt to authenticate by using the LOGIN command

…but RoundCube doesn’t. To temporarily fix the problem you should be able to set “imap_auth_type” to “plain” so that RoundCube always uses PLAIN to authenticate against the IMAP server. However, I found that this does not work in 0.7.1 because of the following problem:

RoundCube passes options to the iil_Connect function of the Iloha IMAP Library. The options object includes a property “auth_method”, meant to control the authentication method, but this is never used, because the iil_Connect method expects this property to be named “imap”.

Line 382 of /program/include/rcmail.php is where the options object is defined – I simply changed ‘auth_method’ to ‘imap’.

Email signatures rant

I hate long email signatures, they are one of the most unnecessary, pointless, wasteful and irritating things I can think of. And what the hell does it even tell me? I mean perhaps, if it was the first email I ever sent to someone outside of my workplace I’d want to write my name, company and perhaps my website address in my signature, but if we work at the same company, or have known each other for 18 years, wtf are you sending me this information for?

Who are the idiots who write their email address in their signature? I know what your email address is! You just emailed me!…and there are TWO, yes two SMTP headers that perform this job – “from” and “reply-to”. Stop it. Stop it now.

Another major annoyance is that they get in the way of quoted text in email threads, so I end up with 500 lines of signature for every 2 to 3 lines of useful text. If you hadn’t already guessed, this makes reading emails really difficult…and I don’t even like reading easy to read emails.

But what really grates is the people who have the audacity to tell me (in green text) to consider the environment and not print the email, when, if I did print it, their signature would take up over 90% of the used paper and ink anyway. Not to mention the additional bandwidth and thus electricity they’re using by sending such a massive amount of text down the line EVERY TIME they send an email. YOU consider the environment and remove that text from your signature.

That leads nicely on to multipart MIME HTML emails. HTML emails are usually sent by your email client as two versions: a plain text version (so recipients who have mail clients that can’t display HTML can read the message) and a HTML version. So every time someone sends out a HTML email they actually send a duplication of the same content. The worst part being that 99.9% of emails sent to me do not need to be HTML and are often only sent as HTML to accommodate visual layout of information in the signature. So stop sending me HTML emails unless it is truly necessary! You’re wasting my bandwidth.

Integration testing transactions and optimistic locking with Spring and JUnit

I did some badness today.

I wanted an integration test to check to see if my @Transactional method actually did rollback on an optimistic lock failure. It’s a pretty business critical method so I wanted to be sure that the whole @Transactional and @Version annotation voodoo actually works.

The approach I took was to inject, what I called a “slowRepository” into the object with the @Transactional method. The slowRepository is just a mockito mock of the object which “answers” (using “thenAnswer”) to the “findBy…” with some custom code.

The idea is that at some point during the @Transactional method the slowRepository is called and the “answer” code is invoked. In the test, the answer code follows along the lines of:

1. Grab the data it requires from the (not slow) repository (the repository the slowRepository is mocking)
2. Switch back the slowRepository for the not slow repository
3. Invoke the @Transactional method again on a separate thread
4. Sleep for a bit
5. Wake up and return the data from step 1

The @Transactional call on the separate thread completes and commits before slowRepository wakes up. When it does, slowRepository returns stale data and the @Transactional method then fails accordingly.

The problem I encountered was using the setters for my @Autowired dependencies on the bean with the @Transactional method. I needed to use the setters so that I could switch in and out the slowRepository, but couldn’t because Spring AOP proxies are implementations of an interface, not subclasses (by default – yes I realise you can get Spring to proxy the target class using CGLIB). I found this blog post which explains how to get at the target object behind a Spring proxy. I converted it to Scala:

…which is pretty ninja.

So, what the hell happens when you try and save changes to an @Version’d entity when another thread saves it’s changes before you?

Imagine you’re setup with a JPA 2 environment, using Spring Data JPA and Hibernate is your JPA vendor. You have a UserRepository interface which extends JpaRepository and therefore a Spring generated userRepository bean for all your user DAO needs. You also have a User domain object with a field annotated with @Version.

FYI, your find, save, delete etc. methods in your userRepository are automatically @Transactional. Thank you, Spring.

Ok, lets formalise this a bit:

Thread 1 finds User record with ID 1
Thread 2 finds User record with ID 1
Thread 2 changes the email address field
Thread 2 saves the changes to their version of the user object
Thread 1 changes the email address field
Thread 1 saves the changes to their version of the user object

At the point where thread 1 saves changes, it’ll check the version field of the user object and also the version of the user object in the database. They’ll be different, and a HibernateOptimisticLockingFailureException will be thrown.

Note also that the exact same thing happens if either thread 1 or thread 2 had deleted the object. So, all well and good huh? Check this out for sensible:

Thread 1 finds User record with ID 1
Thread 2 finds User record with ID 1
Thread 2 changes the email address field
Thread 2 saves the changes to their version of the user object
Thread 1 changes nothing, but then saves their version of the user object

What happens here? I originally thought thread 1 would throw an optimistic locking exception. Except, it doesn’t – Spring/JPA/whoever is in charge back there is smart enough to know that nothing changed, so it doesn’t save anything, and you end up with thread 2′s changes as you wanted. Clever girl.

Woes with Scala native types, Spring, JPA and FreeMarker

I’m working on a project using Scala, Spring, JPA and FreeMarker (as you probably guessed from the title of the blog post). I love working with Scala because for the most part, it makes my code more succinct and it is really exciting to work with. I mean, there are obviously other reasons but those are two big ones.

What I don’t like is having to convert between Scala and Java equivalents, but sadly it is unavoidable. The scala.collection.JavaConverters and scala.collection.JavaConversions objects make this a lot easier through implicit and explicit conversions but it does make you wonder what the performance hit is…

I mostly choose to ignore that there is a performance hit – performance problems are a happy problem anyway right?…and at the moment the benefits of being able to write code in Scala greatly outweigh any of my worries.

I use Scala types wherever possible in this project i.e. List, BigDecimal etc. but sometimes it’s just not possible. My domain objects have to use Java types because the JPA vendor doesn’t know how to translate the Scala types into SQL column types. The same goes for objects that are going to be passed to FreeMarker. FreeMarker cannot iterate over a Scala list using the <#list scalaList as item> syntax as you normally would since it doesn’t “know” what a Scala List is. As far as FreeMarker is concerned a Scala List is a java.lang.Object (see scala-lang.org/node/71).

My repositories are generated dynamically by Spring. Basically you define an interface and Spring magically generates a class that extends Spring’s JpaRepository and implements your interface. There is no two ways about it, I want this magic, but it means that the “interfaces” (which are coded as traits in Scala) have to return Java lists for findAll or findBy operations – boo.

Overall it isn’t actually that much of a pain, so I don’t know why I’m complaining about it. It is just something that slightly tarnishes the code, but I guess I can live with it. After all I did decide to use a bunch of Java frameworks with my Scala code. Why didn’t I just use Scala frameworks? Well, mostly because I’m pretty mad, but I’m not completely insane. I figured switching from Java to Scala was a big enough brain realignment without changing everything else I’m used to using as well.

Spring Web Flow – displaying your JSR-303 validation messages using FreeMarker

Instead of creating a custom validator in the traditional sense I’m relying on JSR-303 annotations to specify validation constraints, because, it is the future, and I shouldn’t have to do a load of coding to specify common validation requirements like an email address regular expression or a numeric value that is required to be within a defined range.

Using Spring web flow, I found that binding error messages weren’t available when I called “<@spring.showErrors ‘<br/>’/>” in my freemarker view (after I had bound the field I wanted to show errors for of course).

So where are my error messages?

It turns out that Spring Web Flow has a different way of providing the user with feedback messages. The Spring Web Flow reference documentation says: “Spring Web Flow’s MessageContext is an API for recording messages during the course of flow executions”.

The message context (with all your binding error messages in it) can be found here: flowRequestContext.messageContext. I’ve written a couple of macros to make retrieving error messages from this object a little easier:

<#--
 * Shows flow messages (which reside in flowRequestContext.messageContext)
 *
 * @param source Name of the field that caused the error
 * @param severity String representation of org.springframework.binding.message.Severity
 * @param separator the html tag or other character list that should be used to
 *    separate each option. Typically '<br>'.
 * @param classOrStyle either the name of a CSS class element (which is defined in
 *    the template or an external CSS file) or an inline style. If the value passed in here
 *    contains a colon (:) then a 'style=' attribute will be used, else a 'class=' attribute
 *    will be used.
 * @param tag The HTML tag to wrap the error in
-->
<#macro showFlowMessages source severity separator classOrStyle="" tag="">
  <#assign messages = flowRequestContext.messageContext.getMessagesBySource(source)/>
  <#if (messages?size > 0)>
    <#list messages as message>
      <#if message.severity?string == severity>
        <#if classOrStyle == "" && tag == "">
          ${message.getText()}
        <#else>
          <#if classOrStyle == "">
            <${tag}>${message.getText()}</${tag}>
          <#else>
            <#if tag == ""><#local tag = "span" /></#if>
            <#if classOrStyle?index_of(":") == -1><#local attr="class"><#else><#local attr="style"></#if>
            <${tag} ${attr}="${classOrStyle}">${message.getText()}</${tag}>
          </#if>
        </#if>
        <#if message_has_next>${separator}</#if>
      </#if>
    </#list>
  </#if>
</#macro>

<#--
 * Shows flow messages (which reside in flowRequestContext.messageContext) in an ordered or unordered list
 *
 * @param source Name of the field that caused the error
 * @param severity String representation of org.springframework.binding.message.Severity
 * @param classOrStyle either the name of a CSS class element (which is defined in
 *    the template or an external CSS file) or an inline style. If the value passed in here
 *    contains a colon (:) then a 'style=' attribute will be used, else a 'class=' attribute
 *    will be used.
 * @param ordered Whether or not the macro should output the list as an <ol> or <ul>
-->
<#macro showFlowMessagesList source severity classOrStyle="" ordered=false>
  <#local errorsList><@showFlowMessages source, severity, "", "", "li" /></#local>
  <#if errorsList?trim != "">
    <#if classOrStyle == "">
      <#local attr="">
    <#elseif classOrStyle?index_of(":") == -1>
      <#local attr=" class=" + classOrStyle>
    <#else>
      <#local attr=" style=" + classOrStyle>
    </#if>
    <#if ordered><ol${attr}><#else><ul${attr}></#if>
    ${errorsList}
    <#if ordered></ol><#else></ul></#if>
  </#if>
</#macro>

I’ve added these to my spring extensions, which you can download here: springx.ftl

FreeMarker pagination macros

I couldn’t find a good set of macros to help navigate through paginated data so I wrote my own. The macro names echo the names of the CakePHP PaginationHelper functions because…well, they’re reasonably sensible.

The macros expect you to add a model attribute called “paginationData” which contains (at least) the following properties:

pageNumber -> The current page number
pageSize -> The number of items in each page
pagesAvailable -> The total number of pages
sortDirection -> The sorting direction (ascending or descending)
sortField -> The field currently sorted by

In my implementation, my paginationData also contains a property called “pageItems” – the actual items in this page.

The macros available to you are:

  • first
    Outputs a link to the first page
  • last
    Outputs a link to the last page
  • next
    Outputs a link to the next page
  • previous
    Outputs a link to the previous page
  • numbers
    Outputs links to other pages. The macro takes a parameter called “maxPages”, which is the maximum number of pages that should be output. There is also a second parameter “separator”, which is the text/html that should be output between each page link.
  • page
    Outputs a link to a particular page. It takes the page number and link text as parameters. If you omit the link text it’ll default to the page number
  • counter
    Outputs the current page number and the total pages e.g. “1 of 20″
  • sort
    Outputs a link to sort by a field. Takes the field to sort by as its first parameter. If field is different to the current sort field, the link will change the sort field but not the sort direction. If the field is the same as the current sort field, the link will change the sort direction. The second parameter is the link text, which defaults to the field name with an upper case first letter. It also takes a parameter called “directions”. This is an an array of two items: the words being used in paginationData.sortDirection to describe the sorting direction of ascending or descending. Default: ["Asc", "Desc"]. So it can compare the current sorting direction and switch to the converse.

Note, ordinarily you’ll probably not need to use the “page” macro.

How to use

In Java, add a model attribute “paginationData” containing the properties I’ve described above. Then, in your view, import the library:

    <#import "pagination.ftl" as pagination />

Call the macros:

    <nav style="float:right;">
        <@pagination.first />
        <@pagination.previous />
        <@pagination.numbers />
        <@pagination.next />
        <@pagination.last />
    </nav>
    <@pagination.counter />

Which’ll give you something like:

For table headings that allow sorting:

    <table>
        <tr>
            <th><@pagination.sort "forename" /></th>
            <th><@pagination.sort "surname" /></th>
            <th><@pagination.sort "email" /></th>
            <th><@pagination.sort "created" "Registration date" /></th>
        </tr>
        ...
    </table>

Download

You can download the pagination macros here.

How to switch to/from HTTPS using Apache as a proxy to Tomcat

I’m writing this down because it too me an age to figure out a way of doing this. I have a website which Tomcat is happily serving. Areas of the site require a secure connection so I’m using Spring security to require particular URLs to be accessed over HTTPS. It means that when I access http://example.org:8080/webapp/login, it’ll bump me to https://example.org:8443/webapp/login. Note: Tomcat is setup with the SSL connector and a self signed .keystore see (http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html).

I have two vhosts setup in Apache, one for the http://example.org and one for https://example.org. They are both using mod_proxy to ProxyPass and ProxyPassReverse requests to the appropriate Tomcat URL’s. The problem comes when switching to HTTPS from HTTP and vice versa. Ideally I wanted some sort of ProxyPassReverse declaration in my config for http://example.org what would change HTTP headers (that Spring sets) for https://example.org:8443/webapp into https://example.org. Except ProxyPassReverse doesn’t work like that.

Now, I realise I could simply not use Spring to manage which parts of the site should be accessed over HTTPS and which should not…and just setup Apache to redirect as appropriate. I don’t want to do that though, because that makes the task of adding these restrictions a deploy time task, rather than a development time task. I don’t want to risk someone forgetting to add new restrictions when deploying the webapp and I’d much rather the developer added these restrictions when they were working on the task and really thinking about where and when they are needed.

So, how do I solve the problem so that the app can manage its secure-ness and I can setup Apache once and forget about it? The answer is to ProxyPassReverse onto a “special” URL, which when accessed will redirect to the HTTPS (or HTTP) site. For example, if the HTTP site needed to redirect to the HTTPS site, I’d add rules like so to perform the redirect:

    # Proxy a request (from the server) to switch to https onto a special URL "/2https/"
    ProxyPassReverse /2https/ https://example.org:8443/webapp/

    # When a client requests a URL prefixed with "/2https" map it onto the secure site
    RewriteRule ^/2https/(.*)$ https://example.org/$1 [R,L]

…and you’d add something similar to the secure site Apache config. As long as I don’t mount any pages at /2http or /2https I should be ok. Note a couple of things:

  • You’ll need “SSLProxyEngine on” and “RewriteEngine on” and obviously the appropriate Apache modules loaded for these commands.
  • Because of the redirect between HTTP <-> HTTPS you won’t be able to POST data between them directly (I’m not sure why you’d NEED to though)
  • Obviously you’ll need to setup Apache with an SSL certificate…but that is a different story

I should say a special thanks to this random site – from whence the idea actually came from. If anyone has any better ideas on how to do it I’d love to hear them. Please comment below.

Simple FreeMarker random number generator

FreeMarker doesn’t have a random number generator function. I needed a really simple solution that would allow me to pick a random image URL to be displayed on the homepage.

<#--
* Generates a "random" integer between min and max (inclusive)
*
* Note the values this function returns are based on the current
* second the function is called and thus are highly deterministic
* and SHOULD NOT be used for anything other than inconsequential
* purposes, such as picking a random image to display.
-->
<#function rand min max>
  <#local now = .now?long?c />
  <#local randomNum = _rand +
    ("0." + now?substring(now?length-1) + now?substring(now?length-2))?number />
  <#if (randomNum > 1)>
    <#assign _rand = randomNum % 1 />
  <#else>
    <#assign _rand = randomNum />
  </#if>
  <#return (min + ((max - min) * _rand))?round />
</#function>
<#assign _rand = 0.36 />

I’ve added this function to the FreeMarker Spring extensions I’ve been building up. You might use it in your code like so:

<img src="<@spring.url "/images/" + springx.rand(1, 10) + ".jpg" />" />

Note that there are obvious issues with this function. Aside from being based heavily on the second the function is called, you’ll notice that the min and max have less probability of being generated as the other values…what? I said it was simple.