среда, 31 декабря 2014 г.

5 Types of Goals

This time of the year lots of accountants validate balances in the books of their firms. In quite a similar fashion when December draws to a close my thoughts constantly come to back to the goals that I had set back in January. Seeing how my results of the year vary from area to area, it is quite easy and at the same time interesting to break these goals into categories based on both the degree of success with them and the reasons that stand behind it.

First and most simple, there is a number of things that I wanted to do this year, but didn't even start to shoot in their direction. In some cases I did make initial steps but lacked persistence to see even minor results and gave it up. The failures of this kind are mostly caused by mere laziness. However, sometimes the problem is deeper: many of unachieved goals were not understood and formulated properly. This does actually boil down to laziness, but that's the laziness that occurred back on the stage of planning and prevented me from setting myself up for success. The presence of such failures reminds me that I must put more thought into planning and avoid picking goals, of which I am not sure whether I really want to pursue them and why.

Fortunately, neither of the past years were marked with complete failures only and 2014 is not exceptional. With some of the plans I came very close to the objectives. Even though I fell short of my own expectations in regards to, for example, the number and frequency of posts in this blog, I am still quite happy with the fact that I was regularly working on this goal. The interesting thing about this kind of under-achievements is that they bring me almost as much satisfaction as the act of reaching a target. The only difference is that when I don't manage to fully meet a goal I have to analyze the reasons that stand behind underperforming - mostly to allow for more accurate planning, but also to find where exactly I am inefficient and try to improve on that in the next year.

On the other side, there are some points on my 2014 checklist, which I pursued hard and can easily mark as achieved, but which gave me mostly disappointment. A nice example of a goal of this kind is to accrue a certain amount of savings over the year - I do have this on my list. I was quite disciplined, got some help from outside - particularly, from my employer - and, thanks to that, succeeded at having the desired amount on the bank account by the 31st of December. The problem is that the currency that I used to set the goal and accumulated my earnings - the Russian Rouble - depreciated by more than 30% since the moment when I had prepared my plans for the year. I could lament a lot about the world and Russian economy, in particular, being unfair and make a tragedy of this in dozens other ways, although this would be attributing my own planning mistake to someone else's actions. Even though the goal looked pretty reasonable at the time of setting, it did become 30% less relevant over the course of the year. Despite the fact that this happened due to external events and without much of my own guilt, I should have had some measures of protection against depressions of this kind. Instead, I assumed that once an objective is set I only have to work hard to meet it, but goals may and do lose value over time because both we and the world change, so one has to monitor and adjust them accordingly.

Judging by the above paragraphs one might think that I am mostly a failure, because I either don't achieve my goals or they bring little satisfaction to me. Fortunately, that's not the case and I did proudly check-marked some important points on the 2014 to-do list. These took quite a lot of work and made me feel happy about both putting in all the efforts and getting to the finish line. Success of this kind not only lets me see what I am good at - it also encourages me to continue working simply because I know I am capable of something. Add the feeling of proudness for merely getting planned things done and you'll see that a year worth of time wasn't wasted and opened new roads for the next one.

Finally, there is a type of goal or achievement in my classification that looks quite strange to me. Back in the beginning of the year along with a list of goals I wrote down some thoughts on where I want to move with my job and which new responsibilities I am willing to take. Since I had only vague understanding of what my goals in this area are and how one can achieve them, that was merely a set of ideas with only a few action points, which could hardly lead to any serious changes. Still, despite the fact that this didn't look like a plan at all, by virtue of luck, other people's will and who knows what else, during the year I was moving in the direction defined by these notes and in December found myself far beyond what I could have expected and hoped for in January. Things like this - when I get something that makes me happy without any definite plan - happened to me occasionally, but it is the first time when I had something written down, didn't understand how I can get it and nevertheless got much more. 

Even though the "overachievements" like this do bring tons of happiness and satisfaction, it is also crucial to analyze them carefully once you're done with the celebration. That's a good thing to do because understanding one's strengths helps both planning and working, and, more importantly, because it is useful to know how big is the share of mere luck and colleagues', friends' and relatives'  help in that success.

So, now I have a great basement for building and meeting plans for the next year. I see where I underperformed, what I am capable of and how I can improve my results in certain areas with little effort. Equally important, I know the things, which I shouldn't pursue the next year and also understand how to avoid getting too disappointed about failures, a certain amount of which is inevitable. All this will help me make clearer goals, do better at working towards them, adjust the direction on the way and avoid getting depressed when I fail or find that I have been pursuing the wrong target.

I wish you to set your goals for 2015 and make it a Happy New Year through achieving them!

четверг, 18 декабря 2014 г.

Running a Podcast for a Year

Last January, together with Michael, we recorded the first episode of the Code Speak Loop podcast. Shortly after that I set up a goal to record 20 to 22 episodes over the year 2014 and it looks like we  are succeeded at it. At the moment, after almost 11 months since publishing the episode 1 we have released 19 shows and will release one more in less than a week. The experience was magnificent!

While this marks a great achievement for us, there are lots of spots where we didn't perform as good as we would like to. For example, we could have released more episodes - 22 or maybe even 24. This a bit more impressive milestone would be easy to reach if we managed to maintain more regular release schedule. Unfortunately, during the last 3 or 4 months we failed to record a number of episodes on time, thus loosing several weeks. This also resulted in larger gaps between episodes - at least once the gap exceeded 4 weeks, which hardly complies with our original plans.

While the lack of discipline that led to these gaps is shameful, it didn't prevent us from reaching the original goal in terms of the number of episodes - both because the goal was quite cautious and, surprisingly, we were not as bad as expected. On the other side, there are things that look much worse. One of the most important problems is the quality of the content that we produced. Sometimes our accents, the level of control over our voices as well as the quality of the language were bad enough to make me seriously consider throwing the recording away. However, I believe that we were right to ignore this problem for some time and push the first episodes in public despite it - over time we did much better. Not only did this allow us to achieve sort of satisfactory quality, it also helped us improve general communication skills significantly and do better at our jobs. I am more than sure that the trainings that I delivered this summer as a part of my job would have been much worse if I didn't spend that much time chatting with Michael, listening to the recordings and preparing them for publication.

However, all the above problems are basically technical ones and I understand how to control them. What bothers me most in regards to the Code Speak Loop is that now, after releasing 19 episodes we don't really understand whether we have any listeners and what do they think about the show if they exist. This sounds stupid, but that's how it is. Thanks to the awesome Signal Leaf podcast hosting service we do know that we have some subscribers, but it is not clear whether these are real people or some sort of a Skynet trying to understand its enemy better.

Furthermore, the fact that over a year we got virtually no comments - both in the blog and on our social media pages - makes me think that the number of our actual listeners is quite close to zero. I know that some of my friends did check out the podcast, some even listened to quite a few episodes and I am extremely thankful to them. Still, because we didn't receive a word from anyone outside our usual circles it feels like we were broadcasting to nowhere. To be honest, this isn't too much of a surprise because, as usual, we avoided going into public and promoting the podcast at all costs. Additionally, the topics that we discussed are maybe too diverse, so we failed to position ourselves for any definite audience. Bad news here is that this is not the first time I make a confession of this kind: earlier this year I wrote that I had completely failed at marketing my Windows Phone app and actually I did this with two of them (the third is coming soon). Even though Michael worked much harder in this area, the efforts were still not enough to attract at least some angry comments to our blog.

Peeking into the future, I can say that whatever we will chose to do about the podcast there are some things that I am willing to change. First of all, we are not going to set any goals for the number of episodes or any schedule for the next year. Sometimes under pressure of our own plans we didn't pay enough attention to the quality of the show and that's not something that we want to repeat in future. I do believe that this was the right thing to do at the first stage when it was crucial to see whether we are able to produce anything at all, but now we can turn to recording only when we want to and have something really interesting to discuss.

Making the schedule less strict will also allow us to focus on getting more guests into our studio. We had several fabulous people this year and the episodes with a guest were definitely a lot denser with content and more focused than the ones with only two of us. Pursuing the same goal of making episodes more interesting and increasing their value, I'd like to put more work into choosing and preparing topics, so let's see what happens.

Finally, we do have to put significant effort into attracting and engaging listeners. Even though for us this is likely the most difficult part, there are some ideas, which may execute well. We don't have much choice here actually: while running a podcast without listeners for a year is possible, we don't look crazy enough to do that for two years in a row.

We wouldn't be willing to continue this enterprise without the support that we got from our friends who listened to the show and gave extremely valuable feedback as well as the awesome guests who came and shared cool stories and valuable experience. Our thanks go to all these fabulous people! And even though our subscribers are invisible, we know someone is refreshing the feed from time to time. So thank you for listening and tune in next year!

пятница, 21 ноября 2014 г.

Zipping Sequences in Clojure

It is no secret that most of our programs involve processing sequences and this is especially true about programs written in any flavor of LISP, including Clojure. Because sequences play such an important role we frequently want to combine them into something bigger. There are dozens of possible ways to get a collection from several collections, but I want to stop on zipping sequences together.

Zipping means taking two (or more) collections and transforming them into a collection of tuples (pairs in case of two collections), so that we get a tuple of first elements of every input sequence followed by a tuple of second elements and so on. In most programming languages that I am familiar with there is a function that performs the zip operation. Here is what it looks like in C# (Zip is an extension method that lives in the System.Linq namespace):

var a = new List<int> { 1, 2, 3 };
var b = new List<int> { 10, 20, 30 };
a.Zip(b, (x,y) => new Tuple<int, int>(x, y))

//=> [{Item1:1, Item2:10}, {Item1:2, Item2:20}, {Item1:3, Item2:30}]

For the LINQ's Zip we have to provide a couple of collections and a function that will produce an element of the result from the elements of the inputs - I have chosen to create a Tuple of two ints, but it is possible to do virtually anything here. Obviously, one can go further and zip 3 or more sequences through chained calls to Zip:

var c = new List<int> { 100, 200, 300 };
a.Zip(b, (x,y) => new Tuple<int, int>(x, y))
    .Zip(c, (xy,z) => new Tuple<int, int, int>(xy.Item1, xy.Item2, z));

//=> [{Item1:1, Item2:10, Item3:100}, {Item1:2, Item2:20, Item3:200}, {Item1:3, Item2:30, Item3:300}]

As you can see, in C# this can get quite verbose, but that's the price of control over the form of the resulting items. Python takes a different approach and simply assumes that when we are zipping something we want a list of tuples back:

zip([1, 2, 3], [10, 20, 30], [100, 200, 300])

#=> [(1, 10, 100), (2, 20, 200), (3, 30, 300)]

That's what Python is praised for - it's simple and succinct.

Now let's go and see what Clojure has to say on the matter. The first impression from peeking into the cheatsheet is that there isn't any analog for zip - the closest thing that one can find is the zipmap function. zipmap actually does something quite similar, but as its name suggests it produces a hashmap instead of a vector or list:

(zipmap [1 2 3] [10 20 30])

;=> {3 30, 2 20, 1 10}

Most importantly, this means that when using zipmap we have to assume that one of the sequences holds keys, which is not always possible. Additionally, hashmaps don't preserve the order of items essential in many cases. And to take it even further, zipmap doesn't play well with several collections - it takes only two arguments and it is not easy to combine the result with another sequence in a meaningful way. Does that mean that Clojure, which is so passionate about data structures and sequences in particular, can't zip two vectors? Absolutely not - it's just the case that the simplest way to do it is a bit obscure.

Clojure has a vector function that wraps whichever parameters you pass it into a vector. Basically, (vector 1 2 3) yields [1 2 3]. Another Clojure function that is relevant here is map. This one calls a given function with each element of the sequence passed to it and returns a collection of results. The cool part is that map can take any number of input collections extracting one argument for the function being mapped from each of the collections. Essentially, this means that map over several sequences zips them into one according to the provided function. Thus, if we want an analog of Python's zip it is as simple as this:

(map vector [1 2 3] [10 20 30] [100 200 300])

;=> ([1 10 100] [2 20 200] [3 30 300])

So the next time you need to zip several vectors or lists into a sequence of tuples of respective elements you know how to do it. One may argue that the Clojure's "version of zip" is less elegant than the Python's one and that's true to some extent. However, it is almost a rule that when something is very easy to do using the existing building blocks Clojure doesn't implement it for us. In this particular case I do like this approach because it allows me to think of zipping as a more general operation than just combining a number of collections into a single collection of tuples. In some sense it feels like Clojure's way to zip collections combines the elegance and simplicity of the Python's zip with the high level of control of the C#'s one. For example, if my goal is to add a couple vectors together, I could do something like this:

(defn addvec [a b]
    (map (partial reduce +)
        (map vector a b)))

(addvec [1 2 3] [10 20 30])

;=> (11 22 33)

But taking into account the aspects of map function that we have discussed, it is very easy to simplify this dramatically. Do we really need this intermediate representation of the result - the collection of tuples? No, we want only the sums of elements, so let's avoid verbosity and simply map + over both collections at once:

(defn addvec [a b]
    (map + a b))

Furthermore, map allows for arbitrary number of input collections, so why focus on the edge case of adding together two vectors? The only thing that we have to add into our mix to allow for arbitrary number of input vectors is the apply function:

(defn addvec [& vs]
    (apply map + vs))

(addvec [1 2 3] [10 20 30] [100 200 300])

;=> (111 222 333)

This ability to produce concise and generic solutions is maybe the most impressive feature of Clojure. Of course, you can use all these functions to achieve your own goals and even though they are likely much more ambitious than summing vectors I bet you will be amazed by the simplicity of the solutions.

Here are some links to the documentation:

1. vector
2. map
3. apply
4. zipmap
5. mapv (if you want the result as a vector instead of a lazy sequence)

And a couple of insightful Stack Overflow discussions, which helped me find the way to zip collections in Clojure:

1. One
2. Two
3. Three

пятница, 24 октября 2014 г.

Fixing the Wrong Problem, Once Again

The last week I spent two or three hours looking for the cause of a bug really close to the place where it was actually located – but not close enough. From the exact moment when I opened the bug description in Visual Studio everything pointed me in the right direction – towards a couple lines in our code base, fixing which would have resolved the issue. However, when reproducing it according to the steps outlined in the issue tracker and exploring the related behavior of the system I have caught a glimpse of something peculiar, that could have been the reason of my issue, and it made me waste a lot of time on trying to fix the wrong problem.

A particularly important detail about this new problem that I discovered was that its incorrectness didn’t look like something trivial and minor. It seemed to me that I was looking at a very severe bug spanning a number of components of our system. At some point, I was almost sure that I have found a really big problem, much broader than what the reporter could have thought. This feeling of stumbling upon something significant that might have far-fetching consequences quickly made me forget my initial objective.

Moreover, I was excited in a very negative way by the idea that I see something that other devs on the team have overlooked, and the desire to show everyone how cool I am pushed me even further away from what I should have been fixing. The fact that the newly found problem looked much more complex than what followed from the original description also meant that it was more interesting. Hence, my eagerness to fight it instead of taking a broader look and examining all the aspects of the problematic process.

It took me a couple hours of digging and a relatively long break to persuade myself that I should try and go through all the steps that cause data corruption and the related error. Initially, I have stopped my attempts to find the exact source of the issue once I had spotted something strange and unexpected. Should I just follow the description of the issue, carefully crafted by the QA engineer, up to the point where the problem is clearly seen, I would have reached the root of the problem much faster – maybe in a matter of minutes.

What happened instead is actually very familiar to me – that’s not the first time I make this mistake. From time to time, when tracing down a bug I face something strange-looking in the middle of the way and start exploring it. I do this despite being fully aware of the idea that any problem must be solved from where it manifests itself down to its actual reason – without taking any shortcuts. I know that one should under no circumstances skip any steps on this way, but I still do this occasionally and it never brings me close to where I would be happy to find myself.

This exact problem is what the strategy of moving slowly and in tiny steps aims to solve. For me the nature of the issue began getting clearer once I pushed aside the keyboard and started writing – with pen and paper. These tools make one work slower, they don’t allow to hack one’s way through the code and skip crucial points. Moreover, they facilitate deeper and calmer thinking, while rapidly stepping over the lines of code in a debugger or executing numerous variations of an SQL query usually descends one into a state where only wild guesses and ‘let-us-try-this’ hacks are possible.

My takeaway is once again that one must not rush to solve any problem that they happen to see – frequently what we initially perceive as incorrect behavior turns out to be desired in the end. Instead, one should work towards fixing what they aimed at first. In many cases problem solving works better if one maintains a queue of the problems – not a stack. Fortunately, even though our minds seem to be using a stack internally, we can always push extra problems out of it – to a bug-tracker or to a good old sheet of paper – so that the original, the right problem can be fully comprehended and solved, giving way to new glorious fixes.

пятница, 10 октября 2014 г.

User's Expectations and Business Process

During a recent discussion with my friends I have faced an interesting contrast between what looked like the right thing to do in a mobile application and what a more traditional UX common sense suggests. While the way I thought the application should work seemed pretty reasonable from the business point of view, my friends were reluctant to accept it. The strength of their resistance to my approach descended me into doubtfulness and made me search for flaws and the ways to align the app’s behavior to the expectations of potential users without losing the emphasis on the business process.

My task was very simple and can be described in general terms without diving into details. The application deals with jobs, which user is supposed to create. In addition to that, he needs to do some work with the existing jobs – particularly assign workers to them. In terms of UI, this boils down to several application pages, only two of which are of interest now. The first one provides the inputs to set various attributes of a new job. The second view comes with a list of jobs, which were just created but haven’t been assigned a worker yet – user picks one there and selects workers for it. Once this is done, the job leaves the list and moves further along the pipeline. The problem sits right between these two pages: views on how the transition between them should happen when user fills all the properties of a new job and presses Create  might differ a lot.

One approach, which you will find in many widespread feed-based applications and which looks quite natural, suggests that once the job is created it should appear on the top of the list and be the first thing to draw user’s attention there. That happens, for example, when you post something to Twitter or Instagram – when you tap Post in the corresponding application, you are transferred back to the feed and see your new photo or message added on top of it. This is the approach that my friends were proposing. It actually has lots of benefits: user sees the item they have just added and can not only check that everything is correct, but also take some action. For example, right after publishing this entry I will want to look through it to locate things like typos or extra blank lines between paragraphs, and it is great that the blogging engine shows me the latest post first. However, for my application it didn’t feel suitable.

What I have chosen to do instead was to keep the list sorted by creation time in ascending order – the older the job the higher it appears in the list. Moreover, the app doesn’t scroll the list down to the newest item when the user adds it. For my friends this made an impression that when the list is already long new jobs just vanish unnoticed somewhere under the tower of older ones. While that certainly looks like that, there are some reasons to lean towards this approach. In my opinion, these fully justify the initial confusion that can arise from the fact that the feed does not work the way Twitter does.

Firstly and most importantly, the application’s purpose is to assist an employee in their interaction with a queue of customers and the pool of workers. In such a situation, when service time is critical, the older jobs are likely the hotter ones, meaning that if you have a job, which lies unassigned for 30 minutes already, you definitely want to process it before the one created a moment ago. Furthermore, since customers are served in FIFO order, in a situation when the list of unassigned jobs is already long and all workers are busy, arrival of a new customer should not interrupt the ongoing process. Under these circumstances, the manager handling the queue – our user – only has to put the new customer on the queue and forget about them. It is not necessary and possibly even harmful to draw too much attention to this new customer – instead of messing with a client, who has to wait for their turn, we should first process the jobs, which entered the queue earlier. (Of course, I don’t mean that the manager should literally forget about new customers, but he must be focused on the ones who have already spent some time in the queue.)

These considerations make me confident about the second approach, which pays close attention to the process that the application is supposed to help with – maybe in favor of complying with user’s initial expectations. At the same time, the discussion that we had made me thoroughly consider the problem. The least important and most obvious thought that I had in relation to it is that our views on how something should work and look might be affected too much by the things, which we interact with on the daily basis, despite the fact that the difference between the two can be vast. Here I don’t mean that my friends are totally addicted to Twitter. Instead, when they voiced their concerns over my design I have found myself in doubt and seriously contemplating adoption of the more popular approach. It looks like we are too used to feeds of tweets, photos, links and news growing from the ground up and sometimes oversee the fact that not everything should work this way.

More important takeaway is that doing the right thing is not always the same as taking a popular and well-established approach to UI. This flows naturally from the previous paragraph and merely suggests that before jumping on the bandwagon of any time-honored and user-approved UX pattern we must carefully study the goals of our products and how they can help users achieve their own objectives. This means that behind any application there is a process, which we try to enhance, improve and optimize and it is vital to take every detail of it into account when designing the application.

Last but not least, what we face here is the contradiction between user’s expectations and the goals of the software. When facing something like this, we should not forget that keeping users comfortable and amused is not the only purpose of a mobile (or any other) application. It should be easy to work with a product and the latter must not stay in the way of doing one’s job, but a business app occasionally has to make the user uncomfortable and draw their attention to something that they might be unhappy about. In the end, kittens, food and beautiful landscapes are for Instagram, while business apps are all about doing business.

пятница, 5 сентября 2014 г.

There Is No Arcane Knowledge

When starting to learn things about the world of computer programming I was sure that there is some piece of knowledge that I had to discover to become an actual programmer. My perception was that there are facts, ideas and thoughts, which were not available to me at the moment due to some reasons, but that once I learn them, it will be the moment when a programmer is born.

In childhood I used to think this way about all the things that people do: for example, I could have learnt some elementary physics in school and thought that while it looks and feels like physics, that’s certainly not something actual scientists work with. I understood that to learn what real physicists do I should go to a university, spend several years and then one day, somewhere around the fourth or fifth year they will raise the curtains and show me what is the science about and how it is totally different from what I imagined.

I don’t know whether such mindset is a common thing for children and youngsters, but it is certainly not a very good thing for students. My confidence that really important knowledge will be opened to me only at some distant point in future was a serious blocker for learning. The problem here is that when you think that the things taught to you are not that important – at least not as vital as something else that you are supposed to face later – you are not going to learn much. Additionally, having to study something useless and feeling obliged to wait for important portions of knowledge leads to frustration, at best. What’s more, the feeling that I will be taught crucial stuff in future made me believe that the only thing that I must do is waiting, while what a student (and as you gradually discover, everyone) has to do is actually learning – that is trying to do things, asking questions and looking for answers everywhere their mind can reach.

This way, it turns out that the most valuable thing that I learnt in the university, while trying to become a programmer, is that there is no arcane knowledge, which someone is concealing from me. In other words, there isn’t a piece of any particular science or profession that one has to wait to learn and that, when finally discovered, turns the student into a professional, be it a programmer or a physicist. Instead, there are lots of details behind each and every subject and each of them might seem lacking significance, not important or even irrelevant at some moments. However, the truth is that one has to study as many of these details as possible to get a feeling and to draw some unifying concepts from the experience, which means understanding what one is learning.

Once I have come to appreciate this obvious idea the game has changed and it felt like lots of doors opened, letting me learn whichever subject I found worthy. It suddenly appeared to me that the only blocker to learning new ‘real’ stuff is the limited time that one can devote to it – apart from that you can go anywhere and study whatever you wish. A good finding, ain’t it?

The nasty part is that I still sometimes have to remind myself that there is no knowledge that someone is trying to hide from me, preventing me from being a better programmer, advancing from a junior to a senior position and so on. As before, I sometimes find myself thinking that one day I will learn the last piece required to be a great programmer and will finally turn into one, although I have already got used to telling myself that becoming a better programmer is not about studying a certain amount of required things. I will keep facing the idea that something is not available to me yet, but, unlike with my first years at the university, now I know that the reason is not that the knowledge is hidden, but that I didn't try the right things and didn’t ask the right questions yet. Hence, I should keep asking new ones and looking for answers to them, because that’s the only way to learn something new. What’s even more important, this process of searching for answers and bringing them to the world is what delivers me the experience and being a professional is more about the experience than anything else.

This said, eager to become a great software developer, instead of waiting for someone to teach me a magic trick or two I will better sketch some design diagrams and put together a bit of code – that seems to be the only reasonable way to achieve my goals.


суббота, 16 августа 2014 г.

Combining Liberator and Korma to Build a Web-Service in Clojure

In a recent episode of the Code Speak Loop podcast I mentioned two Clojure projects: Liberator, designed to build REST services, and Korma, allowing to talk to relational databases easily. I’ve been working with these libraries lately and it turns out they play quite nice together. In this post and the related repository on GitHub I will show the way I combined Liberator and Korma to build a simple RESTful application so that anyone who wants to do something similar has an example. I did not put much effort into separating concerns and making code clean in this sample, still I think it conveys the general ideas properly.

Here we will set up a task-list application, which would allow to view, add and edit tasks over HTTP. There is a bit more to it on Github, but I won’t cover many of the details here. For a database I used a local Postgres installation with a very simple table structure – there is a schema.sql script in the repository. It should not matter much whether you use Postgres or not, although if you pick some other DBMS you will have to change the DB connection configuration in the application (see below). Besides, some problems may arise with timestamps.

Let us start with the database. To talk to it we use Korma, and Korma in turn uses entities. These are the descriptions of the database tables written in Clojure with a defentity macro. Entity definition normally includes a set of keys, a list of fields to select from the corresponding table by default and possibly a name of the table – if it differs from the name of the entity. Additionally – and that’s the coolest part – entities might include relationships, which allow to extract linked entities seamlessly.

(declare tag)

(defentity task
  (pk :task_id)
  (entity-fields :task_id
           :title
           :description
           :is_done
           :is_cancelled
           :created_time
           :finished_time)
  (many-to-many tag :tasktag))

(defentity tag
  (pk :tag_id)
  (entity-fields :tag_id
           :tag)
  (many-to-many task :tasktag))

(defentity tasktag
  (entity-fields :task_id :tag_id))


In our application there are only three entities – task, tag and tasktag. Both in task and tag we specify that tasks are related to tags with a many-to-many link – to do this we only need to specify the second entity and the name of the linking table (:tasktag) in our case. We don’t define any relationships for the tasktag entity – that’s because we need it only to insert and delete records, which link tasks and tags together. To achieve our other goals the relationships defined on task and tag are pretty enough. (Note however, that I don’t show tag and tasktag entities at work in this post – take a look at the code on Github.) You can find a lot of info regarding entities on the Korma site.

Once we defined the entities, we have to tell Korma where we want to get them from – that is specify a database connection. In Korma you do this by means of defdb macro passing it a connection description generated from a dictionary. I use postgres function provided by Korma that will setup all the required parameters for connecting to a Postgres database. There are plenty of other helpers like that in Korma – check them in the docs.

(def dbcon 
  (postgres {:db "libekorma" :user "postgres" :password "Aw34esz"}))

(defdb dbconnection dbcon)


Now that we defined the entities it’s time to give access to them through a resource. Resource is the fundamental concept in Liberator, which binds together various handlers and parameters, that define what will it do under which conditions. We create a resource with defresrouce macro and for our simple case we will specify only the :available-media-type – we deal with JSON, :allowed-methods – GET is enough so far, and a function to :handle-ok – this one will get tasks from the database and encode them in JSON format – that’s what users will get. Even this simple example shows that Liberator allows to manage a lot of HTTP-stuff without much ceremony. The most important part of the resource for now is the :handle-ok function, called when the resource thinks it should respond with 200 HTTP code – that’s what happens when user sends GET request because we don’t have any restrictions and in this case we should respond with a list of tasks.

(defresource tasks-r
  :available-media-types ["application/json"]
  :allowed-methods [:get]
  :handle-ok (fn [_]
        (json/write-str
          (select task (with tag)))))


To make it all work we have to do only one more thing: define a Compojure route that will expose the tasks-r resource. Its definition starts with ANY, which means that the route accepts any HTTP method. Liberator handles allowed methods through its own mechanism (:allowed-methods) and thus there is usually no need to make Compojure expect a specific verb.

(defroutes app
  (ANY "/tasks" [] tasks-r))


(You can check the project.clj for a list of libraries that we use.)

However, if you try to request some data from /tasks, you will likely run into an error message telling that the app can’t produce JSON output because it doesn’t know what to do with timestamps. Even though this does sound scary, thanks to the extensibility of clojure.data.json this problem is pretty easy to deal with – we just extend the Timestamp type with a simplistic implementation of the JSONWriter protocol:

(extend-type java.sql.Timestamp
  json/JSONWriter
  (-write [date out]
  (json/-write (str date) out)))


Now you can check that everything, including this last trick, works fine. That means one can retrieve the list of tasks with GET request and observe the records stored in the database (be sure to insert some for testing – there is a sample.clj, which can do this for you). This is not too impressive though, so let’s proceed and allow for tasks creation with POST in the same resource:

(defn tasks-request-malformed?
  [{{method :request-method} :request :as ctx}]
  (if (= :post method)
    (let [task-data (util/parse-json-body ctx)]
      (if (empty? (:title task-data))
        [true {:message "Task title missing or empty"}]
        [false {:task-data task-data}]))
    false))

(defresource tasks-r
  :available-media-types ["application/json"]
  :allowed-methods [:get :post]
  :malformed? tasks-request-malformed?
  :post!
    (fn [{task-data :task-data}]
      (let [data (into task-data {:created_time (util/cur-time)})]
        (insert task (values data))))
  :handle-ok (fn [_]
        (util/to-json
          (select task (with tag)))))


Here we add 2 things. First, when someone’s posting data to us we want to check that it complies with our requirements. Validation of this kind can be done in the malformed? handler of the corresponding resource. Particularly, for tasks we don’t allow empty :title, so for requests with bad title our tasks-request-malformed? function returns a vector of true (yes, the request is malformed) accompanied by an error message. If, on the other side, a proper title is present in the posted data, the function will return false – not malformed – and a dictionary including the parsed request data under the :task-data key. This illustrates the proper way to pass data between various decision points in Liberator: if along with the result of the check (true or false) you return a map from the handler, liberator will merge it into the context and downstream handlers will have access to whatever there is in the dictionary.

In our example the data provided by the malformed? is used by post! handler, which gets it from the context by means of destructuring. Beside this, in post we add the :created_time field to the same dictionary and call Korma’s insert with it. That’s it, we enabled creating tasks – core functionality is here!

One particular thing to note are the calls to the cur-time function. There is nothing magical about it – I just use it to abstract away instantiation of the Timestamps for the time columns in the database:

(defn cur-time []
  (Timestamp. (.getTime (Date.))))


If you take a closer look at the malformed? handler above, you’ll notice that it uses the parse-json-body utility function. This one combines two other functions and json/read-str to get the JSON body of the request from context, turn it into a Clojure map and transform its string keys into keywords. In other words, the function creates an easy to handle dictionary from a raw stream buried deep in the context. Be aware that the keywordify function used here is not recursive, so only the top-level keys will become keywords, while nested dictionaries will still have string keys.

(defn body-as-string [ctx]
  (if-let [body (get-in ctx [:request :body])]
  (condp instance? body
    java.lang.String body
    (slurp (io/reader body)))))

(defn keywordify [dict]
  (into {}
    (map (fn [[k v]] [(keyword k) v]) dict)))

(defn parse-json-body [context]
  (if-let [body (body-as-string context)]
    (keywordify (json/read-str body))
    {}))


Liberator allows to define a lot of various handlers thus opening doors for managing any particular condition in proper place and time. The general idea is that when processing a request Liberator will navigate the decision graph and execute handlers defined for visited nodes. In the example above we used only the malformed? decision point to parse and check the incoming request. Next, we will implement a separate resource for deleting and updating individual tasks and see how one can implement other handlers.

Let us start with something simple – deletes. We actually need to define only two handlers: delete! and exists? As a bonus, we will also implement one under :handle-ok to allow for getting tasks by ID – just because it is very easy:

(defresource one-task-r [task-id]
  :available-media-types ["application/json"]
  :allowed-methods [:get :delete :put]
  :exists?
    (fn [_]
      (if-let [task
        (first
          (select task
            (with tag)
            (where {:task_id task-id})))]
        [true {:task task}]
        [false {:message "Task not found"}]))
  :delete!
    (fn [{{task-id :task_id} :task}]
      (delete task
        (where {:task_id task-id})))
  :handle-ok
    (fn [{task :task}]
      (json/write-str task)))


Here we remove tasks in the delete! handler with a simple call to Korma’s delete with task entity and a where clause restricting the ID of the task. However, the function provided under the :delete! keyword gets called only in case the one specified with :exists? yields truth or a vector starting with truth – there is little sense to deleting missing tasks. Our implementation of the exists? handler attempts to select the task from the database by its ID and upon success returns it together with true. Here the pattern is the same as in the malformed? handler – we use the dictionary to pass data around so that , for example, downstream handlers don’t have to query database once more. In case you update or delete records it makes a lot of sense to retrieve them in the exists? handler and then use them when they are needed.

What makes this resource very different from the previous one is that it takes an argument – task-id. This might seem strange because otherwise resources look more like dictionaries, but that’s the thing that Liberator handles without any work required from us – we just accept this gift. As for passing the parameter in, we do it in the route definition like this:

(defroutes app
  (ANY "/task/:task-id" [task-id] (one-task-r (Integer/parseInt task-id)))


Now we can remove the tasks – and only the existing ones. Deletes, however, are very simple in comparison to updates, which we are going to implement next. First thing that we need is a malformed? handler that will parse and validate the request – we have already seen something like this in the previous resource:

(defn task-update-request-malformed?
  [{{method :request-method} :request :as ctx}]
  (if (= :put method)
    (let [task-data (util/parse-json-body ctx)]
      (cond
        (empty? task-data)
          [true {:message "No new values specififed"}]
         (and (contains? task-data :title)
            (empty? (:title task-data)))
          [true {:message "Empty title is not allowed"}]
        true
          [false {:task-data task-data}]))
    false))


The new thing is the conflict? handler. In our case, it is pretty simple and just verifies that the task doesn’t end up completed and cancelled at the same time – this is a forbidden state:

(defn task-update-conflict? [{new-task :task-data old-task :task}]
  (let [combined (into old-task new-task)]
    (if (and (:is_done combined) (:is_cancelled combined))
      [true {:message "Invalid state after update"}]
      false)))


As you might guess (or discover from the decision graph), Liberator invokes the conflict? handler for put requests somewhere between the exists? and put! handlers. This means that you already have access to data extracted by exists? and can check whether the update can cause any problems here, without turning the actual put! handler into a state validation mess. Note that the OK return value here is false – meaning no conflict.

Having this handler separated is cool because there are usually quite a few other things that you have to decide upon when executing update, so it might end up messy by itself:

(defn update-task [{new-task :task-data old-task :task}]
  (let [just-finished?
      (or
        (and (:is_done new-task)
           (not (:is_done old-task)))
        (and (:is_cancelled new-task)
           (not (:is_cancelled old-task))))
      finished-time-dict
      (if just-finished?
        {:finished_time (util/cur-time)}
        {})
      updated
      (into finished-time-dict
        (filter
          (fn [[k _]] (#{:title :description :is_cancelled :is_done} k))
          new-task))]
    (update task
      (set-fields updated)
      (where {:task_id (:task_id old-task)}))))


Our update-task function evaluates the :finished_time field if needed and passes it to Korma’s update together with the values coming from the user. It is important to filter the latter and exclude the fields that should not be updated under any conditions – e.g. :task_id and :created_time. That’s what we do when assembling the updated dictionary.

(defresource one-task-r [task-id]
  :available-media-types ["application/json"]
  :allowed-methods [:get :delete :put]
  :can-put-to-missing? false
  :malformed? task-update-request-malformed?
  :conflict? task-update-conflict?
  :put! update-task
  ; + a bit more - see above
  )


In the resource we specify these routines under the malformed?, conflict? and put! keys. Additionally, we have :can-put-to-missing? set to false, which prevents updates to non-existent tasks. The update-related functions do look quite complex, but they’d be absolutely dreadful without the separation of concerns offered by Liberator and that is its main strength.

Another cool thing about this library is that it automatically manages the execution flow in a fully HTTP-aware way. That is, when building an app, you don’t need to think, for instance, of producing appropriate error codes and giving them back to user in the required form – Liberator will do everything on its own. On the other side, it doesn’t stand in your way and allows to fine-tune the resources the way you need them.

As for Korma, it’s key advantege is simplicity – at least that’s what I love about it. It isn’t an overweight ORM or something – it just allows to run queries against a database from Clojure, but it does it in a very natural way. However, I also can’t leave out the relationships feature of Korma, which permits linking the entities together in a straightforward manner.

Our simple example also shows that these two libraries work great together thanks to how Liberator makes you consider only one thing at a time and how Korma simplifies access to data. I began using these about a month ago and I must admit that I truly love this way of producing REST services.

The sample project that I show here is a bit bigger than I managed to stick into the post. For example, it allows to add tags to the tasks and to ask for a full list of tasks tagged with a particular word. This doesn’t introduce many new concepts apart from demonstrating Korma’s relationships in action – I just tried to go further along the road of showing Korma and Liberator. You can find full sources on GitHub – do clone the repo and play with the code!

I will appreciate your comments – both regarding the examples here and about your experience at building web services with Clojure! If you spot an error in the code or just can’t figure out what a particular piece does – please let me know. I’ll be happy to fix the mistakes and explain what I meant!

пятница, 1 августа 2014 г.

Aging

I was pretty fortunate to be born in the middle of summer as this gives me two very well defined and perfectly positioned points in a year when I am willing to reflect on the results of recent months. In January everyone’s year ends to give way to a new one and in July my own age gets incremented – both events suit well for peeking into past and future. So, a couple weeks ago I turned 25 and this somehow made me think about what does it mean to get older apart from the fact that every year you have to tell different numbers when answering a question “How old are you?”

I mean I am not a complete idiot an I know that over the time of my existence the Earth has done roughly 25 full circles around the Sun, cells in different parts of my body died off and got replaced with new ones many times and all that stuff. In other words, something changed in the world and something changed in me as well: I used to be an infant, then a little boy, then kept growing larger and stronger and now I am this guy with a laptop in a park. I also know that from this point on I will likely grow weaker – not stronger, not until I put some serious effort into avoiding this. Still, physiological aging feels as boring as it is inevitable.

Being a bit more intelligent than a rock, I also acknowledge that there is a social aspect to aging. When I was born I wasn’t actually a member of any society at all and was hardly capable of being one. I gradually started interacting with my parents and other family members and being a part of a small group of people. After learning some stuff from them as well as other guys of my age one day I became eligible to enter a larger community – an elementary school. From there I advanced to the high school, then university – both of them taught me some crucial skills and knowledge making me potentially valuable for Humanity. Finally, somewhere close to the 7th of 6 years of my study at the university I learned that I can not only consume what society gives me, but also give something back. That’s when I took my first part-time job in software development. A year ago I finished study at the university and working as a programmer became my fulltime activity, so that I moved a bit closer to the production side of the consumption-production gauge. This is certainly a very important aspect of aging, yet the same as the biological part of the matter it doesn’t seem interesting.

Continuing the search for the “essence of aging”, I also remember of responsibilities: the older one gets the more responsibilities they are supposed to carry. Until some time you have only to be doing what others tell you, so you are only responsible for not breaking the simple rules they set for you. Later, one takes the new, sometimes less clear responsibilities associated with study or job. Furthermore, one may take responsibilities for others' well-being and well-fare when they get married and give birth to children. So, the general idea is that children are responsible for nothing, while grown-ups are in charge of everything and that’s what I was taught in school and elsewhere. Even though I believe that’s true – to the extent anything about the way human society functions can be true – that’s not something that I have been facing a lot during the recent years. I mean I did get new responsibilities when starting to work as a developer or when obtaining the driver’s license, but all these feel so natural that it is hard to define aging in these terms.

What does seem to be essential to aging for me is learning and, most importantly, learning that most things won’t do themselves. Learning itself is crucial and it is often true that the older you get the more you learn – sometimes that’s what makes you capable of going further. Still, it just happens to come along with aging. On the other side, understanding that some things are just not going to happen the way you want them until you push them hard into the desired direction comes to me as a manifestation of the fact that I am getting older. 

There are plenty of examples of how this simple idea might jump at one here and there. For instance, long ago the laces on my shoes used to get tied properly without any help from me. Later the magic vanished and I discovered that one has to put effort to get out of home without losing their shoes. Something very similar happens with keeping shoes clean – for some time while you are trying to accustom yourself with the laces, magic keeps cleaning your shoes, but later it evaporates as well. You notice that shiny footwear becomes bleak and dusty, dirt sticks to it and doesn’t go away overnight any more. This way every year one sees some magic being drained away from their world.

This thought is not something original or new, but it struck me this year. I mean my parents stopped cleaning my shoes long ago – that’s other things that drew my attention to it. For example, I always used to be a very slim person. During some periods of my life this was supported by my attempts to do running consistently, but even when I was much less physically active I remained slim and hardly seen any body fat. However, the situation changed dramatically about a year ago. Somehow, after I took my current position and started to work full-time at the office my body began to build up fat. I don’t fully understand why did this happen – the amount of physical activity didn’t change much between the university and the job. One of the reasons may be that my life became more determined than a couple years ago when I had much more options for where to sit with a laptop and study and when to change the place. Anyway, some months ago I noticed that I got much fatter than I have ever been before and that my body is not going to remain slim or fit without some help from me anymore.

I also made similar discoveries closer to the software development field. While I already knew that code doesn’t write itself (although, there is meta-programming!), it turned out that there is still some place for surprise. My Windows Phone development adventures taught me one thing for sure: the products that I make won’t market themselves – not until I put significant effort into making them stand out and forcing potential customers to see them. Put in other words, my customers won’t come to me until I reach out and find them. So, basically nothing is going to happen by itself in terms of distributing and marketing my products – I have to do something to show them, explain them and sell them.

All these “discoveries” of things, which I believed to be driven by magic, but which can actually be driven only by me, do feel obvious. At the same time I’m pretty sure I will have a lot more of them in future. None of them is too difficult to predict now – almost everything that I might rely on will one day require some payment from me and stop working silently the way I am used to. However, like with shining shoes, testing my code and marketing my apps, I will definitely be very surprised by each and every thing that didn’t seem to need any help to get done but suddenly starts to do so when I get a little bit older.

пятница, 20 июня 2014 г.

A Project Makes You Sick? Just Work Harder!

A couple weeks or so ago I faced a problem with a project I am working on now. On that stage I was busy creating a mobile application and really struggled with it. I felt that the UI is ugly, usability sucks in every possible way and some features are almost impossible to implement with the approach that I have chosen. This all led to a wave of disappointment – maybe the strongest one over the last months – crushing on me and to a sensible desire to give the whole thing up. Fortunately, this time I managed to overcome these feelings relatively easy and soon got excited over the project once more – at least in comparison with how severely negative I felt about it in the middle. The way I did this was very simple: I used a pretty effective weapon to fight tiredness – I have chosen to merely put more time and effort into what pissed me off.

I can’t remember where I picked up the idea that when feeling tired one has to work harder and it will get easier – it just feels pretty natural to me and I have been trying to follow this piece of wisdom for several years already. This way when I really feel exhausted with everything that I am busy with at the moment, I can use this idea to give myself a shake and get going further. The truth is that even though I don’t completely understand how this can possibly work, it keeps helping me get rid of apathy and tiredness. Besides, it is cool to be that nasty guy who responds to someone lamenting about being too tired with a phrase like “you just don’t work hard enough”.

At the same time I always thought that this approach to getting out of "too tired" mindset has anything to do only with the amount of work that one does and the actual tiredness. However, it turns out that it’s not the case and this same approach scales down to the level of particular projects pretty well. In my most recent case just forcing myself to fight through the unwillingness to continue the project literally made me see a lot of important things from different angles and actually understand much of what I failed to grasp before. However, even though it might seem that I got an “energy replenishment” of some sort from this act of overcoming exhaustiveness, I believe this works in a bit different way.

The real reason behind my apathy was that I started seeing a lot of mistakes that I had made in both design and implementation of the application and didn’t know how to fix them in any reasonable way other than rewriting everything from scratch. Moreover, the more I worked on the thing the more problems surfaced and the more depressed I got with them – finally arriving at the state where I wished nothing more than quitting the project. On the other side, when I built up some strength to overcome this desire and continue pushing the thing further it did get much better. Quite quickly I got valuable insights from my furious attempts to improve the application in various ways, which made me see all my faults and problems clearly. These, in turn, gave me the idea of the right way to tweak existing code, which would both save me from some of those errors and allow to move further.

The overall lesson is that the advice of just working harder when one feels tired is not as dumb as it may sound at first. Of course, sometimes it is a good idea to actually get some rest, but in other cases making yourself continue to put effort into a project even when you can’t stand it anymore may be very fruitful. Part of the trick is that, like with my mobile application, the same issues that make you hate the project in combination with the time spent thinking about it will likely help you understand what you want the thing to look like. So, when you face the idea that what you are working on is utter crap and you don’t want to see it even a moment longer, consider putting these thoughts away. Instead simply try to push harder, hack some code and make a bad UI even uglier – I am sure this will help you see your problems clearly and give you and idea where to go next.

What do you do when a project of your's becomes unbearable? How do you help yourself pass the stage of despair and apathy? Or maybe something deep in your mind allows you to work smoothly without even facing the problems of this kind? It'd be cool to hear from you in the comments!

(By the way, it may be not evident but I was literally on the edge of giving up this post somewhere between the 2nd and 3rd paragraphs. Still, I made myself continue and finally ended up with something that I can publish.)

пятница, 6 июня 2014 г.

How to Make an SQL From Clojure or My Macros Kata: Part 2

In a recent post I showed some strange things that one can do with macros to be able to write SQL-like queries right in Clojure. Several weeks ago we enabled selecting fields from records as well as filtering them with where clause. In the end of the first post I promised you that we will soon add sorting capabilities to our query language and enable joining “tables” together. Not that I am going to totally break this promise, but I will deliver on it only partially: this time we will do sorting and the sweetest part, joins, we leave for the final post.

Before proceeding, let me remind once more that the code obtained through this exercise is not something that can be used in any real application – it sucks from both performance and feature-richness perspectives and is hardly idiomatic, nice or readable. Furthermore, while doing the exercise I gradually came across the idea that what I show here is by no means a good way to utilize the power of macros. Still, I am sure that implementing a query language like this is a good exercise and can really help one understand what macros are about, how to handle them and, at least, where one should not use them. This way of learning is hard but it pays back a lot once you put enough effort into it.

Let’s recap where did we stop last time. First of all, we mastered picking fields from records and can write simple queries like this one:

(def table [
  {:a 1 :b 100 :c "100" :d 4}
  {:a 2 :b 200 :c "200" :d 3}
  {:a 3 :b 300 :c "300" :d 2}
  {:a 4 :b 400 :c "400" :d 1}])

(select :c from table)
;=> ({:c "100"} {:c "200"} {:c "300"} {:c "400"})

However, since this is hardly an impressing achievement and can be done in a much more concise way, we have gone further and allowed for filtering – starting from very simple conditions and proceeding to quite long and complex ones:

(select :a :b from table where :a < :d)
;=> ({:a 1, :b 100} {:a 2, :b 200})

(select :a :b from table where :a = 2 or (:d < 3 and :b > 300))
;=> ({:a 2, :b 200} {:a 4, :b 400})

Our key goals were to make the syntax as close to SQL as possible and I think we succeeded at it – at the cost of introducing a lot of complex stuff, whose main purpose is to deal with symbols like from, where and others. Now let’s take a look at our macros and see what we can do to make ordering possible. The key player in our team is select:

(defmacro select [& what]
  (let [fields (set 
          (take-while #(not= 'from %) what))
      source (fnext 
          (drop-while #(not= 'from %) what))
      conditions (next (drop-while #(not= 'where %) what))]
      `(map (fn [record#]
            (into {} (filter #(~fields (first %)) record#)))
          (filter #(condition % ~@conditions) ~source))))

For now, select is aware of from and where clauses, which it uses to find the desired fields, the source table and the filtering conditions in a query. To enable sorting we have to pay attention to the order by clause as well and here I am going to cheat. For the sake of making code a tiny bit simpler I will use a single orderby word without a whitespace – this will save us a couple keystrokes. The only place in our code where we have to pay attention to this symbol is in the select macro, where we have to retrieve the actual ordering conditions from the original query. For this we use the same approach as with where – basically skip everything up to the orderby symbol and the symbol itself. At the same time, we should remember that the new clause at the end of the query might introduce a problem: we don’t want this stuff to get into the condition macro, because upon meeting orderings it will immediately spit an “Unable to resolve symbol: orderby” error. Thus, we also restrict the conditions list on both sides:

(defmacro select [& what]
  (let [fields (set 
          (take-while #(not= 'from %) what))
      source (fnext 
          (drop-while #(not= 'from %) what))
      conditions (take-while #(not= 'orderby %) 
             (next (drop-while #(not= 'where %) what)))
      orderings (next (drop-while #(not= 'orderby %) what))]
      `(map (fn [record#]
            (into {} (filter #(~fields (first %)) record#)))
          (filter #(condition % ~@conditions) ~source))))

Now, once we have the orderings list extracted from the query, we can use it in some way. Because the select macro doesn’t look pretty straightforward already, it is a good idea to implement the sorting stuff elsewhere and just use it in select. For this purpose we are going to introduce another macro - order. To start with, let us make it distinguish between 2 situations: when no ordering is requested and when there is actually some stuff in the ordering conditions list. For now we can simply return nil in the latter case. The place to plug ordering in select seems quite obvious: sorting should be done after filtering so we will just wrap the existing body of select (the syntax-quoted part) in a call to the new macro:

(defmacro order [orderings what]
  (if (empty? orderings)
    what
    nil))

(defmacro select [& what]
  (let [fields (set 
          (take-while #(not= 'from %) what))
      source (fnext 
          (drop-while #(not= 'from %) what))
      conditions (take-while #(not= 'orderby %) 
             (next (drop-while #(not= 'where %) what)))
      orderings (next (drop-while #(not= 'orderby %) what))]
      `(order ~orderings
        (map (fn [record#]
            (into {} (filter #(~fields (first %)) record#)))
          (filter #(condition % ~@conditions) ~source)))))

If you execute any query with the updated macro you will see that the only thing that has changed is that now we get nil when there is a non-empty orderby clause:

(select :c from table where :a > 1)
;=> ({:c "200"} {:c "300"} {:c "400"})

(select :b :c from table where :a > 1 orderby :b)
;=> nil

Let’s take this as an evidence that we didn’t break anything and proceed with sorting. First, what are we expecting to get in the list of ordering conditions? Definitely, it should contain keywords denoting the fields to order by. Besides, each of these keywords may have an optional asc or desc symbol after it, standing for ascending and descending order, respectively. When the keyword is not followed by such a symbol, we will assume ascending order, similarly to what SQL dialects do. Now, this is not a completely trivial thing to implement, so let us first do something simple – for example, handle the case when there are only keywords in the orderings list. For this we only need to use the built-in sort-by and juxt functions in the order macro:

(defmacro order [orderings what]
  (if (empty? orderings)
    what
    `(sort-by (juxt ~@orderings) ~what)))

(def table [
  {:a 1 :b 400 :c "200" :d 4}
  {:a 2 :b 300 :c "100" :d 3}
  {:a 3 :b 200 :c "400" :d 2}
  {:a 4 :b 100 :c "300" :d 1}])

(select :b :c from table where :a > 1 orderby :b)
;=> ({:c "300", :b 100} {:c "400", :b 200} {:c "100", :b 300})

(pprint (select :a :b :c :d from table orderby :d))
;=> ({:a 4, :c "300", :b 100, :d 1}
;=> {:a 3, :c "400", :b 200, :d 2}
;=> {:a 2, :c "100", :b 300, :d 3}
;=> {:a 1, :c "200", :b 400, :d 4})

The examples show us that filtering and the simplest sorting facilities both work nicely. Still, if we chose to ask for ascending or descending order explicitly, the compiler will award us with something like “Unable to resolve symbol: asc”. Let us allow for these symbols and get closer to our goal. Here I want to start with something that I have been avoiding since the beginning of our strange exercise – we will get rid of the asc and desc symbols in the order macro well before doing anything else. Even though while implementing select and condition I got used to handling symbols in macros, they still bother me a lot and I want to force them out as fast as possible. We will replace asc with 1 and desc with -1 – this feels pretty logical and saves us from comparing everything to asc and desc as well as from the attempts to prevent Clojure from evaluating these symbols.

(defmacro order [orderings what]
  (if (empty? orderings)
    what
    (let [orderings-desym
        (vec (map (fn [s]
        (cond (= 'desc s) -1
            (= 'asc s) 1
            (keyword? s) s))))]
    `(sort-by (juxt ~@orderings) ~what))))

Next, we need a way to understand which keywords should go with which order. This would be very simple should we not allow to skip the asc symbol, but with this extra convenience feature we have to find a way to handle this without falling into some non-trivial loop-recur construct. Fortunately, this can be done in a not very elegant but satisfactory manner:

(defmacro order [orderings what]
  (if (empty? orderings)
    what
    (let [orderings-desym 
        (vec (map (fn [s]
        (cond (= 'desc s) -1
            (= 'asc s) 1
            (keyword? s) s))))
          orderings-separated 
          (->> orderings-desym
              (partition-by keyword?)
              (map (partial interpose 1))
              flatten
              (partition-all 2))]
    `(sort-by (juxt ~@orderings) ~what))))

This seemingly strange sequence of functions transforms the incoming list of ordering conditions into a sequence of pairs of the form (:key 1) or (:key -1). To achieve this result we first separate the list into sublists of keywords without ordering markers and those of ordering markers themselves. Then we use interpose, which will add missing 1‘s, and flatten the resulting list to finally split it into pairs. To make this easier to understand let’s take a look at the example – here are the stages of this transformation for a particular list of ordering conditions:

;0. [:a :b :c 1 :d :e -1 :f] ;initial orderings
;1. ((:a :b :c) (1) (:d :e) (-1) (:f)) ;partition-by keyword?
;2. ((:a 1 :b 1 :c) (1) (:d 1 :e) (-1) (:f)) ;map (partial interpose 1)
;3. (:a 1 :b 1 :c 1 :d 1 :e -1 :f) ;flatten
;4. ((:a 1) (:b 1) (:c 1) (:d 1) (:e -1) (:f)) ;partition-all 2

As you see, each keyword apart from the last one gets its 1 or -1. – missing tail is, in fact, not a big problem and we will deal with it shortly. Now we have set up everything except for the actual sorting part – we need to transform the new sequence of conditions into a list of functions suitable for sort-by. For this purpose, we can introduce an auxiliary function and consume it in the order macro:

(defn order-fn
  ([k]
    k)
  ([k ord]
    (if (neg? ord)
      `(fn [r#] (- (~k r#)))
      k)))

(defmacro order [orderings what]
  (if (empty? orderings)
    what
    (let [orderings-desym 
        (map (fn [s]
        (cond (= 'desc s) -1
            (= 'asc s) 1
            (keyword? s) s)) orderings)
          orderings-separated 
          (->> orderings-desym
        (partition-by keyword?)
        (map (partial interpose 1))
        flatten
        (partition-all 2))
          order-funcs 
          (map #(apply order-fn %) orderings-separated)]
    `(sort-by (juxt ~@order-funcs) ~what))))

(pprint (select :a :b :c :d from table orderby :a desc))

;=> ({:a 4, :c "300", :b 100, :d 1}
;=> {:a 3, :c "400", :b 200, :d 2}
;=> {:a 2, :c "100", :b 300, :d 3}
;=> {:a 1, :c "200", :b 400, :d 4})

It works! Let’s try to understand why. The key piece is the order-fn, which generates sorting functions for us. The one-argument version is trivial – it takes a single key and returns it to the caller. Why do we need something this stupid? Well, that’s simply our way to handle the last keyword in the orderings sequence when it comes without explicit asc or desc. As seen in the example above in this case it won’t have an ordering marker (1 or -1) in the orderings-separated list, but with the help of order-fn defined as above we can just ignore this and everything will work fine. 

The two-arguments version of order-fn is a bit less trivial but still manageable: it will either return the keyword if we want ascending order, or wrap it into a function that will "invert" the numeric value retrieved with the keyword otherwise.  At the same time, here we have one nasty feature: it returns a syntax-quoted fn form instead of a mere function. This might seem weird – and it is – but at the same time that’s the only way it is going to work. The problem here is that we use order-fn in a macro to produce another function – a closure around the k argument. This combination of macro, function and closure somehow makes compiler vomit “No matching ctor found for class user$order_fn$fn__*” errors. I must admit that I failed to understand this problem – if you can explain it I would be grateful. Still, for now we, at least, can go on with quoting.

The way we use order-fn is very simple – we map it onto the list of (key, sorting order) pairs and then pass the resulting functions to juxt. The latter produces a single function, which extracts all the values used for ordering from a record. Finally, in combination with sort-by this gives us a properly ordered results set.

Now, you might have spotted several problems with our solution. The most severe one (apart from the approach in general) is that we allow sorting only by the numeric fields – strings won’t do. This is because to produce an ordering function for desc we use simple numeric negation, which isn’t compatible with strings, sequences or anything else. I have tried to come up with a more generic solution, but the time and brainpower that I had put into it were not enough. Fortunately, my goal of understanding macros better and learning to be not afraid of quoting and unquoting allows me to live with this. However, if you come up with a more generic implementation of order-fn it will be a great pleasure for me to see it – please share it in comments!

On the other side, the use of quoting in the order-fn makes me think that something is definitely wrong with our approach to the problem. The degree of complexity of our macros also strengthens this feeling – I still can manage them, but it ceased to be an easy task long before we produced a working implementation of orderby. I have already stated that this way of implementing SQL-ish queries doesn’t look like an idiomatic way to use Clojure macros and all these concerns only reassure me in this idea. Still, the audible tension that I had in my brains when fighting this stuff confirms that the exercise is a damn good kata.

Here are some of my takeaways from this iteration:
  • Maybe the most frequent error that I faced here was the “Wrong number of arguments passed to a keyword” one. The reason for this was always that Clojure tried to evaluate a list starting from a keyword when I didn’t mean it to be evaluated. This same problem manifested itself in some other ways and I find it the most important thing to tackle: you should always pay close attention to what and when you are evaluating.
  • I have expected to face some difficulties with sorting over several fields – until I remembered that there is juxt in Clojure. So one of the lessons from each and every exercise in programming is actually that there is likely a function to solve your problem. That's actually one of the reasons why it is good to do these exercises – you discover and remember a lot of useful stuff.
  • One sign of using a wrong approach to a problem is that you have a lot of bindings or variables and find it difficult to come up with proper names for them (order, order-fn, orderings-separated, etc simply don’t sound very good). Maybe the most appropriate thing to do under such conditions is to step back and try to rethink the way you reason about the problem.

I will add joins to the mess in the next post – as promised. For now, feel free to play with what we already have – here is the code. By the way, there is a problem with our new select macro, which limits sorting capabilities, but is very easy to fix. Did you spot it? Let me know in the comments!

In addition to showing me where I messed up, please tell whether you find this exercise a good way to accustom oneself with macros or do you think it is as useless as it is long and complex? Maybe you have your favorite Clojure kata, which helps to explore and understand macros better? I will be glad to hear about it!