воскресенье, 17 июля 2016 г.

Everyone on your team is important

Over the course of my career in software development I got used to the idea that the key thing we deal with everyday is complexity. It shows up on different stages of our jobs, but most interesting cases are of course tricky bugs. Sometimes the issues reported by our QAs and especially by end-users get so messy that it takes a full crew of professionals to fix them. I have faced one such example recently and in addition to providing valuable experience and the feeling of satisfaction in the end, it served as an evidence supporting that all the roles we have in our team are of great importance and play best when assembled into a team.

About a year ago we have fixed a couple minor issues in our system. To be sure that we have that piece of functionality working properly despite any future changes we decided to create automated tests for it. Our QA engineers did get to this task, but because life is what happens when you have other plans, that happened only recently. Roughly at the same time another member of our team has found an issue in a distant part of the system, totally unrelated to that first bug, and fixed it using a certain code construct.

Once these two things met in our repository the new tests failed. It so happened that the developer who was digging into the failure wasn't in any way familiar with any of the two changes. The error conditions turned out to be tricky, so some folks joined investigation and it was eventually traced down to that seemingly unrelated last fix to be routed to the appropriate platform developer and fixed there. The importance of this event however took some more time to be recognized.

Another chapter of this story starts several months earlier. One of our customers filed an unpleasant data bug that we never managed to reproduce and understand. It was hanging there unresolved and made partners, so to say, unhappy. It's high severity kept it visible for us, but we still weren't able to do anything about the issue - till the moment when the events described in previous paragraphs took place. When I was doing a daily review of recent and priority items, I stumbled upon this nasty bug first and then spotted the one that caused all the mess outlined above. Something about them looked similar. The way the latter one was fixed suggested a new idea for repro steps for the former. Late at night this fresh approach was attempted and yielded positive result - we not only understood the reason behind the high-priority issue, but also knew that it is now fixed and could report that to everyone concerned - the problem was with the same code construct and the same platform issue that started the first bug.

This basically means that the data problem that we were so worried about took us a couple QA engineers, two developers, a product manager and a team-lead doing their usual jobs to track it down and fully understand - not to mention the significant amount of time spanned by all the activities that led us to this result. This might indicate problems with the product being developed or with the process, but the truth about software development is that such problems that eat enormous resources do happen and, if you are working on a large and sophisticated piece of code, you can only minimize them - not get rid of them completely. Complex issues also require different people to handle them - sometimes just to throw enough pairs of eyes and points of view at the problem. Even worse, there is an element of a slot machine to this requirement, because you can only partially design how and when someone of your team-mates will play his role in such a process - in most cases these revelations and discoveries happen merely because a number of people are doing their work properly.

Software development is sometimes messy work - especially when we speak of business applications. I would like to say that it doesn't have to be like that, but it seems that for most products getting complex enough to give birth to problems of this kind is just a matter of time. As a result we do need all of our people in their positions - QA engineers, developers, analysts and so one. This incident that we faced highlights the fact that even when I have no clue at all about the job that some people are doing at our office, it is still very valuable. The beauty of a development company and its processes is that one day any one of us - or rather an unexpected combination of our efforts - may save the company from something nasty or bring it to a new solid achievement. This simple idea is also a great motivator, because when I get tired with the routine and think that the work I am doing is useless, I can remember how almost the whole our team won the battle with that bug and see that one day me doing my job may save some thousands bucks for the company and countless hours of work to my colleagues.

вторник, 14 июня 2016 г.

Take-aways from the Getting Things Done book by David Allen

Over the recent years I heard about the Getting Things Done methodology here and there, but only recently I finally read the book by David Allen and got closely acquainted with its idea. It turned out that I already employed some of the techniques that make up this famous self-organization system, but in most cases never acknowledged that there are bits of GTD among my tools. More importantly, the book taught me some new tricks and the proper way to combine and use the tools.

The most well-known bits of GTD are inbox and action lists. The inbox is used to record every bit of information that arrives to you for subsequent sorting. The action lists - to keep track of the actions that you need to take on duty, at home, when there is time to make a call, etc. I was already confidently using these by the time of reading the book, so this part didn't surprise me. The concept of projects, however, was more of a new thing to me. In Allen's terminology a project is anything that you're going to do and that will take more than one step to accomplish. This a bit unusual understanding of the term is combined with the idea that one must maintain a full list of their projects and review it regularly to decide on next actions. This implies that every week you take time and go through the entire list and for each project decide what you will do next to get the thing moving forward - and add these actions to your lists. This (quite obvious, to be honest) way to handle projects is my most valuable take-away from the GTD book, because when I am serious about weekly reviews it results in advancing a lot of things that I would normally forget about. Instead of making me drown in a chaos of numerous small projects, which I originally expected, this approach actually helps me follow all the projects that I deem important and makes sure that each of them is gradually moved to completion.

Another idea that resonated with me greatly was that one should have a reference info storage and have it in perfect order. And the order here is not just a matter of beauty, but a quality that allows one to put something in easily or to get whatever they need at a particular moment quickly. One manifestation of this idea is that reference information gets separated clearly from the next actions information. For me this played a huge role, because I'm the kind of a person who's obsessed with historical data, archives of all sorts and being able to remember what I was doing on a particular day three months ago. While this might be important in some cases, to achieve results one should see the current context and aim for future, using the past only as a reference. This transformed some of my routines a lot and helped to progress more efficiently both at my work and with other parts of my life.

Another goal that one sets when implementing GTD is to build a reliable system of reminders and ensure that one remembers the right things at the right time. That's the famous wait list as well as the usage of the calendar and even the checklists (which I embraced as a great tool before diving into GTD). It turns out, however, that there is only one way to achieve this goal - that is to organize all the information on next actions and events properly and review it regularly. Regular routines seem to be the second pillar of GTD - together with ordering your life. These are applied virtually to everything - from the mission and principles to the projects and next actions lists - and I can confidently say that regular reviews of my current standing are so critical to efficiency, that I could get little use from the GTD system without this bit. On the other hand, having finally developed a habit of doing these reviews, I now see way more control over different parts of my life and a significant increase in ability to control progress towards multiple unrelated goals.

There are many other useful ideas, but David Allen certainly explains them better than I, so if you're interested you should read it. I was quite impressed - in particular, because it was easy to apply some of the suggested approaches and to see how these yield more control over my life. Another thing that makes the book a great read is that it is very practical and goes to the level of an engineering textbook into the details of organizing yourself. At the same time it doesn't offer a magic pill being pretty honest about the amount of time and effort that one has to invest into personal management to implement the suggested approaches. If you're OK with that and are interested in pursuing efficiency, I do recommend to read the book and to make that investment because it will pay off quite quickly. Have a nice reading!

воскресенье, 15 мая 2016 г.

Using an external JavaScript library in your ClojureScript application

My recent playing with ClojureScript and Om was a great trip into the field of functional front-end programming. However it was disrupted by a small yet irritating problem. I wanted to consume an external JavaScript library - the client for Trello API - in my Om application and call a method of the object defined there. It all worked fine while I was developing the thing, but the first attempt to deploy it to the production server resulted in my application not working and spitting errors like "Trello.rf is not a function".

This problem is caused by ClojureScript munging names during compilation, which is disabled - as well as other optimizations - in the default dev profile. Because the idea of using an external JS library is not something completely stupid, there are some ways to avoid munging external names - outlined here and here. The approach with the externs file is quite easy to adopt, but it took me some experimenting to make it work, so I will describe it step-by-step here to have a reference in future.

Inputs first. I have a ClojureScript Om application enabled by cljsbuild 1.1.3. The compiled JavaScript is served from the /resources/public/js folder (or from somewhere under it). My application uses the Trello client JS library to make one call to the Trello.authorize method. The Trello library has to be brought in through the index.html like this:

<script src="https://api.trello.com/1/client.js?key=myappkey"></script>
<script src="js/compiled/trellodonelist.js" type="text/javascript"></script>

To prevent ClojureScript compiler from turning the names defined in the external library into strange symbols we only need to introduce an externs file and supply its name to the compiler. All this is done in three simple steps:

1. Create a plain JavaScript file and store it in a place where the compiler will be able to find it. I chose /resources/public/js/externs.js (the name doesn't matter that much).

2. In the externs.js file touch all the names from external libraries that you plan to refer in your app. For me that looked like:

var Trello = {};
Trello.authorize = function() {};

3. In your html refer the externs file along with the external library and your compiled ClojureScript:

<script src="js/externs.js" type="text/javascript"></script>
<script src="https://api.trello.com/1/client.js?key=myappkey"></script>
<script src="js/compiled/trellodonelist.js" type="text/javascript"></script>

4. In the project.clj file add the path to the externs file in a vector under [:cljsbuild :builds :app :compiler :externs]. You aim for something like this:

(defproject trellodonelist "0.1.0-SNAPSHOT"
  ; various meaningful things
  
  :cljsbuild 
   {:builds
    {:app
      {:source-paths ["src/cljs"]
       :compiler {:main trellodonelist.core
                  :asset-path "js/compiled/out"
                  :output-to "resources/public/js/compiled/trellodonelist.js"
                  :output-dir "resources/public/js/compiled/out"
                  :source-map-timestamp true

                  ; this one
                  :externs ["resources/public/js/externs.js"]}}}}

  ; other meaningful things
)

I had to play a bit with various locations of externs.js and missed one thing or another, but the above setup finally did the trick. Now, even if the ClojureScript is compiled with all the optimizations, the compiler is aware of the names brought in from the external libs and won't change them.

There are some other ways to make external names work and I recommend to read the articles mentioned above to get a better understanding of the reasons behind the issue and a wider set of alternative solutions. For my simple case the self-made externs file seemed the easiest and the most concise approach - maybe it will suit you as well. Happy coding!

воскресенье, 1 мая 2016 г.

Initiative

Some months ago we wanted to get a new developer on the team. My personal desire was to get him as soon as possible and of course I needed our recruiter's help with this. We had a short discussion with the lady about the kind of a person we'd like to hire and I was sure that we got on the same page and seeing the right candidate is only a matter of days. However, some weeks passed and I wasn't getting any resumés and not a single interview was scheduled. I was concerned almost to the point of going to the recruiter with the WTF?! expression in my face. Fortunately, though I chose to spend some 20 minutes writing a short description of two types of candidates that fitted my needs and emailing these to her.

What happened next surprised me a lot. Almost instantly I started getting a constant stream of CVs that matched my descriptions. It took us a couple days to schedule the first interview and only two or three weeks later (which means pretty soon in this context) we made an offer to a bright young fellow.

This story taught me a great lesson. I may believe that I have agreed on the goals and plans with someone, but if there is a slightest chance that doing some simple thing may help them get going I should do it without doubt and waiting. Spoken agreements made with your peers in a walkway are rarely clear and may easily get pushed away by newer and more comprehensible tasks coming from elsewhere. On the other side, taking some initiative to follow up and elaborate the problem may bring tremendous results.

Another example that I have on the same matter is concerned with an internal knowledgebase for developers. Despite having a vast code base rich with patterns and non-trivial solutions to various problems, we had limited guidance on this treasure and the reasoning behind its bits. Certainly our developers could benefit from a knowledgebase that would collect advice on various development questions.

I had a plenty of ideas on why we didn't have the knowledgebase, including the conspiracy theory that my other colleagues knew why such a thing wouldn't work. It turned out, though, that the only real reason for its absence was that being loaded with other tasks we simply didn't chose to set it up at some point. Thus, once a dedicated section was created in our internal wiki and declared a place to store all development-related knowledge, I started seeing different people contributing an article or two or simply voicing  support for the idea.

Starting things is difficult and if that's true for you, it's likely the same way for your friends and colleagues. Sometimes people lack a clear picture of the destination, in other cases they simply don't perceive the goal as important to anyone. No matter what's the core reason, just showing some gentle initiative may be enough to start the fire and get things going. It's only important to remember that initiative is not just about talking or thinking - it's all about acting, making the first step and showing the way.


воскресенье, 13 марта 2016 г.

Checklist as an Individual Efficiency Tool

Checklist is a tool that is widely used to facilitate completion of sophisticated tasks that involve many activities and actors. Software development teams benefit hugely from such things as release checklists (make sure that tests pass, release notes are ready, the build is uploaded somewhere, etc) and feature integration checklists (similarly, old tests pass, new integration tests are automated, documentation for the new functionality is written, the thing is accepted by QA, etc). The tools of this kind serve greatly to coordinate the actions performed by different persons or teams. At the same time, I believe that checklists also bear great power as an individual efficiency tool and that's where I didn't see them used much.

I first employeed a checklist for my own use when I tried to add some structure to my weekly task of analyzing team's performance and planning the next sprint. Even though I had a certain routine and even a quite stable form of the final result of this activity (a report and a list of Jira items for the next week), sometimes I forgot to check certain aspect of our performance and highlight it in the report or overlooked some items that needed action during the next week. Thus my key intent was to provide myself a list of things that I must do to consider both the report and the plan done. Additionally, I wanted to review the process of preparing the plans to exclude any obsolete steps. The results were great: once I put the new checklist into action it started saving me about an hour each week (half of it from mere thinking of "what else I forgot to include in the plan"). Additionally, after I defined this routine explicitly it became easier to think about it and spot other items, which don't bring much value or get ignored consistently - and to exclude or rework them, saving myself even more time and nerves.

There are several similar checklists in my toolbelt that facilitate other tasks and work great, but there is also a bit different thing that I consider my greatest finding - that's my daily checklist. The initial idea was to have a short list of things that I wanted to do daily and to go through it every evening before leaving the office. I quickly got used to the routine of tagging portions of job on the list as done at the end of the day and at some stage expanded it to ensure that I don't forget some other things - the benefits turned out to be tremendous.

First of all, making these routine checks helped me acquire the habits I wanted and somewhat refocus my work on the things I consider important. Additionally, since putting this checklist into action I had much less cases of forgetting to do certain things, because I get reminded of them before I go home. One change that I didn't expect is that this routine made me generally less nervous about the job: I don't have to rush into reviewing every new pull-request or responding an email from partner, because I know I will notice these things later: looking into both pull requests and mail inboxes is on the list, like other similar things. Thus the checklist makes me more efficient through decreasing the stress of urgency, serving as a reliable reminder and paving a short path to making myself more disciplined. In the end all this boils down to freeing my mind for more important tasks than just being afraid to skip something important.

Another powerful thing about this list of daily tasks is that it forces me to check some aspects of the team's current situation at the end of the day, which sometimes produces unexpected results. For example, I had several cases when this review allowed to spot urgent tasks and made me adjust plans. Furthermore, sometimes the nightly check made me notice a vague relation between a problem that we were dealing with during the day and a different problem that sit in our backlog waiting for us to find a way to address it. At least once such a discovery allowed our team to track down and fix a data corruption issue that bothered our partners a lot - and it could have evaded us again if I didn't perform the routine review of our status that evening.

To achieve these benefits one must have an efficient tool to implement the process. Ideally it should allow to manage checklists and to perform daily planning at the same place. Since the most common result of going through a checklist is adding something to the tomorrow to-do list and you want as little a gap between these two actions as possible. I use Trello agile boards for planning and to my great luck they support creating checklists inside cards and copying them freely (which also means that they can be included into the daily plan). If you didn't see the service yet, make sure to check it - that's a very handy tool! (Disclosure: even though I'd readily accept money, Trello doesn't pay me for this little advertisement.)



It's true that using checklists routinely - especially daily - may require some discipline and motivation. Thanks to the amazing 'To the Moon' talk by Russ Olsen, when going through my daily checklist I always feel acting like Neil Armstrong and Buzz Aldrin did before transmitting "Houston, Tranquility Base here, the Eagle has landed". It might seem childish, but it's fun to feel like you've just landed a moonship when you're simply leaving your office at the end of a long day - and that's a part of my motivation.

суббота, 27 февраля 2016 г.

Picking a simpler approach to aggregate data in Clojure

The only purpose that I use Clojure for now is talking to our Jira to extract some statistics for analysis and even this still brings a lot of opportunities for discoveries and revelations. The most recent task that I set for myself was to get data on bugfixing activities over some period of time and store it as a table for further analysis with Excel.

I wanted to transform a list of Jira items into a table that shows how many items of different severity each team member had fixed on a certain day. I already had a way to talk to Jira so the key part of the task was to aggregate the list into a table with 3 attributes and one numeric value - the count of items holding this combination of attributes. Aggregation is easily done with reduce so I only needed to chose the form of the result. My first natural response to a problem of this kind is to assemble a structure of nested dictionaries with values of attributes as keys and summed count of items as leaf values, something like this:

{ "2016-02-10"
    { "Ivan Petrov"  { "Major" 1 "Normal" 2 }
      "John Stone" { "Critical" 1 "Normal" 1 }}}

It turns out that Clojure 1.7.0 offers the update-in function that works greatly with nested maps. The thing takes the hashmap, a sequence of keys and a function. It would first retrieve the value currently stored in the nested map under the specified sequence of keys, apply the function to that value and store the result back under the same keys. Thus transformation of the list of items into an assembly of nested maps holding aggregated values will look like this:

(defn resolved-bugs-stats [issues]
    (reduce
        (fn [report {date :resolutiondate assignee :assignee severity :severity}]
            (update-in report [date assignee severity] (fnil inc 0)))
        {}
        issues))

This piece of code yields the figures that I want - the only step left is to transform it into a sequence of rows and this one took me much more thinking. Despite the fact that I was able to find a solution, I also realized that I don't need the nested structure at all. (That is availability of update-in turned out to be a misfortune).

The essence of my revelation was very simple: why would I build a map of maps of maps if I need a list in the end? Since Clojure lives great with vectors as keys I could just use a composite key and go with a one-level hashmap. This single level is still required if I want to have an easy way to sum up the count of items with certain values of attributes, but unlike a nested map it transforms very easy into a simple table. Here is the new function - it looks almost the same but produces a simpler result and, what's more important, makes the code that uses it way cleaner:

(defn resolved-bugs-stats [issues]
    (reduce
        (fn [report {date :resolutiondate assignee :assignee severity :severity}]
            (update report [date assignee severity] (fnil inc 0)))
        {}
        issues))

; generates a result in the form:
{
    ["2016-02-10" "Ivan Petrov"  "Major"]   1
    ["2016-02-10" "Ivan Petrov"  "Normal"]  2
    ["2016-02-10" "John Stone" "Critical"]  1
    ["2016-02-10" "John Stone" "Normal"]    1 }

While modern programming languages offer powerful tools to make complicated solutions real and cheap, there are few cases where the form of intermediate data structures need to be significantly more complex than the form of the desired result. Simply keeping this idea in mind and evaluating our solutions against it may help avoid some of the excessive complexity that we introduce when building systems.

воскресенье, 13 декабря 2015 г.

Focus

There was obviously little activity in this blog during the recent months and I feel great returning here. The blog always gave me an opportunity to step back and think over my actions and choices. However, this year I was so focused on a single thing - my day job - that only one post had made it here. Now I want to check with myself what this attitude gave me and what it stole from me.

Last February my priorities changed dramatically. In about two months after becoming a team-lead I discovered that whatever I was doing was not enough to do good with new duties. I was both overwhelmed with the tasks and problems that fell on our team and saw clearly that we don't perform as good as we could do. Part of my natural response to this discovery was to ramp up the amount of time that I dedicated to the job.

There are lots of reasons why I'm happy with this choice. Firstly, putting more effort into work made me learn a lot and build up new skills in the areas of my responsibility. Because it was my duty to process all the requests that arrived at our queue, I learnt to do this efficiently and obtained a lot of domain knowledge. In other words, it turned out that it's enough to push me into a partially familiar field and show no way out to make me learn it deeply. My expertise grew enormously over the last year and being really focused on the job helped here a lot (yes, there are lots of things in my field that I still don't know or understand not good enough - I'm speaking only of the delta). Moreover, spending a lot of time studying and resolving various issues - some being totally unclear at first - I not only learnt new areas of the domain, but also developed a skill of learning faster.

At the same time, through spending extra hours on duty I came to see clear that sometimes one cannot address problems by simply working harder. When you do a lot and it doesn't help, you start to see deficiencies and acknowledge the need for changes in one aspect of the job or another. A different edge of this same idea is that one cannot do everything on his own. No matter how hard you try, there is always less done than left to do. These discoveries, combined with high exposure to the issues that we face, constant analysis of our work and search for means to improve it - all of these being enabled by having extra time - certainly brought positive results and helped me develop myself both as a developer and as a manager.

On the other side, the same willingness to work more than 5x9 had a negative impact on the manager role of me, because having more time I could in many cases take the responsibility for any new urgent issue or task. While this attitude helped me tackle problems in time and develop my own skills, it hampered the team's collective progress. Any problem brings new knowledge and shows ways for development, so I simply stole a lot of opportunities from my fellow team-members and hampered knowledge distribution.

Generally speaking, while my attitude helped me grow in terms of knowledge, skills and career, it also shadowed both opportunities and problems. It is a very important lesson for me: whenever one decides to tackle a surge in the amount of tasks by throwing in more (his own) manforce, he misses an opportunity to find more intelligent ways to solve problems and to use them as the points of growth .

These are the effects of the extra work on my job, but there are also those that go beyond my office life. First of all, I left the attempts to do programming and software development in the outside - this means all side jobs and projects. While these used to bring new opportunities and ability to study unfamiliar areas, that's not something that I regret much. Being focused on one area allowed me to grow more than attempts to handle both the main job and one or two similarly looking activities would permit.

What I do regret is that leaving less time for myself I started to pay less attention to continued study and learning new things - both around software development and outside this field. I totally stopped exploration of new programming languages and tools and did much less learning in other areas than I used to - namely, I didn't complete a single online course over the recent months. That's certainly something that I shouldn't have forsaken:  these activities could both help me do my job better and make me expand my knowledge and interests.

Another thing that I dislike is that I quit writing for the blog. Here the reason is not only that the blog has become a silent and lonely place - it was never crowded here. The real problem is that writing less I lost the habit to deeply think over whatever I do and to search for interesting problems and irregularities in my activities. Of course, I do reflect on my decisions and actions, but without regular writing I hardly do it in a systematic and efficient way. Moreover, without posting to the blog I avoid sharing my ideas, which is not a big loss for society, but a shameful cowardice of me.

As with any choice, there do exist both positive and negative sides to putting most of your effort into one area of your life. I see these and despite the cons I am going to continue along the road that I started almost a year ago. However, even though I will continue to pursue that sweet feeling of exhaustion, I do need to make certain adjustments to address some of the problems. In particular I will certainly pay more attention to the supporting activities like learning and blogging - these allow to widen perspective and spot issues that stay invisible when you are constantly inside the problem-solving loop. At the same time, staying focused on my job I will have to pay more attention to the things that I am busy with. After all, being totally occupied with the day-to-day activities is a direct way to keep doing wrong things. So I only need more thinking, more analysis, more writing and the same amount of job. Good to know the solution to one's problems, huh?