Do you have IDEA?

Prelude

This post is intended as an introduction to why I love working with IntelliJ IDEA. If you haven't heard of it yet, it is an IDE, just like Eclipse or NetBeans, the only material difference being that it has an edition that costs money (but you can get a free open source license).

During this "review" I will be using the following notation to present my pros and cons:

This is something I like a lot.

This is something I don't like.

Pricing

After I have been using it for a few months I can say that it seems that being an essentially non-free product is a good thing: it feels that they are actively seeking to deliver better and more comfortable usage. It feels that their team doesn't just put in features, they try to do it in a way that is natural and easy-to-use and cool.

Disclaimer

These are just some of the experiences I noticed while coding. IDEA offers a lot more little "wow that was easy" or "wow that was clever" feelings, whenever it does something that feels just right in that situation. Those can only be experienced while watching someone working in IDEA, or better yet, trying it out for a few months.

I will probably post updates with new things in the future -- I will try to document all these moments when they occur.

Mail must be sent a day before commit

A friend of mine has just received a mail, that contains instructions on the project his team is working on (excerpt):

• Every change to XML/XSD/XSL must be reviewed by the (censored) review board group.
• [...] A change in XML needs to be announced to the (censored) mail distribution list. This distribution list does not exist in the exchange server yet so you can send me the email and I can forward it for the moment.
• Ideally a mail should be sent at the start of the work with the expected commit date.
• A mail must be sent a day before commit.

I'm sorry. I understand the need of control, but this is a seriously fucked up way to do it. What I think is:

• This is a distributed project with a lot of teams depending on a central XML/XSD.
• There are no branches, it is a single track development (in fact I know that it is hosted in SVN despite being such a huge thing).
• There are no tests, therefore no continuous integration.
• Instead of fixing things, more bureaucratic weight is placed on the developers to create an illusion of control.
• There isn't a mechanism to easily and effectively distribute change notices through teams.
• The authors are basically stating that developers do not have the necessary knowledge to propose changes in the XMLs.
• This leads to the conclusion of no documentation, from which the developers would learn.

So what

Developers are best at hacking away at code and providing solutions to the requirements. It's their job! They love their job! (Some even grow facial hair because they would rather hack at code than play with sharp blades.)

This kind of control will only throw them off balance, making it impractical to even suggest changes that may make the whole project easier.

Practically the mail makes the XML/XSD/XSL in the system a fixed fact ("Don't touch it, you!"). No one but the bravest will try to fiddle with it, therefore solutions will pop up that go around the problem. They may not be big things -- only there's a possibility that out of two solutions, the better will be dropped.

There still won't be tests -- so we now have hacks in the code, but the management is confident because nothing changed in the critical area.

Commit control?

Tasks that involve adding a single optional attribute that would take anyone 25 minutes will suddenly be days. First the approval, then the announcement, then the "will commit" ping, then 24 hours later, the commit. Changes will be batched, it will be unclear if they're logically grouped or it just happened that way.

Sending a mail that the change will be committed 24 hours later creates a tension that practically will manifest as: done → send mail → commit tomorrow. Bam, we have a 24h delay. Not good for anyone.

Conclusion

I may overreact to this; I can easily imagine a situation that such rigorous verification is necessary. Only that it should be supported via the build and continuous integration, not via pre-commit emails, not via restrictions on design feedback from developers. This is too much of a cost, and ultimately the product has a good chance to be the loser of this.

But soft! what build on yonder server trots?

I am seriously loving Gradle after the initial 15 minutes of confused searching. No, really, if you are coming from the Java side, consider reading this page first: Gradle Java plugin, or be lost for quite some time like me. And you will probably want to run

gradle test

first.

What I wanted

I wanted something that used Maven's repository for fetching dependent open source libraries, while not forcing Maven's heavyweight things on me.

What I got from Gradle

Consider the following snippet: What did the author want?

• Oh hey, this bloke wants Java. Let's "apply" it, whatever that means.
  (Look, this stuff is plugin based!)
• He wanted to use Maven central as a repository.
  (You can stop with Maven right there, thank you!)
• He described his dependencies, 1 for compilation, 3 for test compilation and 1 for running his tests.
  (How do I configure how targets are related?...)
• Define the source set paths, for main java, test java and test resources.
  (Sounds Mavenish, am I not going to be free after all?)

What it really is

That snippet is i7m-common's complete build description.

As far as I can see now, Gradle has certain perks:

• Supported by travis-ci!
• No boilerplate Ant build.xml, no init, compile, copy, test tasks, etc.
• If you're building a simple Java project with the same structure convention as Maven, you only need the first line to get a full, working build that can also run the tests and create JAR packages.
• If you want to use a lot of open source thingy, you can easily manage dependencies without having to commit them to your source control. (Of course, being in an IDE is an entirely different question, you will still want the libraries, or get a Gradle plugin.)
• If you want to use a different structure, just set sourceSets like above, and carry on.
• You don't have to worry about target relationships, because Gradle has a built-in "default" which is OK for most of the Java projects. And if someone needs more, the configuration is as easy as the sourceSets above.

Closing thoughts

I am sure that it has its downsides but right now I'm pretty happy that I did not have to litter build.xmls into my projects.

PS. I stole the title from Why did the chicken cross the road? (Shakespeare), which of course was based on Romeo and Juliet.

Mind the gap

This is probably the most blogged topic ever, but let me try to reiterate it.

I find it interesting that some of the companies or recruiters I had to work with actually place more emphasis on the claimed skills of the individual, rather than the individual's mindset.

Why is the mindset important?

Going back a few years, I was actively interviewing candidates for Java Developer positions. After the initial round we dropped only the ones who couldn't talk at all. I realize that everyone can have a bad day, but those people were actually unable to communicate.

In the second round, they had to solve a problem in front of the computer, without any time limit or restrictions. They were free to ask questions, search on the web. The task was not a hard one: it involved writing a servlet and a JSP. Basically the candidate passed, if he was able to Google "servlet jsp tutorial". The results were astounding:

• majority of the candidates have given up after 5 minutes;
• most of the rest were unable to find the solution on the web;
• some of the rest were able to solve it.

The demography of the results were also very surprising to me:

• fresh graduates who worked on thesis related to the field were unable to solve it;
• people with years of experience were unable to solve it;
• usually random people without proven experience or degree could solve it.

Going back to the mindset

The fizzbuzz example is a similar problem; people with degrees or experience can't solve a problem that involves simple concepts.

And I think it is not the degree or experience that corrupts people, it is the mindset of the people involved (being affected by the other people at the company or university/college).

I have seen people get high and proud doing their thing at a company, and it is fine to a degree. The danger rears its head when these people want to move on and realize that there is a whole different world out there, with acronyms they never heard before. And even for a J2EE developer, it is very unlikely to find a job that requires to do the same thing as before.

Therefore it is necessary have a mindset that embraces new things, is open, receptive and pragmatic. This is sometimes visible in job posts like this Atlassian one.

Going back

I come back to my first statement: I feel that some companies I interview with in Hungary are not actually interested in my mindset, they are interested at static evidence that may or may not mean what they think. I feel that it won't make that much difference to a team if I am the best EJB guru ever born (I am not). I feel that it is more helpful to have someone who can see the concepts and intents of a system. If that is given, I don't feel that it matters what technology we are talking about, since technology can be learned (fast).

I am not saying that companies should not require proven experience from candidates in areas they will work with; I am saying that the focus should not be exclusively in that area.

I feel that these companies would be much more productive if they would hire people who are open to new things. Rands called them volatiles.

I feel like I have to emphasize that I don't think people with proven static experiences can't be volatile or pragmatic. I just think that the recruitment process should assess those qualities as well.

jruby-18mode and 1.9 hash syntax

I have this pet sandbox project named kandic where I try myself at Ruby BDD while creating something worthwhile. I hooked up with travis and codeclimate for good measure too. Travis is set up to test against 4 different RVMs: 1.8.7, 1.9.3, jruby-18mode, jruby-19mode.

And here's the funny thing: as it turns out, the 1.9 hash syntax that was in the code that time properly creates parse errors on 1.8.7, but jruby-18mode can parse it.

I had the same problem with minitest dependency later, the jruby-18mode was happily crunching away.

Just a few things to watch out for if the target is 1.8, I guess.

Lost project found

This was before the day I used any version control, and I have to admit I have totally forgotten about this.

It's my "let's make a hungarian-japanese online kanji reference based on the book I bought" project: Kanji referencia.

First time HTTP servers

There are two things you will need to watch out for if you are to write your own small HTTP server for the first time:

• The request ends with a line containing exactly

  \r\n

  If you go on and read again, the read will block because nothing new will come. This can be conveniently checked using a BufferedReader.
• The minimal response you will need to send is

  output.print("HTTP/1.1 200 OK\r\n");
  output.print("\r\n");
  output.print(response);
  output.flush();
  output.close();
  

  A PrintWriter is a good friend here. The "OK" after the response code is not required, but should be written out, since HTTP 1.1 RFC tells us so:

  Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF
  

Incompatible technical debt cutting

I really like the following statement:

This process [changing the aggression system] is necessarily imperfect, and we're inevitably going to find that some things are slightly easier and other things are slightly harder. This is regrettable, but we're absolutely not going back down the rabbit hole and re-complicating the new system to solve comparatively minor imbalances, because the overall cost is just not worth it.

Source: Introducing the new and improved crimewatch, EVE Online's developer blogs.

I like it because it involves:

• Cutting back on technical debt;
• There's a realization here that there's no need to keep a suboptimal implementation alive, even if it means a change for the customers too -- if the customer (and/or the dev team) essentially gets something better in return;
• No fear! I experienced a lot of fear and uncertainty involving changing existing code, even if it had a good test coverage with good quality tests. They make the project rigid and feared. They certainly overcame that.

Bottom line is I wish more companies would think like that.

Sharp tools are locked down ...

... because no one likes the children in pieces.

Powermock can be useful in trying to handle legacy code, but I think our team made a big mistake by promoting it in an ongoing fresh development.

It allows us to just ignore certain obvious design problems. If we cannot test a code without major headaches, then we have an architectural problem that needs dire attention. Solving the test won't make the problem go away, and we'll have to face it again, possibly with smaller timeframe and a much bigger mess.

To reiterate, doing this in legacy code is fine by me. Doing it in code that we can actually shape and refactor easily -- and is not final or designed in a way that usually prevents testability -- is a major risk and the question "How can we fix this at the source?" is a way better solution than relying on powerful tools to do it for us.

Let it be known

Judge a production code project by its naming. We got the following exception in a 3rd party library we have to work with:

com.[snip].core.HolyGrailException: Error during JMS initialization.

Smooth zoom animation in HTML5

I wanted to insert some nice-looking animation into my HTML5 project when zooming and scrolling. I'm not a math expert, so here's my stab at it.

I decided I'll try an "expected result over time function": I wanted to have something that starts up gradually and slows down when approaching the end of the allowed timeframe.

• Find a function that in the range of [0, f] will have the result of [0, 1].
• Steepness and duration should be part of the function.
• It must be a sigmoid function.

Wolfram Alpha is good resource to start looking when you're not exactly a math expert (I am not...). I ended up with solve coth(s)*tanh((s/f)n) = 1 where s = 4 and f = 120 as a working solution.

• s is the steepness factor;
• f is the total number of frames for the animation;
• n is the frames elapsed since the start of the animation.

The (draft of the) generalized diff function is:

var ValueDiffFunction = function(start_value, target_value, start_frame, target_frame) {
  this.start_value = start_value;
  this.target_value = target_value;
  this.start_frame = start_frame;
  this.target_frame = target_frame;
};
ValueDiffFunction.prototype.next_value = function(value, frame) {
  if (this.target_value == null) {
    return value;
  }
  var result = value;
  if (frame >= this.target_frame) {
    result = this.target_value;
  } else {
    var total_value_diff = this.target_value - this.start_value;
    var total_frames = this.target_frame - this.start_frame;
    var n = frame - this.start_frame;

    var steepness = 4;
    var functionresult = coth(steepness) * tanh((steepness / total_frames) * n);
    result = this.start_value + total_value_diff * functionresult;
  }
  return result;
};

Create toolkits, not drones

Frustration

I tend to claw my face off when I need to do something tedious. When I'm presented with a task that would take half minute with an ideal solution, but takes 30 minutes with crosschecking textual data. I hate to do irrelevant stuff.

The example I will be telling was how we created a toolkit for our project.

Workflow

Our workflow was a bit of a myth. It had to be identified; not everyone followed the same patterns, but overall, the following workflow became visible:

1. create initial dataset
2. deploy and/or run our application
3. analyze generated data
4. optional: remove generated data
5. optional: goto step 2
6. remove initial dataset

Sometimes we would keep deploying and testing with the same initial dataset, cleaning only the generated data.

I'll now describe each step in detail, in the order they became a problem. We were promised there'd be a tool that takes care of the initial dataset problem, so I won't be talking about steps 1 and 6.

#4: Remove generated data

Identify the generated data in the database and remove only the data that was generated. The database is essentially a tree-like structure with "Key=Id" values in leaves (not XML). Delete only given keys with an id regexp for each key that matches only generated leaves.

It was fairly obvious we have to implement a script that deletes these leaves, or we can't test without resetting the whole database. Initial implementation was hard to configure and slow: it started a new session for each operation, which is very slow for this database. This solution was used for lack of a better version, and neglected to be updated, since "we don't have time for that".

This was of course utter bullshit. We were wasting tens of minutes on a slow script for days, instead of investing 1 day for 1 programmer.
Sure, I had to dig into the API of the database for how to do everything in one session and one transaction, but it's now common knowledge (it was also used in the initial dataset scripts to make them faster).
Sure, I had to invent a whole new configuration scheme, and write a couple of ruby classes.

The result: it was half a day, it now runs under seconds, and adding a new key with a new id regexp is about 3 lines.

#5: Goto step 2

As easy as this sounds, we had to use a GUI tool (external product that used our service) to call our application. Instead of simple SSH, we had to use graphical sessions and using products that integrated with our solution. It was slow.

CLI client that would call our service was dropped first, but put back when the GUI tool was late, as a necessary evil.

The result: everything can be automated from scripts, without using graphical terminals.

#3: Analyze generated data

I wasn't aware of this problem until I had to do some testing much later in the project. I was horrified by the amount of monkey-work that needed to be done to test the result. And no one was crying for help from the test team!

This was on Friday. I was so frustrated that I came in Saturday, and created a visual representation of the result using graphs. I could reuse most of the ruby classes from step 4. Graphviz is awesome, and easy to generate input for it.

The result: validating a result is now about 1,5 minutes. Product managers from across the globe actually want the solution to be productified because it actually represents the data in a friendly, colourful way.

#2: Deploy and/or run our application

We actually made this part of the normal development process, but there was a time when the only way to deploy something was to build an official package, scp it to the test server, install and start manually.

Being too lazy, we made a build target that created the necessary JARs instead of a full build, copied them to the server configured, and restarted the service remotely, if needed.

The result: deploying an application is a double-click from Eclipse, and is up-and-running in about 40 seconds if a full restart is needed.

Conclusion

• Investing in toolkit scripts resulted in a far more quicker, "agile" workflow.
• The scripts reduced our outside dependencies and enabled automation to an extreme level.
• It was not part our normal planning process to evaluate potential toolkit components, but since everyone saw how it worked, now it is.

Overall, doing something not visible to the stakeholder is always worth the extra few man-hours, if the overall process will be faster and easier to use, with better quality. The problem is, most companies and customers only want what is immediate value to them, ignoring everything else. Our application now has the lowest number of bugs ever and didn't miss the deadline.