Graeme's Meandering analysis Mon, 22 Jun 2015 10:25:24 +0000 en-US hourly 1 Taylor Swift on Apple Music – translated Mon, 22 Jun 2015 05:41:21 +0000 As Taylor Swift seems to have difficult saying what she means, so I thought I would provide a plain English translation of what she has to say about Apple Music.

Here is the original, but its all quoted below.

I write this to explain why I’ll be holding back my album, 1989, from the new streaming service, Apple Music. I feel this deserves an explanation because Apple has been and will continue to be one of my best partners in selling music and creating ways for me to connect with my fans.

I make lots of money from iTunes, and this could be profitable too, so I had better do enough grovelling first to let me change my mind if I have to.

I respect the company and the truly ingenious minds that have created a legacy based on innovation and pushing the right boundaries.

They are really good at marketing, just like me.

I’m sure you are aware that Apple Music will be offering a free 3 month trial to anyone who signs up for the service. I’m not sure you know that Apple Music will not be paying writers, producers, or artists for those three months.

It is shocking that anything new in the music industry does not make even more money out of fans.

I find it to be shocking, disappointing, and completely unlike this historically progressive and generous company.

Apple is really good at extracting money from consumers, so we want a slice.

This is not about me. Thankfully I am on my fifth album and can support myself, my band, crew, and entire management team by playing live shows.

This is entirely about me, but people are horribly unsympathetic when a super-rich 25 year old complains she is not making enough money.

This is about the new artist or band that has just released their first single and will not be paid for its success. This is about the young songwriter who just got his or her first cut and thought that the royalties from that would get them out of debt.

That is assuming that lots of people listen to them on Apple Music, and no one buys downloads or CDs or anything, and assuming unknowns people actually make lots of money from their first single. Its not like this is an industry where a few stars make all the money of something!

This is about the producer who works tirelessly to innovate and create,

A lot of my rich and successful friends and colleagues will not make enough money out of this either.

just like the innovators and creators at Apple are pioneering in their field…but will not get paid for a quarter of a year’s worth of plays on his or her songs.

I think a bit more sucking up here will make me sound better.

These are not the complaints of a spoiled, petulant child. These are the echoed sentiments of every artist, writer and producer in my social circles who are afraid to speak up publicly because we admire and respect Apple so much

These are the complaints of a spoiled, petulant bunch of rich people who want more money, but are scared of upsetting Apple too much.

We simply do not respect this particular call.

We will respect a call that makes us more money.

I realize that Apple is working towards a goal of paid streaming. I think that is beautiful progress.

It would be really great if people paid to listen to streaming services, and then paid again to buy downloads. No physical product, gross margin of almost 100%. That is what I call beautiful.

We know how astronomically successful Apple has been and we know that this incredible company has the money to pay artists, writers and producers for the 3 month trial period… even if it is free for the fans trying it out.

Apple needs this to work badly enough that my friends and I can squeeze some money out of them, even if they are not making any money out of it.

Three months is a long time to go unpaid, and it is unfair to ask anyone to work for nothing.

I want the money now.

I say this with love, reverence, and admiration for everything else Apple has done. I hope that soon I can join them in the progression towards a streaming model that seems fair to those who create this music. I think this could be the platform that gets it right.

Pay me from the start, then I’ll be happy.

But I say to Apple with all due respect, it’s not too late to change this policy and change the minds of those in the music industry who will be deeply and gravely affected by this.

We can squeeze pretty hard if we want to.

We don’t ask you for free iPhones. Please don’t ask us to provide you with our music for no compensation.

…because the cost of producing an iPhone is near zero just like the cost of producing a copy of music. I may use free software to run my website, but I need more money than a geek! Its not like anyone ever produced art or music without being assured of royalties first.




]]> 0
Beating Node.js with TCL Sun, 14 Jun 2015 20:58:43 +0000 This is partly a reaction to people who talk as if Node.js is unique, and partly to test my code against something that has seen production use. There are all sorts of problems with doing this sort of comparison, and while I would have liked to compare more servers, used a better environment, performance tuned everything, done more measurements etc. but I think what I have done is enough to prove my point.So just what is my point? That it is possible (in fact rather easy) to write a high performance event driven server in TCL. I also dislike server-side Javascript and really think the use case for things like Node should be cases where there is a lot of code that can usefully be shared between the client and the server (i.e. you have large chunks of the same code running on both), and not for other, to quote the Node website, “fast, scalable network applications”.

I did some very rough and simple testing. I ran both servers (Node and this)  and Apache Bench on my laptop. To ameliorate the effects of running everything locally, used taskset to run the servers being tested on one core, while Apache Bench ran on another. Of course I also shut down anything not needed during testing, and I am aware that by putting both on the same core I am assuming that both would use negligible CPU when not responding to requests (seems reasonable?).

Both servers ran a minimal “Hello World”. I was not familiar with Node but in many ways it was a fairer comparison than I expected: Node is pretty low level, relies on nested callbacks etc. Comparing Node to something like Tornado would be less fair. I used the version of Node that was in the Ubuntu 14.10 repos, so it is not the latest and greatest, but it is likely to be still in production use by many people. Anyway, it is new enough to compare with my code from 2008!

The code for the minimal TCL app:

source dandelion.tcl

proc hello_world {sock headers settings body} {
puts $sock [dict get $::dandelion::first_line 200]
puts $sock “Content-Type: text/plain\n”
puts $sock {Hello World!}
close $sock

::dandelion::init handler hello_world port 8081
vwait forever

The Node code is the HTTP hello world from How to Node.

I ran Apache Bench for 5,000 requests with varying levels of concurrency, (except for 2000 concurrent requests I raised it to 6000 requests). The graphs show results:

The x-axis of the second graph shows the natural log of the number of concurrent connections. The concurrency numbers are the same as in the first graph (1, 10, 100, 500, 1000 and 2000).

Dandelion, my light, embeddable, TCL server has higher performance at low load and performance declines more slowly. The identical performance with no concurrency probably reflects the performance of Apache Bench and the OS more than the servers.

If you think this may not be a consistent result, take a look at a graph of the worst performance from Dandelion against Node’s best:

I am fairly confident that that single point where Dandelion does worse is the result of a data entry error: the worst Dandelion results at that concurrency (1,000) is well below the range of the others (at any concurrency), and within the range of the Node results, and the best of the Node results at that concurrency looks suspiciously high too (not so clear cut because Node results varied more at the same concurrency, and there were better results at lower concurrencies).

So does this prove that everyone should be using TCL? No, Node wins when you need to share code with the front end, and there are event driven servers in many other languages. What it does prove is that there is room for TCL event driven servers, and it has satisfied me that Dandelion can perform well. I do no know how its performance compares to TCLHttpd or Wub, which are undoubtedly more sophisticated, let alone to all the other event driven web app servers around (Twisted, Tornado, Lighttpd (which is scriptable in Lua so could serve as an app server on these lines) etc.

It also may encourage me to dust off the Dandelion code, give it a decent name (I am terrible at coming up with names, so suggestions welcome), and put it in a repo (Fossil for TCL code?). Is there a use case for it (answers welcome)?

I also feel that TCL has missed an opportunity somewhere. Its creator was a proponent of event driven programming. It has had an event driven web and app server for at least 15 years but now, when everyone is using event driven internet servers (not just for the web) it is rarely used and few people even consider it.

]]> 21
Python IDEs part 4: Liclipse and PyCharm Wed, 06 May 2015 10:01:32 +0000 I have never liked the user interfaces of either Eclipse or Pycharm, so it is hard to be impartial. Liclipse, for those unfamiliar with it, is an Eclispe based IDE, that is a successor to Pydev. Both are proprietary, but prices are reasonable. After trying them again I still do not like Eclipse or Liclipse, but I do see the appeal of Pycharm.

The reasons I dislike these still apply: they feel less responsive than the other IDEs I have tried, and the UI is not particularly comfortable.

Pycharm does have a lot to like as well. It does a lot to help: for example it will warn you about modules that are imported that are not in your current virtualenv. It has good autocompletion, call tips, a debugger on par with those in Wing and Komodo. It also makes helpful suggestions: for example. it spotted a call to set() that could be replaced with a set literal. It also classifies warnings as weak or strong, and tells you how many there are of each in a file.

On the other hand I still have the UI, it feels sluggish  and it is a memory hog (600MB with a small project open). Given that other IDEs do what I want, I see no compelling reason to buy it.

Liclipse looks promising but us far weaker. Its auto-completion saw not good, and it seemed to have no particular strengths. Nothing in particular to recommend it as far as I am concerned.

]]> 1
Python IDEs part 3: Eric and Wing Mon, 27 Apr 2015 13:01:36 +0000 Continuing looking at Python IDEs, I have been trying Eric and Wing IDE. Both primarily Python IDEs (although Eric also supports Ruby), both are very powerful, but one is free and open source, while the other is proprietary and expensive.

I tried Eric5 as the recently released Eric6 has problems on Ubuntu 14.10. These can apparently be fixed by recompiling one of the QT libraries, but I am not going to got to that much trouble to try something.

Eric seems to have everything, even its own web browser. One may expect that a web browser bundled as a help viewer would be a minimal wrapper around a rendering engine, but the Eric developers think differently. It is a full featured web browser. The one thing it did not seem to want to do was actually display any help. This is a good example of what is wrong with Eric, lots of rough edges and no documentation. I have not been able to discover whether there is any way of setting per project virtual environments.

Eric has what look like a good debugger, but it is erratic. It showed useful data for an error, but I cannot persuade it to break at breakpoints. Wing IDE does exactly what I expect.

Both have nice code browsers, what show all modules, classes, functions etc. in a single tree.

Wing has one big advantage over everything else: its auto-complete is the best I have seen, and it seems able to get everything right.  Wing is expensive and there are no concessions for individual developers, something that both Komodo and Jebrains (Pycharm) offer, other than that prices are similar, but for freelancers and others paying for their own software, Wing is a lot more expensive.

Wing has some disadvantages. I find the UI cluttered compared to the clean look of Geany, Ninja or Komodo. Configuring a project for Django causes it to add TEMPLATE_DEBUG=True to the main settings file, which means manual fixes for some configurations. It is only an irritant, and it tells you its done it, but I dislike unasked auto-edits of code.

I also probably need to go back and compare debuggers as that seems to be a strong point of Wing IDE, and will be important to compare it fairly to Ninja and Python. its Django template debugging does not really seem to offer very much more than Django’s own eorror messages and seems rather less useful than the alternative error page in django_extensions, but for general Python debugging it seems pretty good.

If I narrowed down my choice to Wing and Eric, I would probably pick Wing in spite of the high price, because it works. Given the other options available, Wing’s price is a deterrent given that the only thing I really want from Wing that other IDEs cannot match is the auto-completion. Will getting a higher proportion of difficult auto-completions right boost my productivity?

Part 4: Liclipse and PyCharm

]]> 1
Python IDEs part two: Ninja and Komodo Thu, 23 Apr 2015 17:01:38 +0000 Continuing my review of a number of Python IDEs, I am starting with two IDEs I already know I like: Komodo and Ninja. As I said in the first post before I have used both Komodo Edit (Komodo IDE is Komodo Edit with extra features).I used both on a few things including a Django project, and an XML parsing script (using ElementTree). Komodo really shone on the HTML and Javascript editing. One disappointment was that navigating around a large XML file was not as easy as in Geany which shows CSS selectors in the code tree. Komodo just shows a blank. Its syntax checking for both HTML and CSS failed to detect several deliberate errors. On the other hand its Javascript syntax checking is outstanding and its auto-completion for for HTML, JS and CSS is good.

Ninja has very little support for web development apart from syntax colouring for Django tags. Even the browser preview just opens the file in the browsers, whereas both Komodo and Geany (with an extension) can open a preview URL.

Moving on to Python, I cannot decide whether Ninja or Komodo has better completion. Ninja correctly completes attributes (like .objects) or Django models imported into views, which Komodo does not. Komodo deals correctly with ElementTree imported as etree, which Ninja does not. Ninja fails to complete imported objects, but the Kai plugin fixes this After two days of side by side use, neither is obviously better, so its near enough a draw,

Komodo has calltips, Ninja does not. Komodo detects files changed on disk, Ninja does not. Ninja lints for Python 2 to 3 issues and suggests fixes that can be applied with a click, making it very easy to wrote Python 3 compatible Python 2, Komodo does not. Komodo has more plugins, Ninja has more Python specific plugins.

Komodo is heavier, using more RAM and CPU, but except for the times Komdo’s CPU usage really does up, both are fine on any reasonably modern machine – except when Komodo suddenly decides it wants to use 100% CPU.

Komodo supposedly has refactoring support, but it does not seem to do much more than Ninja’s find usages.

I also tried opening an old TCL project in Komodo. It does have code completion that works reasonably, and rather limited call tips, but it is not as good as I would have expected from a company so involved in TCL. However most TCL devs will probably by the Tcl Studio version, which has more TCL specific features, so that may be a lot better.

If I wanted a single IDE that works with multiple languages, or I did a lot of Javascript, I would probably buy Komodo. It is certainly far better than Ninja for a Django developer who does a lot of front end work

For a mostly Python developer, it is very difficult to pick a winner. On the whole I marginally prefer Ninja, and it is free and open source. I can see myself using Ninja for Python and Komdo Edit for HTML, JS and XML (I do not do enough of these to justify buying Komodo IDE).

Part 3: Eric and Wing

]]> 1
Trying Python IDEs Mon, 20 Apr 2015 10:34:19 +0000 I have been using Geany for a while, because it is lightweight and has a nice UI, and it will be my baseline for this comparison, but it lacks some features I would like to have. The most important is good auto-completion, but refactoring support would be nice and real time linting even nicer. So, I made myself a list of IDEs (and extensible editors) that met my criteria.

One note on open source vs proprietary and free vs paid. I prefer open source all things being equal, but if the proprietary product is better I will use it. This is particular true of development tools where all the data is text files so there is no real danger of vendor lock-in.

I am prepared to pay if it will improve my productivity.

What I knew before I started:

As I said, I have used Geany a lot, and I like it a lot: it is fast, lightweight, has reasonable syntax checking, has (not terribly effective) auto-completion, a very nice UI, and some very useful plugins. It also does a reasonable job of HTML, CSS and Javascript. I have it set up so it runs flake8 as a “build” command. The biggest shortcoming is the auto-completion, and I like it enough that I considered writing a better Python auto-complete plugin myself, and I may still do that.

I have also used Komdo Edit a fair bit, with some addons it makes a reasonable light IDE. It also has a UI I like, and it has good syntax checking and auto-completion for Python HTML, Django templates, XML, Javascript, CSS and supports a good many other languages (from what I hear, many equally well). I have not tried Komodo IDE before, but it is a full version of Komodo Edit, so it is familiar, just with a whole lot of extra features. I want to like Komodo IDE as I like Komodo Edit

I have tried Pycharm and do not feel comfortable with it. Just do not like the UI. It is a purely personal preference, but its not for me. The same goes for Eclipse, though maybe for Liclipse.

I know Vim with a bunch of plugins can be very powerful and productive. Like Komodo it supports a lot of languages (far more, in fact).

I have also use Ninja-IDE before. I like the UI, it has erratically good completion, but a lot of code checking – not just linting, but Python 3 compatibility as well. It has one huge flaw: it fails to detect and warn of files changes on disk.

The shortlist

I looked for IDEs that have the features that are missing from Geany, are reasonably popular, are able to handle multiple projects in different virtualenvs, are maintained, and run on Linux. This left me with:

Eclipse and Pycharm omitted because I dislike them. I would also like to try Liclipse (although I dislike Eclipse, Liclipse is supposed to be significantly different from Eclipse + PyDev). Editra omitted because I cannot get Python auto-completion to work.

I may also try Leo.

First impressions

Komodo IDE has all the strengths of Komodo Edit, but does not add a lot other than a better source tree and refactoring support. The refactoring in both Komodo and Wing IDE only seems to rename classes in the current file, which makes it fairly useless. I may be wrong and I will try to fix this problem: I know from past experience that things like this are sensitive to correct configuration. Ninja’s “find usages” seems to work better for me at the moment, although it does appear to only do a simple strong search.

Komodo and Ninja still have the best UIs, and I am comfortable with them as with Geany.

Ninja has lots of plugins, as does Komodo, but most Komodo plugins (sorry, “addons” as it follows Mozilla definitions as it uses the same platform) are yet to be updated to work with Komodo 9, and Ninja’s are only very briefly described or documented so they need to be installed to see what they really do.

I am also disappointed that a new stable version has not been released for 20 months, and this means a lot of important fixes (warning about changed files, for example!) are only in the “daily” version.

At the moment, I think that if I was using other languages it supported I would go for Komodo, but given that almost all my work is Python, I can afford Ninja is better. I will probably continue to use Komodo Edit for XML and Javascript, but that is really just light use.

My first impression of Eric is not good, but it may just be different.

The two most important things to do next are:

  1. putting in some effort into checking the Vim configuration is optimal, and spending enough time with it to learn at least the important short-cuts, and,
  2. checking Komodo and Wing project configuration is correct,
  3. trying Eric properly, and,
  4. installing Leo.

Part 2: Ninja and Komodo

]]> 3
Its not paradise, but lets not exaggerate Sun, 29 Mar 2015 10:35:15 +0000 I can understand Ankie Renique’s frustration about people who think that living is the tropics and working as a remote freelance is idyllic, but her reaction is to pick on a number of minor issues, or those that say more about her than the country, rather than picking very real problems.Firstly, if you keep meeting people who think that a third world country that only recently finished a thirty year long civil war is paradise, the problem is with the people you meet. I have to say that I have met very few people who think Sri Lanka is a paradise. Many who like it, but few are that deluded.

Her first complaint is power cuts, and the resulting interruptions to her work. In the third world its normal. Buy a generator, an in-car charger or an external battery pack.

Her next complaint was that the bank was closed though bother businesses were open. After 10 years she has not worked out why every calendar in the country shows three different types of holiday: “public” holidays for the state sector, “mercantile” holidays for the state sector, and “bank” holidays for … can you guess? It is archaic, but it has a certain inertia and it would be hard to change if it meant depriving some people of holidays they currently have. It is a nuisance, but it is hardly a major issue.

The next is a failure to understand how she was to behave at a wedding. Again, she does not know a fairly common custom (did I mention she has lived in the country for 10 years?) and is then embarrassed when someone stops her doing the wrong thing. It happens, and its mildly embarrassing. Big deal, I am sure similar things happen to anyone in an unfamiliar culture.

Next, she is patronising at being a guest of honour at the wedding of a couple she barely knows. I am guessing she is a lot more affluent (or perceived as more affluent) than the couple. This is common practice in Sri Lanka: you get the most important person you can. If you have lots of clout you get someone like the president or an ex-president. If you do not have money of influence you get a freelancer who seems more affluent than most people around. It is not a practice I like, but in essence its fairly harmless showing off no worse than a lot of things people do at weddings (and a lot less harmful than spending more than you can afford on them, which is also common).

She then complains that she has to pay more for entry at tourist attractions because she is white. Simply not true. She has to pay more because she is not a Sri Lankan citizen. There are white Sri Lankans (now very few after decades of mixing and emigration) and they will pay the Sri Lankan price. She could have applied for Sri Lankan citizenship if she wants to pay local rates. Contrary to what she says, compared to the privileges many countries give their citizens over resident foreigners, paying more at tourist attractions seems pretty trivial. If anything, Sri Lanka tends to privilege foreigners over its own citizens (see below).

She then talks about how unfair this is to foreign residents on local salaries who pay taxes. Where did she find those? The last European I met who claimed to be on a “local salary” turned out to be getting well over double the pay of Sri Lankans doing the same job in the same organisation. I did once meet someone who was on a genuinely local salary: a temporary job while setting down. In fact, foreigners in Sri Lanka almost always get paid a lot more than locals (usually getting the western salaries that the Sri Lankans who could have done the job have emigrated to get, plus an expat allowance). In fact, people are often paid expat salaries for jobs that it would be easy to hire a local for far more cheaply: apparently some organisation cannot find Sri Lankans to do IT jobs (you know the sort of thing that gets outsourced to South Asia…).  Some organisations even have set separate local and expat salary scales. All this would be illegal in most (if not all) developed countries.

She then objects to referred called a “suddhi” (white woman), but seems to fail to understand that this is not a derogatory term. Race in Sri Lanka is not even primarily determined by skin colour, and the adjectival form even used by some families as part of the nick-name for the lightest-skinned member.

Next comes her objection to being disturbed by an exorcism that she describes as a “Hindu religious ceremony”. I believe that there are such things, but given where she lives it may have been a Buddhist ceremony, or even witchcraft and not a religious ceremony at all. She has a general prejudice against religion, the strangest part of which she is that she describes herself as “a non-believing, non-practising Catholic”. I used to be one of those: I called it “being an agnostic”. The Dawkins like tone of her writing suggests that she is an atheist: was it really necessary to the phrase “loopy shit”? It is a bit strange if you are not used to it, and I do not know much about it, but I am sure you can find quite a lot of equally strange stuff in any country. I do not believe in it, so, apart from the inconvenience of it being noisy, I do not care one way or another if my neighbours do it.

Next she complains about gecko shit. This is an entirely valid complaint, if you think it important or likely to have much impact on your life.

Finally a long rant about belief in karma. She expects this to be controversial, but its an argument that lots of people have made before.


]]> 4
Open sources licenses still not understood (by Hiscox, at least) Sun, 29 Mar 2015 08:24:53 +0000 Insurance company Hiscox has posted a misleading article on “the advantages and risks of open source software” in its “small business knowledge centre”. It is not clear who the article is aimed at, the explanations of legal issues are unclear, and the discussion of security issues irrelevant. They could have saved themselves from looking foolish if they had asked some with some technical knowledge and familiarity with the licences to review the article.The first problem is that it appears to be aimed at people making the decision about whether or not to write proprietary software on an open source platform, or creating a proprietary fork (separately developed version) of existing open source software. However it explains:

Source code is the text commands that tell a software program what to do. Whereas software from the likes of Microsoft contains secret source code that is developed and maintained by the company’s own programmers, in open source code software the lines of code are open for every programmer to view, use, modify and improve.

All perfectly correct, but if a developer, or even a manager, of a small software business does not know what source code it, they have a serious problem.

It then goes on, quite correctly, to explain that open source software is covered by copyright, and license terms must be followed, but its explanation of the terms and the legal consequences is rather poor:

They stipulate that any software created from, or even containing, lines of their open source code must be made publicly available to all other developers.

This, in practice, means one of the GPL licences. This misleading as the criterion is not that the software “contains” lines of code. The commonest version of it is that:

any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof

This can included works that link to GPL libraries (although not if the library uses the LGPL variant), or that is for some reason a derived work (as defined by copyright law). That said, proprietary equivalents also have licences, that can be even more complex.

There are, at a conservative estimate, around 70 licences governing the use of open source code

True, but only a handful are widely used: versions two and three of the GPL license and its LGPL variant, the Apache license, the MIT and BSD licenses (which are very similar), and, perhaps, the Mozilla Public License.

It is also misleading about the legal consequences of a breach:

you may be forced to make freely available the software that you’ve slaved over. Although the licences may not actually preclude you from trying to sell that software also, if users know they can get it for free then they have much less incentive to buy it from you.

Not true, because you have other options. You have to stop distributing the software or open the source code or get a license for use in a proprietary application. You may have to pay damages for breach of copyright, but this rarely happens. If you can sell your software as web app (installed on your own servers) the GPL does not restrict your sales at all (because you are not distributing it outside your own organisation) unless it uses the fairly rare Affero GPL variant. You may be able to negotiate a commercial license from the copyright holder to use the open source code in your proprietary app (some open source developers base their business model on this).

The article then goes on to cover “virus problems”, but the author does not seem to know what a virus is. He refer to ” the recent Shellshock and Heartbleed viruses”. Neither Heartbleed nor Shellshock were viruses: they were vulnerabilities (holes that could be used to infect a system with a virus, among other ways of attacking a system).

If a bug in your software introduces a virus into your client’s systems then you’ll not only face acute professional embarrassment, but also a potential negligence lawsuit.

The first problem with this, is that it could happen with a proprietary library or platform as well. Unless you write everything from scratch, using no external libraries (which is usually impossible) whatever third part software may have security issues. Going by track records, using open source will reduce this risk.

The other problem is that the risk of a negligence lawsuit is tiny. Heart-bleed affected most of the world’s secure web servers (i.e. any page with an address starting “https” and showing that lock icon in web browser). It affected banking sites, and other things people depend on. There were similar bugs found in other major encryption libraries, both proprietary and open source (Apple’s, Microsoft’s, and Mozilla’s).  How many lawsuits resulted? The only one I am aware of was against the NSA who knew about the problem but failed to warn people, not against any software developer.

If there was a significant risk of being held responsible for security vulnerabilities, the software business would be very different, and several (if not most!) of the big software companies would have been sued into oblivion years ago. I am not suggesting that this would be a good thing, as strict security review for everything would make software development too expensive.

Similarly with Shell Shock, except that it is a less widespread thread. The ways I can think of it happening are an insecure server configuration (which can happen with proprietary software) together with user data passed to a newly started process (probably a CGI script started with Bash), or through failure to sanitize user input, or writing a CGI script in Bash (why would anyone do that?).

The useful information in this article can be summarised as “if you use open source libraries or code, check the licences, and, in particular, make sure you comply with GPL terms on distribution and Apache terms on patents”. That, and a link to the GPL FAQ would tell most people what they need to know.

]]> 0
Fossil vs Git Fri, 13 Mar 2015 12:33:43 +0000 I have used Fossil for version control for a few years, and I like it, but this recent comment on the fossil-users mailing list made me think about its limits:

we must agree Fossil [..] much easier and friendlier to use.

It is, and it is not. I do recommend it, and it does provide a a lot of functionality and it is easy to learn and use.

Why I like Fossil

Fossil wins hands down in terms of the time taken to learn the command line, and the ease of use of the command line. I like the work-flow Fossil encourages and have experienced very few problems after several years of use. I use Git because it is what all my clients choose. I do mean all: the only projects I use Fossil on are my own, and where the client leaves the choice of version control system entirely up to and I expect to work on it alone. Long term lock-in is not an issue as Fossil can export a repo to Git.

Fossil’s built-in web based GUI is also easier to use than Gitk. It does not do everything (for example, you cannot commit from the GUI) but given that this is a developer tool and the command line is easy to use this is not really a problem. It also means you can use the same GUI locally and on a server. Git is very powerful, but Fossil does what I want and more, and Git’s additional features often end up making it slower to use and creating more room for error — the need to stage changes is a good example of this.

In my view Git has two real advantages:

  1. Automatic detection of renames. Fossil can be awkward if, for example, you move a lot of files around, for get to rename in Fossil, and then run fossil addremove.
  2. The ability to completely roll back mistakes. Whether this is a good thing or not is a matter of opinion and Fossil is the way it is by design.

These advantages are outweighed by ease of use, and the convenience of having a ticket-tracker and wiki without any installation or integration work.

I am sure that Git has massive advantages for some people, particularly for large projects with huge numbers of collaborators. It was, after all, designed for the Linux kernel: approaching 18 million lines it is, as far as I know, the largest single computer program (in the sense of a single executable and excluding linked libraries), and requiring collaboration between a huge number of individuals in a huge number of organisations. For individual developers and small teams, I prefer Fossil

Git’s big advantage

Having said that Fossil is better, I now come to what changes everything: the tools and services available for Git. Installing Fossil with a wiki and an issue tracker is just a matter of installing Fossil. No configuration needed (except to set appropriate user permissions on a server). However, if you are using Git, you just register with Github (and pay if you need keep your repositories private), and you have much better issue tracking than Fossil (I suspect the wiki is better as well, but I have not used it). Fossil requires only simple server configuration, but Github requires none at all. Bitbucket provides a similar service, gives your private repositories free, and you can use Mercurial if you do not like Git. If you are working on open source software these services make it very easy for others to fork your code, work on it, and make pull requests.

If you do not like to use a repository controlled be someone else, you can use something like Gitlab, Gitblit, Trac or Redmine to provide issue tracking and wiki integrated with Git.  A bit more work to set up, and you lose “social coding” advantages of Github (or similar) but you still have an excellent wiki and issue tracker, and a lot of flexibility — Trac, in particular, has a lot of plugins.

Once you have any of the above running, the ease of use advantage lies with Git. have you ever tried to teach a non-geek to use Fossil tickets? I have, and its not easy. It cannot do email notifications easily and a lot of people rely of them, and that is probably enough to make Fossil’s issue tracking unusable in many scenarios.

On the desktop you have a choice of GUIs. I use Gitg, which is nicer to use than Fossil’s web interface, although it does seem to lack the ability to diff arbitrary commits. Apart from all the standalone GUIs there are plugins for file managers and GUIs.

Fossil’s wiki and tickets are stored in the repo and are distributed do they may still be preferable for people who spend a lot of time working offline. Other than that, Fossil’s advantages, other than the easier to use command line are pretty much negated.

Fossil’s developers claim that having everything in a single executable makes it easier to install. This may be true for operating systems like Windows that have primitive software installer (although, with Windows 8, even Windows has a “store”), but on (Ubuntu) Linux I just open the installer, type “git” into the search bar, tick “git” and “gitk”, and I have everything I need. If I want the latest version I am likely to need to compile Fossil, whereas I can install the latest version of Git from an Ubuntu PPA. On ease of installation and upgrading, there is no difference if you do not care about having the very latest version, and Git is easier if you do, at least if you are on a Unix family OS (including MacOS – I notice MacPorts currently has the latest version of Git).

I still prefer Fossil (just) because of its command line, and the simple work-flow it encourages (not requires). I also like the robust repository format (it is in a SQLite database, which is a very safe and well tested option), but deciding which is easier to use for a particular project is no obvious.

Other contenders

Of course Git and Fossil are not the only DVCSs in the world, still less are they the only VCSs. However, I prefer DVCSs (multiple copies of everything, means multiple backups of everything), and they are what I have real world experience of using. I strongly prefer open source (or at least easy export to a portable format) for any data of value (I have studied to much financial theory to ignore the option value in being able to switch). I want something actively developed, with a community and tools around it. The only real options, apart from Fossil and Git, seem to be Mercurial and Bazaar. Mercurial seems to be more widely used, but both have decent tool integration. Both seem to be a reasonable compromise between Fossil and Git, with a more user friendly command line than Git, but more of an ecosystem than Fossil. Bazaar development seems to have stagnated and it is the less popular of the two, so Mercurial seems a safer choice. More active development means that even the (contested) advantages Bazaar claims over mercurial are likely to be transient.

Both Mercurial and Bazaar are written in Python (which is good, as I am not any good at C), but Mercurial was chosen to develop Python itself in, partly because it is what Python developers (meaning the smart people who develop the language itself) prefer Mercurial. I currently prefer Fossil for my own projects, but I definitely want to try Mercurial.

]]> 3
Why you should not use WordPress Mon, 06 Oct 2014 12:24:01 +0000 The appeal of WordPress is obvious: cheap and easy, and lots of “developers” know it. The biggest problem with WordPress is that something originally designed as a blog platform, has evolved general CMS features, and is widely used a development platform. The problems are that it has security issues, and is neither flexible nor productive when used a a development platform and cheap developers are not good developers.WordPress is a very easy way to get a site up and running, and I do use it myself for small sites – even this blog still runs WordPress (although it may change). So what is wrong with WordPress?


WordPress has a terrible track record, and WordPress sites are the most frequently targeted by attackers. There have been massive automated attacks mounted on millions of WordPress sites simultaneously. WordPress sites are easy for attacking web crawlers to identify without human intervention so the cost of attack is very low on a per site basis.

WordPress advocates will point to improvements in WordPress itself, and claim it is a lot more secure than it used to be. However, it still suffers from bad design, and a poor attitude – for example, WordPress developers actually encourage letting WordPress alter its own files because they is how their update mechanism works.

Even if WordPress is now adequately secure, even its defenders will usually concede that there are many insecure themes an plugins. As the attraction of WordPress is its wide range of themes and plugins this leaves you with limited choices:

  1. Use WordPress only with bundled themes and plugins. This leaves little reason to use WordPress at all.
  2. Security audit the themes and plugins you use: suddenly its not so cheap or fast to set up a WordPress site.
  3. Develop all themes and plugin especially for your site: this makes WordPress an expensive solution if your site requires any real customisation.
  4. Only use WordPress if you are sure it will do what you want with minimal customisation and only themes and plugins you are sure are secure.
  5. Risk it!

Even being this careful, you still have to deal with the fact that a WordPress site is far more likely to be attacked. Even an unsuccessful attack can cause problems (slower site performance, bandwidth consumption from repeated attacks etc.). Certainly, you can (doing a bit more work) solve these problems, but even then any security hole in WordPress is more likely to lead lead to a breach because it is a popular target for automated attacks.

It is not a framework

People use WordPress for all kinds of websites, and even something that are better described as web apps. They have a bad case of “when all you have is a hammer, everything looks like nail”. Anyone who has used a proper framework will tell you that developing a custom site using WordPress is a far longer and less productive process than using framework like Django, Ruby on Rails, or Symfony. If you are a developers, compare this to this, or even better, this.

Similarly frameworks provide productive ways of generating forms that match database tables, or creating a custom admin interface. The work required to deliver the same custom functionality is dramatically lower.

A (good) framework also takes care of a lot of security issues for you — Django, my favourite framework, generates properly escaped database queries by default, and requires CSRF protection on forms by default, etc.

It uses PHP

PHP is a horrible language, and PHP code is harder to maintain. A developer should remember that there are always a minimum of two developers working on any project — you, and you a few months later, and the other one is an idiot. Hard to read and understand PHP code will create more work later. This is just one of PHP’s many, many faults, which are brilliantly covered in depth in A Fractal of Bad Design.

Again, the end result is to make development slower and maintenance more expensive.

PHP advocates like to point to the many large sites like Facebook that use PHP. PHP has worked so well for Facebook that they have resorted to forking it to address some of its more egregious flaws, but they are still constrained by the need for compatibility so it is still far from being a good language. Facebook also uses many other languages: they have released open source components in Ocaml, Java, C++. D, Haskell and more so they presumably use all those somewhere.

Lots of developers, does not mean lots of good developers

A lot of people like the fact that there are more WordPress developers than those working on any other web platform, many working at cheap rates. The reason for this is that it has a lower barrier to entry: it is easy to learn enough to install WordPress and some plugins, perhaps learn a bit of PHP for simple customisation… now you are a developer!

Of course there are plenty of good PHP developers, but I very much doubt there are more good PHP developers than Python or Ruby web developers. On top of that, good PHP developers are unlikely to be pitching for WordPress work and are probably focusing on more skilled stuff using frameworks. If you want cheap, WordPress is for you, but remember who works for peanuts.

What WordPress is good for

I still run my blog on WordPress, and I have used it for some small sites. It is a very capable blog platform, and is acceptable for small brochure type sites. If you are sure that is all your need, and all you are are likely to need, and have a limited budget, then maybe WordPress is a good option, but:

  1. Ensure you install is security hardened: there is a lot you can do to make it harder for automated attacks to identify your site as WordPress based and make sure it is regularly updated. I will be blogging about that soon.
  2. Use a host that gives you ssh access so that you can use wp-cli to run updates: otherwise you will either have to use the horribly insecure built in update mechanism, or update manually.
  3. You have confidence in the security of any plugins and themes you use.
  4. Pay for a decent developer. No one with real skills is going to work for £5/hour, even in a low cost country.

If it is so bad, why is it popular?

It is not a case of it being popular despite it being bad, but it being bad because it is popular. As long as WordPress was used for what it is best at (a blogging platform) it was fine (apart from some security issues). Even extending its usage to personal and small websites was fine. The problems are:

  1. It is being used for things it was nor originally intended for,
  2. It is so hugely popular that it is by far the most popular target for automated attacks.

The first of these causes a productivity problem: it is better to use the right tool for the job. The second means that security problems are far more likely to be found (by the bad guys) and exploited. I am talking here about automated attacks that attack thousands, or even hundreds of millions, of sites, in the hope of finding some that have a particular weakness. From the point of view of someone running one of those attacks, WordPress is probably the most attractive single target because it is popular,

I am not dead set against WordPress, but I am looking for alternatives, especially for small non-blog sites (perhaps for blogs as well) so please look out blog posts on how that goes.

]]> 0