jq – An open source implementation of q

During this lockdown I was due to take some holidays, originally to visit Pisa with Elaine. Instead of visiting Pisa I took a week off to code, for me it was just as much fun, I’m not sure Elaine agreed. This is the outcome of that week:


jq – http://timestored.com/jq/

So far it’s extremely limited, casting, parsing, list definitions and a handful of operations. It has however been insightful. The first 2 days were spent hashing out code to make the pure fundamentals work in any way possible. On the 4th day I began to realise some very verbosely implemented operations could be done in a much simpler way. Then I began to see such savings again and again. Perhaps after the first decade I would have it whittled down to Arthurs two-pager.

An inordinate amount of fun was had when I discovered I could host the application fully in browser  as doppio provides a method of running a full JVM:

jq Online Sandbox – http://timestored.com/jq/

So far it’s useful for basic snippets but I really think such a safe and easily launched environment would be great for onboarding new users to the language.

Quotes that float in my Mind

A colleague asked me to name books or articles that have influenced my thinking (Great question). However my mind works, I seem to store some ideas from multiple places but associated to short quotes. Here’s some of my favourite:

  • You only learn by listening, not by talking.
    • The best thing you can do to make a friend is to really listen to someone.


  • C.A.R.HoareThere are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.
  • Leslie Lamport“State the problem before describing the solution”
  • Often when dealing with messy code or 3rd party teams that aren’t delivering or need help, I try to think to myself “be the change” and try to make a difference. A quote commonly attributed to Gandhi. I had one colleague that would jibe back “Don’t be a hero” that the soldier with his head over the parapet gets shot. He felt triumphant when he realised Gandhi was assassinated. I still try to be the change 🙂
  • Tom Cargill – “The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.”. Why I have to take a deep breath when someone says oh that’s just one line of code, rarely if ever is it one line of code. Especially on a legacy system with no tests and a business that specifies unclear demands.
  • There are two types of companies/people. Those..”that sees quality and efficiency as opposing forces, or one that sees them as inseparable“. If you rush to push things out quick now, the overall delivery speed will be slower. – HN post.
  • Fred Brooks – “The bearing of a child takes nine months, no matter how many women are assigned.”
  • Donald Knuth – “premature optimization is the root of all evil (or at least most of it) in programming.” Until you have a benchmark showing where time is being taken and that this area is the constraint we will not be rewriting this currently simple clear code.
  • Brian Kernighan“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”
  • – “There is no silver bullet that’s going to fix that. No, we are going to have to use a lot of lead bullets.”
  • Martin Fowler – “If it hurts, do it more often”

Work in General

  • If you accept sh*t and shovel sh*t, don’t complain when people bring you more sh*t to shovel. You brought it on yourself. – James.
  • The best way to challenge something, may not be to oppose it but to ask “why” it’s being done.


  • The power of habit – “Practice makes Permanent” – Practice by itself will not make you better, you must reflect and get feedback on the practice you’ve done and consider how you can improve.

Java scratchpad / Snippet Runner

I quite often find myself wanting to write quick snippets of java. Similar to little bash scripts.

This usually means I have to:
– launch eclipse
– start a new project
– create a new class
– write public static void main
– try to remember why I started eclipse…..

Well no more. I decided we could do things better. I present:

JPadIt lets you run little snippets of java code and instantly see the result rendered (hopefully) beautifully as either html tables or in a console mode..

Let me know what you think? Do you find it useful? Can we make it better? Definitely!

Apologies but there’s lots of bugs and we need to improve result presentation)

Julia Programming Language

Recently I’ve been looking into Julia, below are my somewhat random jumbled thoughts so far. I’ve also put together a function listing for julia that should help beginners. I should mention upfront that I actually really like the idea of Julia and that it has a lot of promise.

1. The long time taken for packages/recompilation is annoying and it failed on me a few times.

2. It just annoys me that the index starts at 1 – different than most other languages.

3. One of the most successful modules in python is pandas. Why not take this further and make DataFrames a full part of julia i.e. Make nulls, dataframes and DataArray the builtin for all vectors. If julia is going to be for manipulating data, there are always going to be nulls. Why have the duplication between julia arrays and DataFrames, make them one and the same.

4.Has a number of builtin functions that duplicate functionality available in just as short a form:


They add little and pollute the global namespace. Ken Iverson gives a wonderful talk on reducing a language to the minimal core here:

5. There are many small bugs, inconsistencies and irritations such as:

Should super()/subtypes error on anything? because it does.

julia> super(bb)
ERROR: MethodError: `super` has no method matching super(::Regex)
julia> subtypes(bb)
ERROR: MethodError: `subtypes` has no method matching subtypes(::Regex)
julia> super("2")
ERROR: MethodError: `super` has no method matching super(::ASCIIString)

cd silently fails and isn’t cross platform.

julia> cd("/temp/")
julia> homedir()

It should at least throw an error. Even better would be if julia supported forward slashes regardless of platform. Ensuring julia scripts are cross platform.

run(pipe(`echo world` & `echo hello`, `sort`))

The run command is weird, not like other languages at all. It uses backtick to enclose the command, supposedly to allow easy copy pasting
but what’s the point of that when you cant use piping within a single command, so you will need to break out each command separately anyway. Rather than this string interpolation I could just have used join() myself and have less surprise. This interpolation will also possibly cause issues with user entered commands. (A bit more browsing reveals there is a significant number of people find this a more powerful technique)

Writing to a file isn’t visible. This is very non-intuitive:

julia> f = open("a.txt", "w")
IOStream(<file a.txt>)
julia> write(f, "hello!")
julia> readall(f)
julia> f = open("a.txt")
IOStream(<file a.txt>)
julia> readall(f)

If possible I’d prefer not to have to even specify “w”, rather when its used by a function,
depending if its read or write, those functions handle the underlying details.

Best Method of Creating Windows Installer for Java Programs.


  • Installer that adds link to start menu and file associations to windows
  • Running program appears as an exe and has its’ own file name in windows task manager
  • Use the built in JRE where possible or ask user to download if not available (bundling considered too large)

The finished installer can be seen at http://www.timestored.com/qStudio/


The best solution I found was launch4j and innosetup


Launch4j was the only wrapper I found that let me keep a jar and have multiple exe’s that pointed to the same jar. Allowing me to have two programs but not double the size. One config to keep in mind when doing this is to use the chdir option with . to ensure paths work out properly.


the tricky part with inno setup was getting the file associations to work. Here’s what I used:

Name: "{group}\qStudio"; Filename: "{app}\qStudio.exe"
Name: "{group}\sqlDashboards"; Filename: "{app}\sqlDashboards.exe"
Name: qAssociation; Description: "Associate .q extension"; GroupDescription: File extensions:
Name: kAssociation; Description: "Associate .k extension"; GroupDescription: File extensions:
Root: HKCR; Subkey: ".q"; ValueType: string; ValueName: ""; ValueData: "qKDBsource"; Flags: uninsdeletevalue; Tasks: qAssociation 
Root: HKCR; Subkey: ".k"; ValueType: string; ValueName: ""; ValueData: "qKDBsource"; Flags: uninsdeletevalue; Tasks: kAssociation 
Root: HKCR; Subkey: "qKDBsource"; ValueType: string; ValueName: ""; ValueData: "q KDB source"; Flags: uninsdeletekey;
Root: HKCR; Subkey: "qKDBsource\DefaultIcon"; ValueType: string; ValueName: ""; ValueData: "{app}\QFILE.ICO"
Root: HKCR; Subkey: "qKDBsource\shell\open\command"; ValueType: string; ValueName: ""; ValueData: """{app}\qStudio.exe"" ""%1"""

Combining into an ANT script

You want to

  1. Compile your JAR
  2. Create exes for both programs that point to the jar
  3. Place the exe’s into an installer

Like so (you need to install inno setup):

<launch4j configFile="src/main/resources/launch4j-qstudio.xml" />
<launch4j configFile="src/main/resources/launch4j-sqldashboards.xml" />
<exec dir="." executable="${INNO.SETUP.DIR}">
	<arg line="/cc '${basedir}\src\main\resources\windowsInstall.iss'" />
</exec >

Finished Installer

Can be found at http://www.timestored.com/qStudio/


Setup java speech jsapi using FreeTTS

Getting JSAPI setup to work using freetts seems to be quite difficult as seen by here, here, here…..

Turns out the install instructions were incomplete. The instructions were:

1. Go to the FreeTTS/lib directory
2. Type .\jsapi.exe
3. If the binary license agreement is acceptable, accept
it by clicking “I Agree”. The jsapi.jar file will be unpacked
and deposited into the lib directory.

What you actually want to do is:

  1. Download FreeTTS
  2. Unzip the freeTTS binary package and check inside the \lib directory for jsapi.exe
  3. Run Jsapi.exe, say yes, to unpack jsapi.jar
  4. Find your JRE directory, mine was C:\Program Files\Java\jdk1.6.0_03\jre\lib\ext
  5. Copy all the Jars (jsapi.jar, freetts.jar, cmu_time_awb.jar, cmu_us_kal.jar, etc.) to that directory
  6. Check in netbeans your projects properties ie right click on project->properties->libraries->manage platforms and the jars should be listed there. If they are not, restart and hopefully they should now be there.
    Netbeans Freetts Setup
  7. Copy speech.properties to the relevant folder. To find out where this is run the HelloWorld example that comes with FreeTTS. In my case the file was located in C:\Documents and Settings\Username\java.home\lib and contained the following
    # Modify this accordingly...

    Or you can set the property using

    System.setProperty("freetts.voices", "com.sun.speech.freetts.en.us.cmu_us_kal.KevinVoiceDirectory");

Any problems, post here and I will try to help you out.

Symfony sfGuard – Setting up users,groups,permissions

sfGuard is a Symfony plugin that implements a user management and login system for an application. It supports both groups and individual users… and it saves you from having to ‘roll your own’ user administration system. This guide assumes you have followed the steps given in the readme and that you now want to begin setting up users/permissions etc.

1. Create links in the backend menu to the user/group/permissions tables.

Edit apps/backend/templates/layout.php and add these items to the menu.

<li><?php echo link_to('Users', '@sf_guard_user') ?></li>
<li><?php echo link_to('Groups', '@sf_guard_group') ?></li>
<li><?php echo link_to('Permissions', '@sf_guard_permission') ?></li>

2. Create a login/logout link on the frontend.

Edit apps/frontend/templates/layout.php and add these items to the menu. (Notice the use of $sf_user in the templates.)

<?php if($sf_user->isAuthenticated()): ?>
   <ul><li><?php echo link_to('Logout', '/logout') ?></li></ul>
<?php else: ?>
   <ul><li><?php echo link_to('Login', '/login') ?></li></ul>
<?php endif; ?>

3. Create some users, groups, permissions

for us to play with using the backend. Create user->basicUser, group->basicGroup, permission->basicPermission. I will be using a basic setup where users always belong to a group and the group has permissions. I will not assigning permissions to individual users. therefore give basicGroup the basicPermission. and you will have something similar to this:

sfGuard user

4. Restricting access to certain modules/actions

Similar to how I never set individual permissions for one user I make it standard that I only ever set permissions using credentials. ie. In the application I never restrict security dependent on user or group id only on permission/credentials. This allows greater flexibility in the future. Note sfGuard gets confusing to some people because many documents talk about credentials, well basically credentials are what is called in sfGuard permissions.

If we have a module called “question”, inside of apps/frontend/modules/question we create a config folder and a new security.yml. Inside of apps/frontend/modules/question/config/security.yml we would have

    is_secure: on
    credentials: basicPermission

To set permissions on an action level we would have something similar to the following:

    is_secure: on
    credentials: basicPermission
    is_secure: off
    is_secure: on
    credentials: basicPermission

Part 2 will detail setting up user registration

Past Websites I Worked On

Some of the sites I have worked on are shown below:


MyLanguagePal.com – Online Language Swap

As part of a university project we created a website for users to exchange language skills. eg. A french person practises their english skills with an english person and vice versa. A number of new tools/challenges were faced. It was interesting to learn a PHP framework (symfony) that gave MVC separation. Working as a group meant my first real use of SVN. At last count the site had 267 registered users, WOOhoo!


Hong Kong – Student Affairs Office

While on IAESTE placement in Hong Kong for 3 months one of my tasks was a design overhaul of the student affairs office website and to use a CMS as the secretaries couldn’t edit HTML.


BBWI.net – small ISP

They requested a small “homely” design for their residential ISP. In the end they went with a more corporate professional design.


bKnowledgeable.com – Internet Startup

A website for those who seek knowledge. I never really understood the idea, but they knew the kind of design they wanted. Very little design work involved, was just a case of creating the HTML/CSS.


EnglishTalkOnline.com – University Project

As part of an entrepreneurial project we required a quick website to offer language tutoring. Lets just say we never made it rich.


ALTA – Adaptive Learning, Teaching and Assessment

I worked at ALTA professionally for a year. They had an entirely custom written piece of software that they saw as the future of testing children. It had some very interesting ideas, test questions vary slightly to prevent cheating, tests adapt to how well a pupil is doing etc. I got a lot of experience of MS-SQL and ASP programming, including bug finding. The picture shown here is the “messaging” system I created to allow teachers to message pupils.