A Christmas present from the monks

After what seems an age I finally got promotion to Prior, next stop (in 3000 XP points time) is Monsignor. My best node was Unexpected Output with a score of (strangely) 42.

It's weird, I get such a buzz from helping people with Perl it is almost adictive.

Python msvcrt.locking trials and tribulations

It should be easy. All I wanted to do is to replace my C demos which use flock with a nice homely Python version on Windows. I wanted to stick with the standard library so I could just slot it into the existsing course (so I couldn't use the Locking module on PyPi).

A simple demo to start with:
1. Process 1: lock a region, write a record, pause
2. Process 2: attempt to lock the same region
3. Allow Process 1 to release the lock
4. Process 2 continues

Converting the code was easy (I thought), but no matter what I did I always found the region was locked for the second process - the lock was not being released.

I blame the documentation, and possibly the implementation.

What is not obvious is that the current file position must be reset to the original locking position before the lock gets released. The write (of course) advances the current file position, so by the time we do the unlock we are unlocking a region for which we don't have the lock in the first place! Wouldn't it be better if msvcrt.locking returned a lock object, saving the file position? Anyway, here is my completed demo:

"""
   lock_w.py

   Run two copies, each in its own terminal session.
   Allow one to write a number of records, then switch
   to the other showing that it blocks on the same record.
   Switch back and release the lock, and show that the
   blocked process proceeds.

   Also run with lock_r to demonstrate interaction with
   read locks.

   Default filename is rlock.dat

   Clive Darke QA
"""

import msvcrt
import os
import sys
import time
from datetime import datetime

REC_LIM   = 20

if len(sys.argv) < 2:
    pFilename = "rlock.dat"
else:
    pFilename = sys.argv[1]

fh = open(pFilename, "w")

# Do only once
Record = dict()
Record['Pid'] = os.getpid()

for i in range(REC_LIM):
    Record['Text'] = "Record number %02d" % i
    Record['Time'] = datetime.now().strftime("%H:%M:%S")

    # Get the size of the area to be locked
    line = str(Record) + "\n"
    
    # Get the current start position
    start_pos  = fh.tell()          
    
    print "Getting lock for",Record['Text']
    msvcrt.locking(fh.fileno(), msvcrt.LK_RLCK, len(line)+1)
    fh.write(line)       # This advances the current position
    
    raw_input("Written record %02d, clear the lock?" % i)
    
    # Save the end position
    end_pos = fh.tell()
    
    # Reset the current position before releasing the lock
    fh.seek(start_pos)
    msvcrt.locking(fh.fileno(), msvcrt.LK_UNLCK, len(line)+1)
    
    # Go back to the end of the written record
    fh.seek(end_pos)
    
    print      

fh.close()

Update

I try to keep these posts technical, but I have not blogged in a while so I'll break with tradition.

I have been teaching a lot of Python. Mostly this has been Python 2, but some clients want Py3. Unfortunately not everyone understands how Open Source language releases work, and some internally assumed that no-one would want Python 2 training as soon as Python 3 was released.

I have to confess that, having done a lot of Python recently, moving back to Perl is rather a drudge. Python is not perfect, but I now find the syntax of Perl unnecessarily fussy, and Perl 6 is even worse. I mean fussy in the sense that Perl code is to Python as a doily is to a beer-mat.

Talking of Perl 6, with the release of Rakudo Star I thought (hoped) that we would get a flood of requests for courses, but we have not had one. Rakudo Star is, I guess, still too early, and I suppose early adopters are happy to learn it themselves.

Meanwhile, roughly half of the delegates that I am teaching Python have come from Perl. There does not appear to be a consistent reason for this, and often it is not the practitioner's decision anyway. It appears to be the perception of where Perl is in the scheme of things. Its all about image and marketting. What can Perl do about that? Perl 6, but Python is catching up.

Here is a simple example. One of the nice things about Perl is the way that lists can be used:

($one, $two, @fred) = qw(The quick brown fox);

Cannot do that in Python 2, but in Python 3:

one, two, *fred = ('The', 'quick', 'brown', 'fox')

The * indicates a greedy list (and you though Python didn't have sigils?). No qw() equivalent yet though, and Perl array and hash slices are still more powerful. Unless you know better...

Now I learn that Civilisation V is using Lua as its scripting language because Python (used in Civ. IV) is too slow. Is Lua next on my list?

Sony Vaio Windows 7 "No Internet access"

This error has been driving me crazy. I got it when trying to connect to certain WiFi's, but not all. Surfing the web gave various solutions, none of which worked. I eventually cracked it by accident, so I'm posting it in the hope I might prevent someone else from going insane. It is far too late for me.

Sony bundles all sorts of clever programs on its laptops. It's mostly concerned with multi-media, but one of them concerns us - VAIO Control Center (sic). So go to start/"All Programs" and select it. Now select "Network Connections", then "VAIO Smart Network".
Here be Dragons.

VAIO Smart Network creates "profiles". Click on "Advanced", recognise that stupid dialogue displayed at start-up? Click "Settings" (bet you never thought of going there).
Now select "Profile Settings" from the left panel. Yes, I know this navigation is tortuous, but we are nearly there. From here you can edit one of your profiles, or create a new one. When a new profile is created from the desktop dialogue when it first connects, it defaults everything to the previous settings. It was carrying over a DNS setting I had from another connection, and that was preventing me from connecting. Select the "IP and DNS" tab and ensure that "Obtain an IP address automatically" and "Obtain DNS server address automatically" are both selected (it was the later which screwed me).

The profile is selected by the name of the network. One thing that Sony engineers did not take into account in their design is a name collision. That is, two distinctly different networks having the same name. Many of our training centres have WiFi's, and they all are named the same, however they have different DNS addresses. That allowed me to connect to one training centre successfully, but then fail at all the others because the profile remembered the DNS address from the previous site. Going through the Microsoft Windows network settings did not detect where this was being held, because Sony were causing the problem not Microsoft (for once).

Hope this helps.

A view from EuroPython 2010

I have just returned from EuroPython which was, like last year, in Birmingham, UK. This post will not go into too much technical detail, email if you need more. I'm guessing that the reader does not want to know how Python implements IEEE 754 floating point format (that's even mentioned in the Programming Foundations course, do try to keep up!). It requires about 4000 lines of C code to convert between float and text - no, you didn't want to know that.
Neither am I going to give a blow-by-blow account of each talk - they are available on the europython 2010 website.
There were just under 400 delegates this year - slightly down on last year. The organisers should not be concerned though: early indications from rival conference YAPC Europe (Yet Another Perl Conference) is that attendance is approximately half of last year.
Organisation was slightly better than 2009, a number of lessons appear to have been learnt. Talk streams were still patchy, but I guess that's inevitable.


Multiprocessing, the GIL, and threading
The conference was opened by Russel Winder who impressed last year as well. The theme was very familiar to me, it told the story that Intel, and others, have been banging on about for a couple of years now - the fact that multi-core CPUs are here to stay and currently the only way to get the increase in speed beloved of developers. Russel went much further though, with my brain screaming ME TOO! Where Intel's solutions are distinctly small scale with core numbers in single figures, Russel spoke of much larger clusters (note the term). His contention was that current hardware solutions with cache are not scalable beyond 16, and neither are software solutions using threads. Message passing systems, like MPI, are a more likely future than threading systems like TBB or OpenMP. And no one in their right mind would be programming native threads. Here, here. A thousand times, here, here. All these lessons were learnt on mainframes in the 1970s: those that ignore history are destined to repeat it. It is a great shame that Russel's talk was rushed.

That did not stop discussion in the conference about the vagaries of the Global Interpreter Lock (GIL), the much maligned excuse for not doing multithreading in Python. Hey guys: multithreading is hard, error prone, and not necessarily all that faster. Let's just accept that and move on. There will be a reworked GIL in Python 3.2, work which has come out of Google's "Unladen Swallow" project. It will be interesting to see how much that helps (?) and what excuses people will use to avoid multithreading in the future.

I was browsing one of the book stalls, looking at an Advanced Python book when it hit me how much Python has moved forward in the past couple of years. The section on Multiprocessing mentioned an out-of-date module, not Subprocess and Multiprocessing. I dismissed the book as "out-of-date" even though the first edition was 2008 - QA's own Python courses have always covered those two modules. Then I realised that we have only had our own courses for about a year, although I have been tinkering with Python for (wow!) over ten years.


Python's progress
Version 2.7 of Python was released on 4th July. OK, I'll update the QAPYTH2 course material as soon as someone gives me the time. (Actually many of the significant changes in 2.7 are back-ports from Python 3.1, so I can flitch the material from my Python 3 course. Just don't tell the boss). Python 2.7 is the last Python 2 major release. Version 3.1 is now the only Python development stream, although 2.7 will continue to be maintained for around five years. When I started writing our own Python course material in late 2008 I was interrupted by the Python 3.0 release literally as I was writing the PowerPoint slides. I decided that a new course on Python 2 was daft, and switched to Python 3. It just so happened that I thought that Python 3 was a vast improvement for the language as well (don't cry for me, Perl 6). Six months later at EuroPython 2009 I realised I might have made a mistake, so I back-ported the course material to Python 2. I ended-up with two courses and "let the market decide". One year on and we have taught many more Python 3 courses than Python 2. and we are in a great position to move forward, unencumbered by legacy Python. Maybe it wasn't such a bad idea to target Python 3, but it was scary to be ahead of the curve.

Python 3.2 should be out around the end of the year. One speaker gave January 2011 and another said "before the end of this year". Take your pick. Christmas? Python 3.3 was mentioned a few times, it should include Google's "Unladen Swallow" (yes Ian, it is from "Monty Python and the Holy Grail") with a target of a 5 times performance improvement (European or Asian?). No dates yet.

My impression is that many developers still have not realised the benefits of Python 3, and have no plans to move over. They will.

It looks like Python will overtake Visual Basic in the Tiobe stakes in the next few months (it passed Perl a couple of years ago). Are developers realising the benefits of Python? They are.


The cheese shop (the Python module repository: PyPi)
(Recent course delegate: "but the Monty Python cheese shop had no cheese in it!". Me: "neither has PyPi")

One of the main benefits of Perl (what?) is CPAN. Dave Cross (well-known Perlmonger) has said that this was the main reason for not moving to Python. Well Dave, Python just had its 10,000th module uploaded (round of applause). Actually I'm not so sure that is such a good thing, duplication and crud makes navigation difficult. Still, it's an indication of popularity and progress.


One of the main benefits of Python 2 against Python 3 is the cheese chop. Here too Python 3 is catching up. The module numbers are still only a few hundred, but there major modules are appearing all the time, for example NumPy (numerical processing) was just released for Python 3.


Wot I learnt
Quite a lot, as always. I learnt the expression for such as __path__ is "dunder path". Nice one. I also learnt about the importance of the new unittest module (that'll have to go in) and changes in the way import works for Python 3.2. I discovered how technicians lie about language comparisons to their managers to justify using a cool product (actually, I already knew that). Speakers were at times less than accurate with their comparisons against Perl.

I learnt how HTML5 is going to make Microsoft Silverlight obsolete. Well the speaker did not actually say that, but I can dream.

There were some fascinating statistics, like there are more people in India with access to a mobile 'phone than to a flush toilet (please don't take the comparison further). I would love to use that in a course if only I could find the derivation.

The portability issues of using ActiveX are well-known to anyone using Microsoft's remote desktop (there are no portability issues - it's not portable), but apparently not to the South Korean government.

The spec. for HTML1 was three pages, for HTML5 is 900. That's progress.

Oracle had a stand at the conference, and they were giving out DVD's with developer's resource on for, let's see, Linux, PHP, Ruby, Python, and Oracle VM. Hummm, spot the missing language. Me: "What about Perl then?"; Oracle chappie: "oh, that's only used by system administrators for scripting". Go figure.

And you can write cool games in Python, drive neat little robots, automate PowerPoint slides (I gotta get me some of that, and the robot stuff). And the Python Software Foundation are as cheerfully disorganised as everyone else. As with all conferences, it is always nice to have opinions confirmed, and to be able to say from time to time "actually, I knew that".

Oh, and I saw a great example for "If then else for the lazy" in our UNIX fundamentals course:
./configure && make


Finally...
Python is self-assured, confident, and looking forward. The community is proud of its product, and itself. So they should be.

Python exercise - alternative solution

I promised to post an alternative solution to a Python exercise for an on-site today. For other interested parties, the code lists unused ports from /etc/services:

import sys

# set the file name depending on the operating system
if sys.platform == 'win32':
    file = r'C:\WINDOWS\system32\drivers\etc\services'
else:
    file = '/etc/services'

# Create an empty dictionary
ports = dict()

# Iterate through the file, one line at a time
for line in open(file):

   # Ignore lines starting with '#' and those containing only whitespace
   if line[0:1] != '#' and not line.isspace():

       # Extract the second field (seperated by \s+)
       pp = line.split(None, 1)[1]

       # Extract the port number from port/protocol
       port = pp.split ('/', 1)[0]

       # Convert to int, then store as a dictionary key
       port = int(port)
       ports[port] = None
   
       # Give up after port 200
       if port > 200: break

# Print any port numbers not present as a dictionary key
for num in xrange(1,201):
    if not num in ports:
        print "Unused port", num

Here is a smaller solution using Regular Expressions:

import sys,re

file = r'C:\WINDOWS\system32\drivers\etc\services' \
    if sys.platform == 'win32' else '/etc/services'

found = set()
for line in open(file):       
    m = re.search(r'^[^#].*\s(\d+)/(tcpudp)\s',line)
    if m:
       port = int(m.groups()[0])
       if port > 200: break
       found.add(port)
     
print set(range(1,201)) - found   

On the takeover of Sun by Oracle

I guess I might as well comment - everyone else and his dog is doing so. A collegue said "A sad, sad day for FOSS". I know what he means, but actually I disagree. Take-overs and mergers will always happen.

It is occasions like this that demonstrate the power of FOSS. FOSS gives us choice, and we can choose to stay with MySQL or migrate to PostgreSQL, which all the smart people were using anyway. IMHO, technically PostgreSQL knocks MySQL into a cocked hat, and the Oracle takeover of MySQL could give PostgreSQL the increase in popularity it deserves. Other databases are available.

Sun blew hot and cold on FOSS anyway, I could never figure out what their policy was. Java probably has too much momentum of its own for Oracle to screw it up, although if anyone can.... Never underestimate the ability of corporates to kill things off by hubris (I used to work for Computer Associates).

There will always be other languages, other databases, other operating systems.

Other hardware?

Back in the 1980s a collegue (Haydn Moston - are you still around?) told me that CISC (Complex Instruction Set Computer) chips could never last beyond 2000, and RISC (Reduced Instruction Set Computer) chips were the only way to go. The i386 familiy is CISC, and Sun Sparc is the most well-known RISC.

Intel are running out of steam and having to use multi-core, ten years out was actually not bad Haydn. I'm not sure that the switch to multi-core is connected specifically with CISC, but the possible loss of RISC machines is my biggest worry with the Oracle takeover. The number of mainstream instruction sets out there is disapointingly small. How I hate monocultures.

File notification events on Windows

One of my most popular posts is example code for Inotify on Linux. I have also been asked for similar code for Windows, so yer tis:

There are two different interfaces available on Win32, I prefer ReadDirectoryChangesW because it is easier to control:

// ------------------------------------------------------------------
// Clive Darke QA Training
// ReadDirectoryChangesW example
// ------------------------------------------------------------------

#define _WIN32_WINNT 0x0400      // <<<<<<<<>
#include <iostream>
#include <windows.h>

void DisplayLastError( LPSTR lpszText );

// ------------------------------------------------------------------

int main ( int argc, char *argv[] )
{
   HANDLE hDir;
   DWORD dwReturned;
   BOOL bResult;
   FILE_NOTIFY_INFORMATION *pNotify;
   
   if ( argc < 2 )
   {
      cerr << "You must supply a directory name" << endl;
      return 1;
   }

   // Note FILE_FLAG_BACKUP_SEMANTICS, which is the strange
   // attribute required to get a handle to a directory.   

   hDir = CreateFile (
          argv[1],                            // pointer to the file name
          FILE_LIST_DIRECTORY,                // access (read-write) mode
          FILE_SHARE_READ|FILE_SHARE_DELETE,  // share mode
          NULL,                               // security descriptor
          OPEN_EXISTING,                      // how to create
          FILE_FLAG_BACKUP_SEMANTICS,         // file attributes
          NULL                                // file with attributes to copy
        );

   char Buffer[MAX_PATH] = {0};

   while (TRUE )
   {
      char szAction[42];
      char szFilename[MAX_PATH];

      bResult = ReadDirectoryChangesW (hDir, &Buffer, sizeof(Buffer),
      TRUE, FILE_NOTIFY_CHANGE_FILE_NAME, &dwReturned, NULL, NULL);
    
   if ( !bResult )
      break;
    
   pNotify = (FILE_NOTIFY_INFORMATION *) Buffer;

   switch (pNotify->Action)
   {
    case FILE_ACTION_ADDED :    {
     strcpy (szAction, "added");
     break;
    case FILE_ACTION_REMOVED :
     strcpy (szAction, "removed");
     break;
    case FILE_ACTION_MODIFIED :
     strcpy (szAction, "modified");
     break;
    case FILE_ACTION_RENAMED_OLD_NAME :
     strcpy (szAction, "renamed");
     break;
    case FILE_ACTION_RENAMED_NEW_NAME :
     strcpy (szAction, "renamed");
     break;
          default:
     strcpy (szAction, "Unknown action");
    }

      wcstombs( szFilename, pNotify->FileName, MAX_PATH);
     cout << "File " << dwerror =" GetLastError();" lpmessagebuffer =" NULL;">

ASSOC on Windows 7

A while back I posted some suggestions for .bat files to run cmd.exe sessions for Python. I use these a lot for PHP and Perl as well. However there is an issue with them on Windows 7. "Out of the box", the ASSOC command gives "Access is denied", even though the user is an Administrator.

My first thought was to turn off the dreaded User Account Control settings, but that changed nothing. The eventual solution was to go into regedit and grant my user full control on registry key HKEY_CLASSES_ROOT, which is where the file associations are held. Fortunately Windows security for keys (and directories) is based on inheritence, and so keys within that hive will have the same permissions by default.

I can see why file association sould be seen as a security risk, so why don't Microsoft finally admit that the filename is not a good way to identify the file type, and use magic numbers like everyone else? Microsoft Office files, and other Microsoft proprietry formats, all contain magic numbers, and if you place one of these files onto Linux then it identifies them correctly without the need for file extensions.

UNIX's #! line for script files is a consequence of the magic number system. It is simple and incredibly flexible. When C# was developed the format of the .exe files (PE) had to change to support it. If they implemented #! then Microsoft would not have had that problem.

When I wrote the yash shell I simply read the commands from STDIN. On UNIX I needed no special code to support yash scripts - just the #! line. But on Windows I had to associate a file extension.

If I want to run Python 2 and Python 3 on the same machine on UNIX or Linux then I just have different #! lines at the top of the script. On Windows, because they have the same file extension, I have to redo the file associations every time I want to run one of them. Dire.

A winning feature of Perl 6

So what features do Python programmers love to say about their code? First off they like to crow about how clean their code looks without sigils, those nasty $, @, %, &, * that Perl programmers have to put up with. Ah, but that means you cannot do any interpolation in Python strings, because Python does not know the difference between bare text and a variable name.

Anyway, Python uses _ and __ (double underscore) to change the behaviour of variables, if these are not sigils then what is? Python uses an @ to prefix a decorator. Sounds like a sigil to me.

Perl 6 mandates that variables are declared using 'my' (or some other scope indicator), but Python does not require that. This means that in Python you cannot declare a variable for scope inside a local block, like an 'if' or loop.

Python does not use braces to delimit a block. Nice, unless you need to declare an empty block, so you need a 'pass' statement (reminds me of NEXT-SENTENCE).

Most of that is trivial to be honest, none is important enough to make a strong case either way.

So what is? Threading.

A recuring theme at the EuroPython conference was the fact that CPython is not truely multi-threaded because of the GIL (Global Interpreter Lock). Ruby has the Global VM Lock. There are various implementations which bypass this issue, Jython and Iron Python do not have a problem with multi-threading but both are several versions behind and I won't be holding my breath for a Python 3 version. Google is apparently working on a CPython version which eliminates the GIL, but again that is some way off.

In a classic "sour grapes", the Python community say that threading is too difficult anyway, so use processes instead. Very few people at the conference were swallowing that line.

Multithreading should be a selling point of Perl 6 and Parrot, but unfortunately is not yet complete and will not be in Rakudo Star. That is a great pity.

Perl 6 gets a release date!

Big news from the current Perl conference YAPC Europe 2009 here in Lisbon is that Rakudo * (star) will be released in spring 2010. Rakudo is Perl 6 on Parrot, which is an equivalent to the JVM or CLI.

Is that an official Perl 6 release? Well, it is the nearest to official that you will see. The Perl 6 team are keen to point out that there will be no such thing as a official version, just different implementations. I guess they are looking at Python for their inspiration there.

This could be make-or-break for Perl. It shouldn't be, Perl 5 will continue no matter what, but the new stuff is what will make Perl cool again in the eyes of the fashion conscious. We all know that fashion is fickle, illogical, and driven more by hormones than intellect. We also know how important it is in decision making. Much as it would like to, the Perl community cannot pretend that Perl 6/Rakudo is too far away to worry about. It is six months away.

My worry is that expectations will be high - the release will be incomplete and they are very open about that. The Rakudo "pumpking" (project manager), Patrick Michaud, sees this as a test release to encourage people to write applications to find bugs and feed back into the development. Trouble is that companies have been burned too often in the past to invest in applications on an incomplete and possibly flaky platform.

The dilemma is that if something is not released soon then Perl will have missed the boat - which is already upping anchor and about to steam away.

Update: read the comments from Pm - they are very good.

Perl 6 Questions and further information

This is following a recent Perl 6 seminar I gave.

Perl 5 style is still supported.

Named parameters go into %_
Unnamed parameters go into @_

Subroutines can be predefined with:

sub foo {…}

… is known as the "yadayadayada" operator.

Lexically scoped subroutines:

my sub mysub () is context { …}

Optional parameters: ? now comes after the parameter name.

sub mysub ($value, @harry, %ash, $opt?) {…} 

Named parameters: prefix is now : , instead of + and are optional unless they have a ! suffix.

sub named ($value, :@harry!, :%ash, :$other) {…}

...

named(42, other = 'fred', harry = @files);

Slurpy parameters, like perl 5 lists and arrays, are prefixed *

C style prototypes
Are not supported. It appears that when a subroutine is called before it is defined the parameter resolution still works correctly. However a runtime error is reported, not a compile time error (as it would with Perl 5 prototypes).

Dynamic types
These are called "traits". See Synopsis 12: Objects.

Zip operator
Has changed, it is now 'Z'. Behaviour with different length arrays has also changed between pugs and Rakudo, the result uses the shortest:

my @array = qw(The quick brown fox);
my @brray = qw(Now is the time for all);

my @all = @array Z @brray;
say @all.elems();
say @all[];

@all = @brray Z @array;
say @all.elems();
say @all[];

Gives:

8
TheNowquickisbrownthefoxtime
8
NowTheisquickthebrowntimefox

Important further change

When testing these changes in Rakudo I ran up against a change I had almost forgotten. When calling a subroutine, no whitespace is allowed between the sub name and the open parentheses. That slowed me up until I remembered. A backward step IMHO.

MySQL VARCHAR limits

A delegate recently queried the size limit on a VARCHAR field in MySQL. Our course documentation states a maximum of 255 characters: that is correct. For larger fields use TEXT.

Perl 6 Questions

I recently gave a short talk on Perl 6, and a number of questions arose. I will answer those questions as soon as I can, but unfortunately other fires need fighting right now.

One of the reasons for the delay is that the syntax for subroutines for Perl 6 has changed again, and in fact my slides are now out of date. That also means that pugs and Rakudo are incompatible in a major way - they have always been out of step to some extent.

Moving into the light?

A long time since my last post. Aside from bread-and-butter stuff, my activity has been on Python. There have been various upheavals in the business, and we aquired a Python 2 course. It wasn't too hot, and needed updating. Around the end of 2008 Python 3 was released, so I decided it would make sense to create a new course based on Python 3.

That is almost finished, but it turns out that it might have been premature, since no one seems to be in a particular hurry to move to Py3. That's a shame, because I am quite impressed. There have been issues, performance has been one of them (although I can't say I noticed any problems). In June 2009 a new version, 3.1, should be released, and that has a number of new features. So, having done all that work on a new course, I will have to hack it again come June. That's the problem with the bleading edge.

I have read of Python's ease of use, and never really believed it. After all, there are so many vested interests and religious wars that it is difficult to get a balanced view. I have to say that, after a little practice, Python is very easy to pickup. I have found myself just coding. The number of times I have to look something up gets less and less. I'm not sure I am fluent yet, but I'm close, and I reckon that is the fastest I have ever picked up another language.

Will Perl 6 be as easy?

App::sh2p Version 0.05 now on CPAN

I always knew this would be a long project, although I'm heartened as to how far it has gone. It appears to have been useful to some people. It is a feature of Open Source that the only time I can be really sure it is being used is when I get bug reports - which could be rather off-putting to anyone without the skin of a rhino.

This latest release, version 0.05, fixes quite a few bugs, and adds support for things like trap (signal handling). In my defence, most of the bugs are missing support for certain features, and it is still in beta. I still have not got pipelines sorted out (that is a huge job), or $?, but they will come eventually. I only get to work on this in odd moments, evenings and so on, and it is difficult to find the time for development stints of more that two hours. I could really do with spending a couple of solid days on it.

One problem I had was in inserting a marker into a string which would indicate a token delimiter. Everything I tried clashed in someway with a real data item. I solved it eventually by using a gash reference, but I can't help thinking there must be a more elegant solution.

I have also started using OO techniques more - this product really does benefit from that.

I am now using the profiler and Coverage for my tests, but have not sorted out a good way to integrate my command-line tests. The "Perl Testing" book has ideas, but they require CPAN modules and I would rather not increase the number of dependencies. If you download this program and are dismayed at the tests (I basically check the thing will load) then be assured that I have a test suite of over 90 scripts, just one of which is my regression test script which currently has 55 test paragraphs within it. Each time I find a bug or add a feature I add a test, just like I should.

Shock! Horror!! A delegate ate my machine!

OK, he didn't actually eat it, but he did exploit a "feature" of CentOS I was unaware of. The halt, reboot, and poweroff programs can be called from an ordinary user if that user is logged on to the console at the time. Our classroom users have very simple passwords, but the point is that these can be called from a non-root user.

I found a fix (with thanks to Mr. Google) but I'm sure there is probably a better one. Right now this will do, since it appears to work, but I'll try and find an alternative.

In /etc/security/console.apps there is a config file for the programs mentioned (and others). Set each one to something like (for example):

USER=nobody
PROGRAM=/sbin/poweroff
SESSION=true

Of course having better passswords would help...

Function redirection

I just came across this:

function myfunc {  

    echo 'Hollow world' 

} > outfile

Works with Bourne style functions as well, and is in the POSIX standard. It should not be a surprise, since it just follows the same standard as 'if' and loops, but I have never seen it before. It certainly is not our shell course (yet). Seen it before?

Here is a neat trick:

outfile=one.out

function myfunc {
    echo 'Hollow world'
} > $outfile

outfile=two.out
myfunc

Which file gets written to, one.out or two.out? Interestingly it is two.out. In other words the variable is resolved at runtime, not function declaration time.

App::sh2p - shell to perl converter

Just uploaded to CPAN, this has been a pet project of mine for a while (see perlmonks here ). It is currently only a alpha release, and you know what that means.

Converting UNIX shell scripts creates problems and my script (with associated modules) do not answer them all. The syntax of shell programs is very loose, almost anarchic. This is not a criticism of shells - they have a dual role to play in providing a usable command-line interface, as well as acting as a programming language. I target the POSIX shell, but there are many variations, not least those created by Sun. The Korn shell has moved on considerably from ksh88, and the syntax of the latest versions is complex. I make no attempt at the more esoteric parts of ksh93 syntax, expansions and extended globbing. There are annoying inconsistencies, fore example, what is the difference between:

[ $x -a $b ]

and

[[ $x -a $b ]]

I shall leave the reader to ponder (hint: -a has a different meaning to the Bourne-shell [ and the Korn shell [[).

As you might gather, this is not a simple syntax conversion. Take $? for example: that can be applied to any shell command you like and at any point in the program. Not many languages work like that, certainly Perl does not. Fortunately not many people use $? in shell scripts for anything other than testing the result of a child process - not unlike its use in Perl (there is a difference in the detail).

Right now App::sh2p attempts to remove some of the leg-work from conversion, and I hope t is useful.

Flip-flop operator

I have been using the flip-flop operator in Perl for ages, equating it with sed and awk line addressing. However a question just came up that asked how we process from a specific line until end-of-file. In sed and awk the last line of the file is denoted by $, but that does not work in Perl for obvious syntax reasons.

I had to think about that: the solution is to use eof(). So, to list a file from line 20 we can use:

while (<>)
{
   print if 20..eof 
}