The Darke Side

Perl - the way ahead

2011-08-17T09:54:00.001+01:00

Wow! I've just been knocked off my feet.

I'm at YAPC::Europe (Yet Another Perl Conference) in Riga, and just listened to the vision for Perl 5's future. Jesse Vincent is the Perl 5 Pumpking, essentially the project manager. His vision is to evolve Perl 5 into a modern language (he did not use those words, but that's the gist).

Old Perl is 5.14, the current stable release. The idea is to introduce changes, like a fixed smartmatch, and deprecate the stuff that was preventing Perl 5 from moving forward because of the need for backward compatibility. Backward compatibility in the future will be provided by stating which version of Perl we should use - but still complete backward compatibility will not be guaranteed.

By moving platform specific features out of the base and into modules (as in Python), the base gets to be small and easier to maintain, and maybe even faster. Deprecating features then removing them should have the same effect.

What a shame this was not decided on ten years ago, still, better late than never. There will be issues with CPAN. Modules are usually only maintained for one version, the current stable one if you are lucky. Backward compatibility will be an issue with modules. The 'use version' directive (hopefully) will be lexical, so modules will be able to run on a later release (mostly, maybe). But of course they won't be able to run on an earlier release - but we have that issue right now. CPAN will need to search/tag/group modules by base release, that should not be hard, but it needs to be done before the issues arise at 5.16.

So where does that leave Perl 6? Perl 6 is not dead, but rather it takes the pressure off it. Most people (me included) looked to Perl 6 for the future, and all the smart stuff that's in Python that I can't do in Perl. No longer. Perl 5 will move forward, alongside its sister language. The possibility of Perl 5 overtaking Perl 6 was not discussed.

The times, they are a-changing.

Installing Python 3.2 Tkinter on Linux

2011-04-07T10:18:00.005+01:00

A quick note for the install. Tkinter is not installed by default with Python 3.2 on Linux unless the tk development pack is installed. Where that is documented I could not say.

On Red Hat derived distributions this would be using (under user root):
yum install tk-devel
Make sure you do this before building Python. It is required for idle, which is named idle3 (and that is not obvious either).

A Christmas present from the monks

2010-12-27T11:49:00.002Z

After what seems an age I finally got promotion to Prior, next stop (in 3000 XP points time) is Monsignor. My best node was Unexpected Output with a score of (strangely) 42.

It's weird, I get such a buzz from helping people with Perl it is almost adictive.

Python msvcrt.locking trials and tribulations

2010-11-01T16:52:00.004Z

It should be easy. All I wanted to do is to replace my C demos which use flock with a nice homely Python version on Windows. I wanted to stick with the standard library so I could just slot it into the existsing course (so I couldn't use the Locking module on PyPi).

A simple demo to start with:
1. Process 1: lock a region, write a record, pause
2. Process 2: attempt to lock the same region
3. Allow Process 1 to release the lock
4. Process 2 continues

Converting the code was easy (I thought), but no matter what I did I always found the region was locked for the second process - the lock was not being released.

I blame the documentation, and possibly the implementation.

What is not obvious is that the current file position must be reset to the original locking position before the lock gets released. The write (of course) advances the current file position, so by the time we do the unlock we are unlocking a region for which we don't have the lock in the first place! Wouldn't it be better if msvcrt.locking returned a lock object, saving the file position? Anyway, here is my completed demo:


"""
   lock_w.py

   Run two copies, each in its own terminal session.
   Allow one to write a number of records, then switch
   to the other showing that it blocks on the same record.
   Switch back and release the lock, and show that the
   blocked process proceeds.

   Also run with lock_r to demonstrate interaction with
   read locks.

   Default filename is rlock.dat

   Clive Darke QA
"""

import msvcrt
import os
import sys
import time
from datetime import datetime

REC_LIM   = 20

if len(sys.argv) < 2:
    pFilename = "rlock.dat"
else:
    pFilename = sys.argv[1]

fh = open(pFilename, "w")

# Do only once
Record = dict()
Record['Pid'] = os.getpid()

for i in range(REC_LIM):
    Record['Text'] = "Record number %02d" % i
    Record['Time'] = datetime.now().strftime("%H:%M:%S")

    # Get the size of the area to be locked
    line = str(Record) + "\n"
    
    # Get the current start position
    start_pos  = fh.tell()          
    
    print "Getting lock for",Record['Text']
    msvcrt.locking(fh.fileno(), msvcrt.LK_RLCK, len(line)+1)
    fh.write(line)       # This advances the current position
    
    raw_input("Written record %02d, clear the lock?" % i)
    
    # Save the end position
    end_pos = fh.tell()
    
    # Reset the current position before releasing the lock
    fh.seek(start_pos)
    msvcrt.locking(fh.fileno(), msvcrt.LK_UNLCK, len(line)+1)
    
    # Go back to the end of the written record
    fh.seek(end_pos)
    
    print      

fh.close()

Update

2010-10-13T11:12:00.002+01:00

I try to keep these posts technical, but I have not blogged in a while so I'll break with tradition.

I have been teaching a lot of Python. Mostly this has been Python 2, but some clients want Py3. Unfortunately not everyone understands how Open Source language releases work, and some internally assumed that no-one would want Python 2 training as soon as Python 3 was released.

I have to confess that, having done a lot of Python recently, moving back to Perl is rather a drudge. Python is not perfect, but I now find the syntax of Perl unnecessarily fussy, and Perl 6 is even worse. I mean fussy in the sense that Perl code is to Python as a doily is to a beer-mat.

Talking of Perl 6, with the release of Rakudo Star I thought (hoped) that we would get a flood of requests for courses, but we have not had one. Rakudo Star is, I guess, still too early, and I suppose early adopters are happy to learn it themselves.

Meanwhile, roughly half of the delegates that I am teaching Python have come from Perl. There does not appear to be a consistent reason for this, and often it is not the practitioner's decision anyway. It appears to be the perception of where Perl is in the scheme of things. Its all about image and marketting. What can Perl do about that? Perl 6, but Python is catching up.

Here is a simple example. One of the nice things about Perl is the way that lists can be used:


($one, $two, @fred) = qw(The quick brown fox);

Cannot do that in Python 2, but in Python 3:


one, two, *fred = ('The', 'quick', 'brown', 'fox')

The * indicates a greedy list (and you though Python didn't have sigils?). No qw() equivalent yet though, and Perl array and hash slices are still more powerful. Unless you know better...

Now I learn that Civilisation V is using Lua as its scripting language because Python (used in Civ. IV) is too slow. Is Lua next on my list?

Sony Vaio Windows 7 "No Internet access"

2010-08-05T08:46:00.004+01:00

This error has been driving me crazy. I got it when trying to connect to certain WiFi's, but not all. Surfing the web gave various solutions, none of which worked. I eventually cracked it by accident, so I'm posting it in the hope I might prevent someone else from going insane. It is far too late for me.

Sony bundles all sorts of clever programs on its laptops. It's mostly concerned with multi-media, but one of them concerns us - VAIO Control Center (sic). So go to start/"All Programs" and select it. Now select "Network Connections", then "VAIO Smart Network".
Here be Dragons.

VAIO Smart Network creates "profiles". Click on "Advanced", recognise that stupid dialogue displayed at start-up? Click "Settings" (bet you never thought of going there).
Now select "Profile Settings" from the left panel. Yes, I know this navigation is tortuous, but we are nearly there. From here you can edit one of your profiles, or create a new one. When a new profile is created from the desktop dialogue when it first connects, it defaults everything to the previous settings. It was carrying over a DNS setting I had from another connection, and that was preventing me from connecting. Select the "IP and DNS" tab and ensure that "Obtain an IP address automatically" and "Obtain DNS server address automatically" are both selected (it was the later which screwed me).

The profile is selected by the name of the network. One thing that Sony engineers did not take into account in their design is a name collision. That is, two distinctly different networks having the same name. Many of our training centres have WiFi's, and they all are named the same, however they have different DNS addresses. That allowed me to connect to one training centre successfully, but then fail at all the others because the profile remembered the DNS address from the previous site. Going through the Microsoft Windows network settings did not detect where this was being held, because Sony were causing the problem not Microsoft (for once).

Hope this helps.

A view from EuroPython 2010

2010-07-25T09:26:00.005+01:00

I have just returned from EuroPython which was, like last year, in Birmingham, UK. This post will not go into too much technical detail, email if you need more. I'm guessing that the reader does not want to know how Python implements IEEE 754 floating point format (that's even mentioned in the Programming Foundations course, do try to keep up!). It requires about 4000 lines of C code to convert between float and text - no, you didn't want to know that.
Neither am I going to give a blow-by-blow account of each talk - they are available on the europython 2010 website.
There were just under 400 delegates this year - slightly down on last year. The organisers should not be concerned though: early indications from rival conference YAPC Europe (Yet Another Perl Conference) is that attendance is approximately half of last year.
Organisation was slightly better than 2009, a number of lessons appear to have been learnt. Talk streams were still patchy, but I guess that's inevitable.

Multiprocessing, the GIL, and threading
The conference was opened by Russel Winder who impressed last year as well. The theme was very familiar to me, it told the story that Intel, and others, have been banging on about for a couple of years now - the fact that multi-core CPUs are here to stay and currently the only way to get the increase in speed beloved of developers. Russel went much further though, with my brain screaming ME TOO! Where Intel's solutions are distinctly small scale with core numbers in single figures, Russel spoke of much larger clusters (note the term). His contention was that current hardware solutions with cache are not scalable beyond 16, and neither are software solutions using threads. Message passing systems, like MPI, are a more likely future than threading systems like TBB or OpenMP. And no one in their right mind would be programming native threads. Here, here. A thousand times, here, here. All these lessons were learnt on mainframes in the 1970s: those that ignore history are destined to repeat it. It is a great shame that Russel's talk was rushed.

That did not stop discussion in the conference about the vagaries of the Global Interpreter Lock (GIL), the much maligned excuse for not doing multithreading in Python. Hey guys: multithreading is hard, error prone, and not necessarily all that faster. Let's just accept that and move on. There will be a reworked GIL in Python 3.2, work which has come out of Google's "Unladen Swallow" project. It will be interesting to see how much that helps (?) and what excuses people will use to avoid multithreading in the future.

I was browsing one of the book stalls, looking at an Advanced Python book when it hit me how much Python has moved forward in the past couple of years. The section on Multiprocessing mentioned an out-of-date module, not Subprocess and Multiprocessing. I dismissed the book as "out-of-date" even though the first edition was 2008 - QA's own Python courses have always covered those two modules. Then I realised that we have only had our own courses for about a year, although I have been tinkering with Python for (wow!) over ten years.

Python's progress
Version 2.7 of Python was released on 4th July. OK, I'll update the QAPYTH2 course material as soon as someone gives me the time. (Actually many of the significant changes in 2.7 are back-ports from Python 3.1, so I can flitch the material from my Python 3 course. Just don't tell the boss). Python 2.7 is the last Python 2 major release. Version 3.1 is now the only Python development stream, although 2.7 will continue to be maintained for around five years. When I started writing our own Python course material in late 2008 I was interrupted by the Python 3.0 release literally as I was writing the PowerPoint slides. I decided that a new course on Python 2 was daft, and switched to Python 3. It just so happened that I thought that Python 3 was a vast improvement for the language as well (don't cry for me, Perl 6). Six months later at EuroPython 2009 I realised I might have made a mistake, so I back-ported the course material to Python 2. I ended-up with two courses and "let the market decide". One year on and we have taught many more Python 3 courses than Python 2. and we are in a great position to move forward, unencumbered by legacy Python. Maybe it wasn't such a bad idea to target Python 3, but it was scary to be ahead of the curve.

Python 3.2 should be out around the end of the year. One speaker gave January 2011 and another said "before the end of this year". Take your pick. Christmas? Python 3.3 was mentioned a few times, it should include Google's "Unladen Swallow" (yes Ian, it is from "Monty Python and the Holy Grail") with a target of a 5 times performance improvement (European or Asian?). No dates yet.

My impression is that many developers still have not realised the benefits of Python 3, and have no plans to move over. They will.

It looks like Python will overtake Visual Basic in the Tiobe stakes in the next few months (it passed Perl a couple of years ago). Are developers realising the benefits of Python? They are.

The cheese shop (the Python module repository: PyPi)
(Recent course delegate: "but the Monty Python cheese shop had no cheese in it!". Me: "neither has PyPi")

One of the main benefits of Perl (what?) is CPAN. Dave Cross (well-known Perlmonger) has said that this was the main reason for not moving to Python. Well Dave, Python just had its 10,000th module uploaded (round of applause). Actually I'm not so sure that is such a good thing, duplication and crud makes navigation difficult. Still, it's an indication of popularity and progress.

One of the main benefits of Python 2 against Python 3 is the cheese chop. Here too Python 3 is catching up. The module numbers are still only a few hundred, but there major modules are appearing all the time, for example NumPy (numerical processing) was just released for Python 3.

Wot I learnt
Quite a lot, as always. I learnt the expression for such as __path__ is "dunder path". Nice one. I also learnt about the importance of the new unittest module (that'll have to go in) and changes in the way import works for Python 3.2. I discovered how technicians lie about language comparisons to their managers to justify using a cool product (actually, I already knew that). Speakers were at times less than accurate with their comparisons against Perl.

I learnt how HTML5 is going to make Microsoft Silverlight obsolete. Well the speaker did not actually say that, but I can dream.

There were some fascinating statistics, like there are more people in India with access to a mobile 'phone than to a flush toilet (please don't take the comparison further). I would love to use that in a course if only I could find the derivation.

The portability issues of using ActiveX are well-known to anyone using Microsoft's remote desktop (there are no portability issues - it's not portable), but apparently not to the South Korean government.

The spec. for HTML1 was three pages, for HTML5 is 900. That's progress.

Oracle had a stand at the conference, and they were giving out DVD's with developer's resource on for, let's see, Linux, PHP, Ruby, Python, and Oracle VM. Hummm, spot the missing language. Me: "What about Perl then?"; Oracle chappie: "oh, that's only used by system administrators for scripting". Go figure.

And you can write cool games in Python, drive neat little robots, automate PowerPoint slides (I gotta get me some of that, and the robot stuff). And the Python Software Foundation are as cheerfully disorganised as everyone else. As with all conferences, it is always nice to have opinions confirmed, and to be able to say from time to time "actually, I knew that".

Oh, and I saw a great example for "If then else for the lazy" in our UNIX fundamentals course:
./configure && make

Finally...
Python is self-assured, confident, and looking forward. The community is proud of its product, and itself. So they should be.

Python exercise - alternative solution

2010-03-04T19:40:00.005Z

I promised to post an alternative solution to a Python exercise for an on-site today. For other interested parties, the code lists unused ports from /etc/services:


import sys

# set the file name depending on the operating system
if sys.platform == 'win32':
    file = r'C:\WINDOWS\system32\drivers\etc\services'
else:
    file = '/etc/services'

# Create an empty dictionary
ports = dict()

# Iterate through the file, one line at a time
for line in open(file):

   # Ignore lines starting with '#' and those containing only whitespace
   if line[0:1] != '#' and not line.isspace():

       # Extract the second field (seperated by \s+)
       pp = line.split(None, 1)[1]

       # Extract the port number from port/protocol
       port = pp.split ('/', 1)[0]

       # Convert to int, then store as a dictionary key
       port = int(port)
       ports[port] = None
   
       # Give up after port 200
       if port > 200: break

# Print any port numbers not present as a dictionary key
for num in xrange(1,201):
    if not num in ports:
        print "Unused port", num

Here is a smaller solution using Regular Expressions:


import sys,re

file = r'C:\WINDOWS\system32\drivers\etc\services' \
    if sys.platform == 'win32' else '/etc/services'

found = set()
for line in open(file):       
    m = re.search(r'^[^#].*\s(\d+)/(tcpudp)\s',line)
    if m:
       port = int(m.groups()[0])
       if port > 200: break
       found.add(port)
     
print set(range(1,201)) - found

On the takeover of Sun by Oracle

2010-01-29T09:40:00.004Z

I guess I might as well comment - everyone else and his dog is doing so. A collegue said "A sad, sad day for FOSS". I know what he means, but actually I disagree. Take-overs and mergers will always happen.

It is occasions like this that demonstrate the power of FOSS. FOSS gives us choice, and we can choose to stay with MySQL or migrate to PostgreSQL, which all the smart people were using anyway. IMHO, technically PostgreSQL knocks MySQL into a cocked hat, and the Oracle takeover of MySQL could give PostgreSQL the increase in popularity it deserves. Other databases are available.

Sun blew hot and cold on FOSS anyway, I could never figure out what their policy was. Java probably has too much momentum of its own for Oracle to screw it up, although if anyone can.... Never underestimate the ability of corporates to kill things off by hubris (I used to work for Computer Associates).

There will always be other languages, other databases, other operating systems.

Other hardware?

Back in the 1980s a collegue (Haydn Moston - are you still around?) told me that CISC (Complex Instruction Set Computer) chips could never last beyond 2000, and RISC (Reduced Instruction Set Computer) chips were the only way to go. The i386 familiy is CISC, and Sun Sparc is the most well-known RISC.

Intel are running out of steam and having to use multi-core, ten years out was actually not bad Haydn. I'm not sure that the switch to multi-core is connected specifically with CISC, but the possible loss of RISC machines is my biggest worry with the Oracle takeover. The number of mainstream instruction sets out there is disapointingly small. How I hate monocultures.

File notification events on Windows

2010-01-07T14:53:00.006Z

One of my most popular posts is example code for Inotify on Linux. I have also been asked for similar code for Windows, so yer tis:

There are two different interfaces available on Win32, I prefer ReadDirectoryChangesW because it is easier to control:



// ------------------------------------------------------------------
// Clive Darke QA Training
// ReadDirectoryChangesW example
// ------------------------------------------------------------------

#define _WIN32_WINNT 0x0400      // <<<<<<<<>
#include <iostream>
#include <windows.h>

void DisplayLastError( LPSTR lpszText );

// ------------------------------------------------------------------

int main ( int argc, char *argv[] )
{
   HANDLE hDir;
   DWORD dwReturned;
   BOOL bResult;
   FILE_NOTIFY_INFORMATION *pNotify;
   
   if ( argc < 2 )
   {
      cerr << "You must supply a directory name" << endl;
      return 1;
   }

   // Note FILE_FLAG_BACKUP_SEMANTICS, which is the strange
   // attribute required to get a handle to a directory.   

   hDir = CreateFile (
          argv[1],                            // pointer to the file name
          FILE_LIST_DIRECTORY,                // access (read-write) mode
          FILE_SHARE_READ|FILE_SHARE_DELETE,  // share mode
          NULL,                               // security descriptor
          OPEN_EXISTING,                      // how to create
          FILE_FLAG_BACKUP_SEMANTICS,         // file attributes
          NULL                                // file with attributes to copy
        );

   char Buffer[MAX_PATH] = {0};

   while (TRUE )
   {
      char szAction[42];
      char szFilename[MAX_PATH];

      bResult = ReadDirectoryChangesW (hDir, &Buffer, sizeof(Buffer),
      TRUE, FILE_NOTIFY_CHANGE_FILE_NAME, &dwReturned, NULL, NULL);
    
   if ( !bResult )
      break;
    
   pNotify = (FILE_NOTIFY_INFORMATION *) Buffer;

   switch (pNotify->Action)
   {
    case FILE_ACTION_ADDED :    {
     strcpy (szAction, "added");
     break;
    case FILE_ACTION_REMOVED :
     strcpy (szAction, "removed");
     break;
    case FILE_ACTION_MODIFIED :
     strcpy (szAction, "modified");
     break;
    case FILE_ACTION_RENAMED_OLD_NAME :
     strcpy (szAction, "renamed");
     break;
    case FILE_ACTION_RENAMED_NEW_NAME :
     strcpy (szAction, "renamed");
     break;
          default:
     strcpy (szAction, "Unknown action");
    }

      wcstombs( szFilename, pNotify->FileName, MAX_PATH);
     cout << "File " << dwerror =" GetLastError();" lpmessagebuffer =" NULL;">

ASSOC on Windows 7

2009-12-08T11:19:00.001Z

A while back I posted some suggestions for .bat files to run cmd.exe sessions for Python. I use these a lot for PHP and Perl as well. However there is an issue with them on Windows 7. "Out of the box", the ASSOC command gives "Access is denied", even though the user is an Administrator.

My first thought was to turn off the dreaded User Account Control settings, but that changed nothing. The eventual solution was to go into regedit and grant my user full control on registry key HKEY_CLASSES_ROOT, which is where the file associations are held. Fortunately Windows security for keys (and directories) is based on inheritence, and so keys within that hive will have the same permissions by default.

I can see why file association sould be seen as a security risk, so why don't Microsoft finally admit that the filename is not a good way to identify the file type, and use magic numbers like everyone else? Microsoft Office files, and other Microsoft proprietry formats, all contain magic numbers, and if you place one of these files onto Linux then it identifies them correctly without the need for file extensions.

UNIX's #! line for script files is a consequence of the magic number system. It is simple and incredibly flexible. When C# was developed the format of the .exe files (PE) had to change to support it. If they implemented #! then Microsoft would not have had that problem.

When I wrote the yash shell I simply read the commands from STDIN. On UNIX I needed no special code to support yash scripts - just the #! line. But on Windows I had to associate a file extension.

If I want to run Python 2 and Python 3 on the same machine on UNIX or Linux then I just have different #! lines at the top of the script. On Windows, because they have the same file extension, I have to redo the file associations every time I want to run one of them. Dire.

A winning feature of Perl 6

2009-08-11T15:56:00.003+01:00

So what features do Python programmers love to say about their code? First off they like to crow about how clean their code looks without sigils, those nasty $, @, %, &, * that Perl programmers have to put up with. Ah, but that means you cannot do any interpolation in Python strings, because Python does not know the difference between bare text and a variable name.

Anyway, Python uses _ and __ (double underscore) to change the behaviour of variables, if these are not sigils then what is? Python uses an @ to prefix a decorator. Sounds like a sigil to me.

Perl 6 mandates that variables are declared using 'my' (or some other scope indicator), but Python does not require that. This means that in Python you cannot declare a variable for scope inside a local block, like an 'if' or loop.

Python does not use braces to delimit a block. Nice, unless you need to declare an empty block, so you need a 'pass' statement (reminds me of NEXT-SENTENCE).

Most of that is trivial to be honest, none is important enough to make a strong case either way.

So what is? Threading.

A recuring theme at the EuroPython conference was the fact that CPython is not truely multi-threaded because of the GIL (Global Interpreter Lock). Ruby has the Global VM Lock. There are various implementations which bypass this issue, Jython and Iron Python do not have a problem with multi-threading but both are several versions behind and I won't be holding my breath for a Python 3 version. Google is apparently working on a CPython version which eliminates the GIL, but again that is some way off.

In a classic "sour grapes", the Python community say that threading is too difficult anyway, so use processes instead. Very few people at the conference were swallowing that line.

Multithreading should be a selling point of Perl 6 and Parrot, but unfortunately is not yet complete and will not be in Rakudo Star. That is a great pity.

Perl 6 gets a release date!

2009-08-05T05:10:00.002+01:00

Big news from the current Perl conference YAPC Europe 2009 here in Lisbon is that Rakudo * (star) will be released in spring 2010. Rakudo is Perl 6 on Parrot, which is an equivalent to the JVM or CLI.

Is that an official Perl 6 release? Well, it is the nearest to official that you will see. The Perl 6 team are keen to point out that there will be no such thing as a official version, just different implementations. I guess they are looking at Python for their inspiration there.

This could be make-or-break for Perl. It shouldn't be, Perl 5 will continue no matter what, but the new stuff is what will make Perl cool again in the eyes of the fashion conscious. We all know that fashion is fickle, illogical, and driven more by hormones than intellect. We also know how important it is in decision making. Much as it would like to, the Perl community cannot pretend that Perl 6/Rakudo is too far away to worry about. It is six months away.

My worry is that expectations will be high - the release will be incomplete and they are very open about that. The Rakudo "pumpking" (project manager), Patrick Michaud, sees this as a test release to encourage people to write applications to find bugs and feed back into the development. Trouble is that companies have been burned too often in the past to invest in applications on an incomplete and possibly flaky platform.

The dilemma is that if something is not released soon then Perl will have missed the boat - which is already upping anchor and about to steam away.

Update: read the comments from Pm - they are very good.

Perl 6 Questions and further information

2009-06-24T14:14:00.002+01:00

This is following a recent Perl 6 seminar I gave.

Perl 5 style is still supported.

Named parameters go into %_
Unnamed parameters go into @_

Subroutines can be predefined with:

sub foo {…}

… is known as the "yadayadayada" operator.

Lexically scoped subroutines:

my sub mysub () is context { …}

Optional parameters: ? now comes after the parameter name.

sub mysub ($value, @harry, %ash, $opt?) {…}

Named parameters: prefix is now : , instead of + and are optional unless they have a ! suffix.

sub named ($value, :@harry!, :%ash, :$other) {…}

...

named(42, other = 'fred', harry = @files);

Slurpy parameters, like perl 5 lists and arrays, are prefixed *

C style prototypes
Are not supported. It appears that when a subroutine is called before it is defined the parameter resolution still works correctly. However a runtime error is reported, not a compile time error (as it would with Perl 5 prototypes).

Dynamic types
These are called "traits". See Synopsis 12: Objects.

Zip operator
Has changed, it is now 'Z'. Behaviour with different length arrays has also changed between pugs and Rakudo, the result uses the shortest:


my @array = qw(The quick brown fox);
my @brray = qw(Now is the time for all);

my @all = @array Z @brray;
say @all.elems();
say @all[];

@all = @brray Z @array;
say @all.elems();
say @all[];

Gives:


8
TheNowquickisbrownthefoxtime
8
NowTheisquickthebrowntimefox

Important further change

When testing these changes in Rakudo I ran up against a change I had almost forgotten. When calling a subroutine, no whitespace is allowed between the sub name and the open parentheses. That slowed me up until I remembered. A backward step IMHO.

MySQL VARCHAR limits

2009-06-23T10:09:00.002+01:00

A delegate recently queried the size limit on a VARCHAR field in MySQL. Our course documentation states a maximum of 255 characters: that is correct. For larger fields use TEXT.

Perl 6 Questions

2009-06-17T10:57:00.002+01:00

I recently gave a short talk on Perl 6, and a number of questions arose. I will answer those questions as soon as I can, but unfortunately other fires need fighting right now.

One of the reasons for the delay is that the syntax for subroutines for Perl 6 has changed again, and in fact my slides are now out of date. That also means that pugs and Rakudo are incompatible in a major way - they have always been out of step to some extent.

Moving into the light?

2009-04-24T08:24:00.004+01:00

A long time since my last post. Aside from bread-and-butter stuff, my activity has been on Python. There have been various upheavals in the business, and we aquired a Python 2 course. It wasn't too hot, and needed updating. Around the end of 2008 Python 3 was released, so I decided it would make sense to create a new course based on Python 3.

That is almost finished, but it turns out that it might have been premature, since no one seems to be in a particular hurry to move to Py3. That's a shame, because I am quite impressed. There have been issues, performance has been one of them (although I can't say I noticed any problems). In June 2009 a new version, 3.1, should be released, and that has a number of new features. So, having done all that work on a new course, I will have to hack it again come June. That's the problem with the bleading edge.

I have read of Python's ease of use, and never really believed it. After all, there are so many vested interests and religious wars that it is difficult to get a balanced view. I have to say that, after a little practice, Python is very easy to pickup. I have found myself just coding. The number of times I have to look something up gets less and less. I'm not sure I am fluent yet, but I'm close, and I reckon that is the fastest I have ever picked up another language.

Will Perl 6 be as easy?

App::sh2p Version 0.05 now on CPAN

2009-02-07T16:03:00.005Z

I always knew this would be a long project, although I'm heartened as to how far it has gone. It appears to have been useful to some people. It is a feature of Open Source that the only time I can be really sure it is being used is when I get bug reports - which could be rather off-putting to anyone without the skin of a rhino.

This latest release, version 0.05, fixes quite a few bugs, and adds support for things like trap (signal handling). In my defence, most of the bugs are missing support for certain features, and it is still in beta. I still have not got pipelines sorted out (that is a huge job), or $?, but they will come eventually. I only get to work on this in odd moments, evenings and so on, and it is difficult to find the time for development stints of more that two hours. I could really do with spending a couple of solid days on it.

One problem I had was in inserting a marker into a string which would indicate a token delimiter. Everything I tried clashed in someway with a real data item. I solved it eventually by using a gash reference, but I can't help thinking there must be a more elegant solution.

I have also started using OO techniques more - this product really does benefit from that.

I am now using the profiler and Coverage for my tests, but have not sorted out a good way to integrate my command-line tests. The "Perl Testing" book has ideas, but they require CPAN modules and I would rather not increase the number of dependencies. If you download this program and are dismayed at the tests (I basically check the thing will load) then be assured that I have a test suite of over 90 scripts, just one of which is my regression test script which currently has 55 test paragraphs within it. Each time I find a bug or add a feature I add a test, just like I should.

Shock! Horror!! A delegate ate my machine!

2008-12-19T08:28:00.002Z

OK, he didn't actually eat it, but he did exploit a "feature" of CentOS I was unaware of. The halt, reboot, and poweroff programs can be called from an ordinary user if that user is logged on to the console at the time. Our classroom users have very simple passwords, but the point is that these can be called from a non-root user.

I found a fix (with thanks to Mr. Google) but I'm sure there is probably a better one. Right now this will do, since it appears to work, but I'll try and find an alternative.

In /etc/security/console.apps there is a config file for the programs mentioned (and others). Set each one to something like (for example):


USER=nobody
PROGRAM=/sbin/poweroff
SESSION=true

Of course having better passswords would help...

Function redirection

2008-09-05T12:15:00.003+01:00

I just came across this:

function myfunc {

    echo 'Hollow world'

} > outfile

Works with Bourne style functions as well, and is in the POSIX standard. It should not be a surprise, since it just follows the same standard as 'if' and loops, but I have never seen it before. It certainly is not our shell course (yet). Seen it before?

Here is a neat trick:


outfile=one.out

function myfunc {
    echo 'Hollow world'
} > $outfile

outfile=two.out
myfunc

Which file gets written to, one.out or two.out? Interestingly it is two.out. In other words the variable is resolved at runtime, not function declaration time.

App::sh2p - shell to perl converter

2008-08-31T19:09:00.003+01:00

Just uploaded to CPAN, this has been a pet project of mine for a while (see perlmonks here ). It is currently only a alpha release, and you know what that means.

Converting UNIX shell scripts creates problems and my script (with associated modules) do not answer them all. The syntax of shell programs is very loose, almost anarchic. This is not a criticism of shells - they have a dual role to play in providing a usable command-line interface, as well as acting as a programming language. I target the POSIX shell, but there are many variations, not least those created by Sun. The Korn shell has moved on considerably from ksh88, and the syntax of the latest versions is complex. I make no attempt at the more esoteric parts of ksh93 syntax, expansions and extended globbing. There are annoying inconsistencies, fore example, what is the difference between:

[ $x -a $b ]

and

[[ $x -a $b ]]

I shall leave the reader to ponder (hint: -a has a different meaning to the Bourne-shell [ and the Korn shell [[).

As you might gather, this is not a simple syntax conversion. Take $? for example: that can be applied to any shell command you like and at any point in the program. Not many languages work like that, certainly Perl does not. Fortunately not many people use $? in shell scripts for anything other than testing the result of a child process - not unlike its use in Perl (there is a difference in the detail).

Right now App::sh2p attempts to remove some of the leg-work from conversion, and I hope t is useful.

Flip-flop operator

2008-07-10T09:37:00.002+01:00

I have been using the flip-flop operator in Perl for ages, equating it with sed and awk line addressing. However a question just came up that asked how we process from a specific line until end-of-file. In sed and awk the last line of the file is denoted by $, but that does not work in Perl for obvious syntax reasons.

I had to think about that: the solution is to use eof(). So, to list a file from line 20 we can use:


while (<>)
{
   print if 20..eof 
}

Promotion!

2008-07-10T09:00:00.001+01:00

When I signed on to Perlmonks this morning I got:

You have 24 votes left today.
You gained 16 experience points. They really complement your eyes.
Your new level imbues you with super-powers! Check out Level 13: Curate for the details!
You have 991 points until level 14 - Priest.

The trials of a CPAN author

2008-06-23T09:39:00.001+01:00

As mentioned previously I have a module called Win32::EnvProcess. The CPAN people (modules@cpan.org) don't like that name, although I'm not sure why. Changing a name is not clearly defined, and I don't get answers to email. Hey ho, it doesn't stop Win32::EnvProcess from being found and downloaded.

And that's the trouble with being a developer – people will actually try and use the stuff. So I get queries, and I get bug reports.

Generally the email queries are sensible, and I have had a few over the past week. First I, or rather a user, had an issue with Microsoft Visual Studio 2005, which is a version I haven't got around to using. It seems that good ol' Microsoft decided to fix the "DLL hell" problem by introducing a 'manifest' file into its link/load phase. So now we get "manifest hell" instead - the issues are well documented on the web. Of course the Perl MakeMaker does not generate a manifest file for its DLLs, and so my module wouldn't load and run. After some blind alleys and investigation I found a patch for MakeMaker, via perlmonks which I passed on. Of course that meant I had to document the thing as well. Meanwhile I found a small memory leak. OK, so that was worth it.

Then I got a bug report saying that the installation failed if the user did not have a C/C++ compiler. OK, I would have thought that if the product included an XS file, and the documentation showed compilation steps, then that would have been obvious. Clearly not, so more lines to the doc.

A user sent a "it would be great if…" feature request. "Good idea", I thought. "Simple patch", I thought. Only one of those thoughts was correct. Well you see it all worked on my Windows machines, it's just that it failed everywhere else. It wasn't until one of the cpan testers sent me a copy of his environment block that I cracked it. Their environment blocks were bigger than a hard limit I set – size matters. I hate hard limits and try and avoid them, but in this case I have to set one to a file mapping object. It's just that I was not handling the overflow case correctly.

There is nothing like a sustained bout of debugging to improve the diagnostics available in code. And of course I found another (unrelated) buglett as well – a possible race condition.

Then another bug report, this time someone is trying to compile the module under cygwin. Cygwin is a UNIX emulation environment, and this module is called Win32::EnvProcess – the clue is in the name. I check the OS name in the tests, but the thing won't compile and link anywhere except Windows. Of course not, RTFM. Alright, so Cygwin runs under Windows but it doesn't give access to the Windows kernel, it pretends it is UNIX.

Don't get me started about Strawberry Perl, just don't. Alright then. Strawberry Perl is a laudable package of open source products to enable you to install Perl without any dreaded Micro$osft licences. Good, great stuff, more power to them. Except that the Perl installation tools, like MakeMaker, doesn't really support it. So more magic in my Makefile.PL, already burdened by this stupid manifest issue. UNIX? Pah, that's easy.

Version 0.05 of Win32::EnvProcess has just been uploaded. God bless her, and all who sail in her.

Win32::StreamNames - new version!

2008-05-09T20:34:00.002+01:00

My CPAN module Win32::StreamNames had a bug - I know, unbelieveable. Happens to the best of us. The symptoms were that empty data streams were not returned. It is now fixed in version 1.03 which I just uploaded.

The bug was an unexpected (and undocumented) behaviour of the Win32 API BackupRead when it finds an empty stream, and I discovered the fix just by experimentation. Good old Microsoft, guarenteed to keep life interesting.

Thanks to Frederic Medico for reporting the error.

Yet another time calculation in Perl

2008-05-01T06:26:00.005+01:00

This is specifically for a delegate I am teaching right now. If I understand the requirements correctly, we hae two times in HH:MM:SS format and we wish to calculate the difference. I know, what if the times are on different days? No answer was given to that one, so I have assumed they are on the same day (today, to be exact).
This will probably change because the specification is a bit "wooly".


#!/usr/bin/perl

use strict;
use warnings;
use Time::Local;

sub str2time
{
   my $in = shift;
   my ($hour, $min, $sec) = split (/:/, $in);
   my ($mday,$mon,$year) = (localtime(time))[3,4,5];
   my $retn;
 
   # timelocal throws an exception on an invalid date/time
   eval {
      $retn = timelocal($sec,$min,$hour,$mday,$mon,$year);
   };
 
   return $retn;   # undef on error
}

print "Please enter the first time in HH:MM:SS format: ";
my $intime = <STDIN>;
chomp $intime;

my $time1 = str2time($intime);

die "Invalid time: $intime\n" if !defined $time1;
  
print "Please enter the 2nd   time in HH:MM:SS format: ";
$intime = <STDIN>;
chomp $intime;

my $time2 = str2time($intime);

die "Invalid time: $intime\n" if !defined $time1;


my $diff;
if ( $time1 > $time2 ) {
    $diff = $time1 - $time2
}
else {
    $diff = $time2 - $time1
}

my $mins  = int($diff / 60);
my $hours = int($mins / 60);
$mins     = $mins - ($hours * 60);
my $secs  = $diff - ($mins  * 60);

printf "Difference: %02d:%02d:%02d\n", $hours,$mins,$secs;

Away from the dark side - Python

2008-04-18T10:51:00.003+01:00

Our Python guy is off sick, so I've spent a week hacking the language. I had looked at Python a few years ago, but this time its personal.

So these are the impressions of a novice - and someone coming from Perl at that. I will probably come-up with justifications and solutions for the shortfalls as I learn, they are probably a mark or my ignorance rather than anything wrong with Python. Having said that, I was surprised how many things are missing.

The good
1. Forced code formatting
2. No stupid sigils
3. OO built-in
4. Default variables in a function are not global
5. Documentation
6. OS specifics are removed from the core
7. Excellent string handling
8. Intuitive file IO
9. Method calls on built-in types instead of weird system variables
10. No GOTO

The bad (IMHO)
1. No constants
Even True and False can be altered!
2. No scoping within loops and 'if' statements
All variables outside functions are globals
3. No interpolation - an effect of the lack of sigils I guess
4. Comparisons between numbers and strings give unexpected results, and NO WARNINGS! A poor and surprising feature
5. I know ++, --, +=, etc. are arguably poor coding, but I missed them.

Still, 10 good and only 5 bad.

Win32::EnvProcess

2008-02-13T13:40:00.002Z

YAPM (Yet Another Perl Module) which I just uploaded to PAUSE.

This module enables the user to alter or query an unrelated process's
environment variables.

Windows allows a process with sufficient privilege to run code in another
process by attaching a DLL. This is known as "DLL injection", and is used here.

SYNOPSIS

use Win32::EnvProcess qw(:all);

use Win32::EnvProcess qw(SetEnvProcess);
my $result = SetEnvProcess($pid, env_var_name, [value], ...);

use Win32::EnvProcess qw(GetEnvProcess);
my @values = GetEnvProcess($pid, env_var_name, [...]);

use Win32::EnvProcess qw(DelEnvProcess);
my $result = DelEnvProcess($pid, env_var_name, [...]);

use Win32::EnvProcess qw(GetPids);
my @pids = GetPids($exe_name);

This is another case where a question on perlmonks generated the interest. How do I get the child to create/alter an environment variables in the parent? Had this been on UNIX then the answer would be simple - you can't without having co-operating proceses. On Windows however DLL injection makes this possible.

Enjoy!

I have done a brief investigation on how this might be achieved on Linux, but I don't think it can fly. The API ptrace(2) is a start, but the problem is in creating a thread in the host. Addresses in the host can be changed, but the environment block is not at a fixed location so far as I know, and without a symbol table I'm not sure how you would find it.

Win32:IdentifyFile

2008-01-30T12:14:00.000Z

I just uploaded my fourth Perl CPAN module to PAUSE (the Perl Author's Upload Server).

This one was written because of a conversation on perlmonks Chatterbox. A module author was bemoaning the fact that Windows files do not have an inode number. I waded in to say that they do, kinda. However it's not easy to use, and requires a C API to be called. Hence the module, written using XS.

There are two functions, IdentifyFile() and CloseIdentifyFile().

IdentifyFile() returns 3 components to uniquely identify the file or directory. In list context these are returned as a 3 item list, in scalar context they are joined together as a single string using ':'. It is therefore simple to compare two identities in perl, using 'eq'.

Just getting this information could result in a race condition. The file could be deleted, possibly by another process, between getting the information and using it in a test. Worse, another file might be created with the same file index meanwhile. To prevent this scenario, files (or directories) are opened internally by IdentifyFile(), and kept open until CloseIdentifyFile() is called (files are not physically deleted until all open file handles are closed).

I wonder how many people think of that when using an inode number on UNIX?

UNIX command equivalents in Perl

2008-01-23T03:23:00.000Z

I realise I said I would post a new update in an ealier post. Well this is not it!
For the very latest list see my perlmonks post: UNIX command equivalents in Perl

First impressions in India

2008-01-21T02:28:00.000Z

On the road again, or rather "10 cramped hours in cattle class".

My hotel is a Marriott, not everyone's cup of tea, but I think they are excellent. In my previous job we always stayed in them, nowadays I rarely get the chance; usually the hotel is um, economic. Anyway, the luxury of the hotel is in stark contrast with the surroundings. Quite a culture shock, it is like living in a travelogue.

Stereotypical taxi from the airport, including the dangly bits on the mirror. Streets being brushed by doubled-up women holding rushes, and this at 1:30 AM. The reflective safety jackets were particularly incongruous. And yes, there was even a cow in the road.

The guards, sniffer-dog, and metal detectors at the hotel entrance were a bit of a shock. The helpful and polite staff are overwhelming. I think I have said "thank-you" more than any other phrase since I got here.

Walking out of the hotel on Sunday morning I was surrounded by tricycle drivers wanting to take me for a tour. When I politely declined they disappeared, and I was not bothered again. Just outside the hotel the poverty is heart-rending, yet the busy road ignores them. I have never seen anyone riding side-saddle on a motorbike while wearing a sari before.

(Windows) What is the difference between a thread's HANDLE and its ID?

2008-01-04T12:01:00.002Z

There seems to be some doubt amoung my peers over the use of a Thread ID, referred to here as a TID.

The most obvious difference between a HANDLE and a TID is that a HANDLE is specific to a process, whereas a TID is system wide.

A HANDLE is actually an unsigned int (whereas a UNIX file descriptor is a signed int) and is an index into the process's Handle Table, which is maintained by kernel.

A TID is a type of Client ID, the other type of Client ID being a PID (Process ID). TIDs and PIDs are generated in the same namespace and used by the kernel to identify these objects system wide.

So, why would you need the TID? You don't. At least, not often. A TID may be passed between processes, and another process can then get a HANDLE to that thread using the OpenThread() API.

Yes, you can manipulate threads in other processes - the caveat being that security may stop you, particularly in Windows Vista. Once you have a HANDLE to another thread you can do all those wonderful things, Suspend, Resume, Wait, GetExitCode, as you can if the thread is in the same process. One of the useful APIs in this context is PostThreadMessage(). This sentence is mis-leading, PostThreadMessage takes a TID, see comment from Hibou57 below.

BTW: make sure that HANDLE is closed, the thread cannot truly die until all HANDLEs to it are closed - that is how GetExitCodeThread() works. And you thought zombies only occurred on UNIX?

It is possible to use DuplicateHandle() to get a handle opened by another process, but that is far more painful than the MSDN implies.

Linux inotify example

2007-12-05T17:33:00.001Z


#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>
#include <sys/inotify.h>

void get_event (int fd, const char * target);
void handle_error (int error);

/* ----------------------------------------------------------------- */

int main (int argc, char *argv[])
{
   char target[FILENAME_MAX];
   int result;
   int fd;
   int wd;   /* watch descriptor */

   if (argc < 2) {
      fprintf (stderr, "Watching the current directory\n");
      strcpy (target, ".");
   }
   else {
      fprintf (stderr, "Watching %s\n", argv[1]);
      strcpy (target, argv[1]);
   }

   fd = inotify_init();
   if (fd < 0) {
      handle_error (errno);
      return 1;
   }
   
   wd = inotify_add_watch (fd, target, IN_ALL_EVENTS);
   if (wd < 0) {
      handle_error (errno);
      return 1;
   }
   
   while (1) {
      get_event(fd, target);
   }

   return 0;
}

/* ----------------------------------------------------------------- */
/* Allow for 1024 simultanious events */
#define BUFF_SIZE ((sizeof(struct inotify_event)+FILENAME_MAX)*1024)

void get_event (int fd, const char * target)
{
   ssize_t len, i = 0;
   char action[81+FILENAME_MAX] = {0};
   char buff[BUFF_SIZE] = {0};

   len = read (fd, buff, BUFF_SIZE);
   
   while (i < len) {
      struct inotify_event *pevent = (struct inotify_event *)&buff[i];
      char action[81+FILENAME_MAX] = {0};

      if (pevent->len) 
         strcpy (action, pevent->name);
      else
         strcpy (action, target);
    
      if (pevent->mask & IN_ACCESS) 
         strcat(action, " was read");
      if (pevent->mask & IN_ATTRIB) 
         strcat(action, " Metadata changed");
      if (pevent->mask & IN_CLOSE_WRITE) 
         strcat(action, " opened for writing was closed");
      if (pevent->mask & IN_CLOSE_NOWRITE) 
         strcat(action, " not opened for writing was closed");
      if (pevent->mask & IN_CREATE) 
         strcat(action, " created in watched directory");
      if (pevent->mask & IN_DELETE) 
         strcat(action, " deleted from watched directory");
      if (pevent->mask & IN_DELETE_SELF) 
         strcat(action, "Watched file/directory was itself deleted");
      if (pevent->mask & IN_MODIFY) 
         strcat(action, " was modified");
      if (pevent->mask & IN_MOVE_SELF) 
         strcat(action, "Watched file/directory was itself moved");
      if (pevent->mask & IN_MOVED_FROM) 
         strcat(action, " moved out of watched directory");
      if (pevent->mask & IN_MOVED_TO) 
         strcat(action, " moved into watched directory");
      if (pevent->mask & IN_OPEN) 
         strcat(action, " was opened");
 
   /*
      printf ("wd=%d mask=%d cookie=%d len=%d dir=%s\n",
              pevent->wd, pevent->mask, pevent->cookie, pevent->len, 
              (pevent->mask & IN_ISDIR)?"yes":"no");

      if (pevent->len) printf ("name=%s\n", pevent->name);
   */

      printf ("%s\n", action);
      
      i += sizeof(struct inotify_event) + pevent->len;

   }

}  /* get_event */

/* ----------------------------------------------------------------- */

void handle_error (int error)
{
   fprintf (stderr, "Error: %s\n", strerror(error));

}  /* handle_error */

/* ----------------------------------------------------------------- */

Just for fun

2007-09-10T13:22:00.000+01:00

Everone knows that the points system on Perlmonks is just a bit of fun, but it does give me a warm feeling to login today and get:

You have 20 votes left today.
You gained 15 experience points, isn't that nice?
Congratulations, you have been promoted to Chaplain!
You have 598 points until level 12 - Deacon.

That is from 272 writeups in about a year-and-a-half.

I wish there was a site like perlmonks for PHP!

Parallelism

2007-08-31T08:15:00.000+01:00

I have been attending an "Train The Trainer" (TTT) session at Intel in order to teach their new Parallel programming courses. An excellent week, surrounded by some real experts in the field. Several points stand out:

The "free lunch" for developers really is over, and the industry will not know what hit it. Software design and building is about to change, and it will be painful. Strangly, a programmer who had been asleep for 20 years and has suddenly awoke would not see anything new. That programmer might well be better equipped for the multi-core world than those that "kept current". Why? Because the skills practiced then will be even more important now, and the sloppy programming that has become so commonplace will no longer do the job.

Revealing comment:
The lead Intel instructor (an authority on parallel processing) was asked how people who have built everything on OO would cope with parallel processing. The answer was: "They are screwed!".

Now you might say that we have been doing multi-threading for years. Yes, but that is not parallel processing. Multi-threading is usually about splitting I/O tasks: GUI windows, overlapped comms and disk I/O are typical. Parallel processing is about splitting (decomposing) processor tasks, which is a whole new barrel of fish. At least, it is for most people.

This revolution has started, but is not yet in full flow, and won't be for a couple of years. The reason is that there is well enough work for an extra CPU with other processes, like services, and the multi-threading we have right now. The real need willl be with quad and 8 (oct?)-core processors.

Watch this space

The real jungle

2007-07-10T14:19:00.000+01:00

Just back from teaching Perl in Singapore. The real highlight for me, apart from teaching Perl of course, was the view of the Malaysian jungle on the flight in. The area looks tiny on the map, but it is vast. An amazing place.

I found Singapore to be hot and steamy, and I am talking about the climate. The place has still not shaken off its colonial past. Its own character does come through sometimes, and more power to it. The people seem to be friendly and open without the hang-ups of many large cities. Of course I was only there for a week and I only saw a small part of it.

As usual I looked at scripts that delegates brought in. I have also had a couple of scripts from other people recently. I see very common bad practices regularly:

No use warnings
No use strict
All production scripts should have these set.

Calling subroutines using the & prefix. Why do people do this? Because that is the way we used to do it in perl 4, and people do not update their skills. The & prefix ignores prototype checking, and passes the current value of @_ into the call if no other argument is specified. Don't do it!

Calling external programs instead of Perl built-ins. I have seen `date` used instead of localtime(), `cd $dir`, `pwd`, `mkdir`, `rm`, `egrep`, and so on. This is not just lazy, it is grossly inefficient. It is also a security risk, by the way. I published a list of UNIX commands and perl equivalents, I will try and update it soon.

The threading jungle

2007-06-25T22:07:00.000+01:00

Multi-threading has been around a long time. I first heard the term in the mid-1980s when discussing how ICL were going to host UNIX on VME. The first real implementation I saw was on Apollo NCS, a system that had many innovative features. ICL, VME, and Apollo have all gone, victims of acquisitions. Threads remain.

Multi-threading hit the mainstream with Windows NT, the system was built on it. The consensus around us UNIX hackers was that Microsoft had to encourage threads because their processes were so inefficient - no fork/exec you see. Probably utter twaddle; you know what UNIX hackers are.

Threads brought synchronisation problems. Of course they did. System programmers were no strangers to this type of issue, just the context of threads was new(ish). Way back when, on ICL VME, all we had were "test and set" and "decrement and set" instructions that were atomic. When I were a lad we had to build our own primitives, there was a particularly handy instruction that let you not only raise an interrupt (event/signal, whatever) but allowed you to send a two-word data block with it. Kids of today … don't know they're born… forty miles 't pit… etc.

I got to write real synchronisation primitives when I did the VME port of the MIMER RDBMS in the early '80s. There is nothing like having to write the damn things to understand how they work. I did not design them, I hastily add - that was done by some egg-head at Uppsala University. The problem then, as now, is not in using the APIs, it is in understanding how they interact with your application and (more important) how the application interacts with itself.

First rule when writing co-operating threads: things only go wrong at the worse possible moment. And that is usually after the thing has gone into production.

The Windows APIs are fairly easy to use, certainly compared to knitting your own. The main wrinkles in using them are a few peculiarities with the C/C++ runtime-library. I find it amazing that so many system engineers insist in using CreateThread instead of _beginthreadex. Back in Visual Studio 2.0 this was understandable, even I did it. The documentation was, um, "challenged". Now, with the MSDN, there is no excuse, the problems are well documented. Still people insist on using the wrong API. My theory, for what it is worth, is that a call to _beginthreadex, with its associated casts, looks messy on the screen, surrounded as it is by unrivalled elegance (sic). CreateThread looks good, _beginthreadex is ugly. For wotsit's sake! A thing of beauty with a memory leak is bad code, and I don't care what it looks like. Humph.

Still, you thought that was ugly? Try finding the thread-safe RTL functions in pthreads. Threads had to be retro-fitted to UNIX. Many fought against it, even the hero Torvold in the clone wars, but resistance was futile. Microsoft made life easy for us developers by having a whole special runtime-library just for multi-threading, ungrateful pups that we are. The pthreads implementations on UNIX have no such luxury. Instead there is a whole mess of different functions that are re-entrant with the _t suffix. Bye-bye portability. Finding which functions these are is a bit hit and miss, sometimes they are not in the man pages, and the standard only lists minimum requirements. Try teaching Condition Variables to a class already spaced out by mutxes. The word "predicate" makes their eyes glaze over. The fixed grin is the give-away: "Gone, solid gone", as Baloo would say. I'm sure Kipling was a coder, who else would write a whole poem about a conditional statement?

What I did on my holiday

2007-05-30T15:09:00.000+01:00

1. Read "Perl Hacks" again, and caught a bunch of stuff I had missed first time around. Like, wantarray returns undef in void context.

2. Read "Parallel Programming in C with MPI and OpenMP" by Michael J. Quinn
Very American (the first computer was ENIAC - really?) No mention of the ICL DAP (Distributed Array Processor) - one of the first commercially available parallel processors.
Nice to see terms like "Functional Decomposition" from the 1970s recycled to be used in a parallel computing methodology.
It really bugs me when authors come up with "Creating Arrays at Run Time" and give code like this:
int *A;
A = (int *) malloc (n * sizeof(int));
When there is a perfectly good feature in C99 to do it:

int A[n];

That's it! This is a C99 Variable Length Array, one of the more obvious features of the changes from C89. The book was published in 2004, so why no mention?
The author uses for loops to access arrays quite often, but there is no mention of optimisers, and the effect of optimisations on parallel processing. Ho hum.

Where have you been?

2007-04-05T13:54:00.000+01:00

Guilty. I haven't blogged for a long time.

What's happened?
Well for starters the company I work for got taken over, and that caused a lot of dust - mostly settled now.

I have been getting into the new Korn shell 93, and I hope to post something on that soon.

Just today I got my "Happy Monkday" from perlmonks - I have been a monk for a year. Good fun!

Oh where is Perl 5.10? Soon? Please!

Notifications

2006-08-31T17:34:00.000+01:00

This is a reminder post for (selfishly) me! Follow along if you like.
(CPAN) Win32::ChangeNotify - does it support (Win32 API) ReadDirectoryChangesW and overlapped IO? If not, how does it avoid missing changes?

(CPAN)Linux::Inotify (and Inotify2) support the Linux kernel API inotify which (perhaps) replaces dnotify. Supported from kernel 2.6.13.
Neither interface is on Fedora core 4, which is 2.6.11.

Further digging required.

Fun with Perl module loading

2006-07-24T20:37:00.000+01:00

Q. So exactly when does a BEGIN block get executed?

A. As soon as it is encountered!

For example, take the following code:


  use AModule;
  use BModule;

  BEGIN {
     print __PACKAGE__." BEGIN block\n";
  }

  use AnOther;

  $syntax_error = 42

  print "Starting main program\n";
  print "Ending main program\n";

The order of execution is:

   AModule BEGIN block
   AModule main body

   BModule BEGIN block
   BModule main body

   main BEGIN block

   AnOther BEGIN block
   AnOther main body

   syntax error at main.pl line 18, near "print"
   Execution of main.pl aborted due to compilation errors.

So, the main program's BEGIN block is not necessarily the last one executed. Granted, normally it is, since we usually use all our modules before the BEGIN block in main, but we don't have to.
Note also that the blocks are executed even though we have a syntax error, and the program fails to compile.

Q. What use is that?

A. We can alter the way subsequent modules are loaded

Usually that will be by altering @INC:


  BEGIN {
     if ( defined $ENV{TESTLIB} ) {
         unshift @INC, $ENV{TESTLIB}
     }
  }

Now all subsequent module loads will search the directory in the environment variable first.

Q. @INC is just a list of directories, Right?

A. Wrong! It can also contain code references.

References to subroutines found by the module loader will be executed as they are found:


  BEGIN {
     print __PACKAGE__." BEGIN block\n";
     my $code = sub { print "\@INC code\n" };
     unshift @INC,$code;
  }
  use AnOther;

Gives:


  main BEGIN block
  @INC code
  AnOther BEGIN block

Q. Is that it?

A. Of course not.

The subroutine is passed two arguments, the code reference itself and the name of the module it is trying to load. This enables us to track every module loaded. So:


  BEGIN {
     print __PACKAGE__." BEGIN block\n";
     my $code = sub {
        my (undef, $loading) = @_;
        my ($package, $filename, $line) = caller;

        print "Loading $loading from $package\n"
     };
     unshift @INC,$code;
  }

  use AModule;
  use BModule;
  use AnOther;

Gives:

main BEGIN block
Loading AModule.pm from main
AModule BEGIN block
AModule main body

Loading BModule.pm from main
BModule BEGIN block
BModule main body

Loading AnOther.pm from main
AnOther BEGIN block<>
Of course, further diagnostics could be added, like filename, line, and date/time. I'll leave that to you.

Q. So, @INC is more powerful than its sister %INC, which is just used for diagnostics.

A. Err, no. %INC is used by perl to see if a module is already loaded.

Q. And how is that useful?

A. We can force a module reload by removing its entry from %INC. Consider:
In one part of our code we want to force a different version of a module to be loaded (don't ask why).

  use strict;
 use warnings;                # DON'T use -w

 use A;
 use MyB;
 use C;

 A::mysub();                  # Original modules used

 delete $INC{'A.pm'};         # Force perl to reload

 unshift @INC,'mydir';        # Change @INC
 {
    no warnings 'redefine';   # No 'redefined' warnings
    require 'A.pm';           # Reload module
 }

 A::mysub();                  # Module in 'mydir' used

Work it out yourself!

Splitting whitespace

2006-05-04T02:34:00.000+01:00

The documentation for Perl is quite good usually, but when it comes to perldoc –f split there is so much magic involved that normal English begins to break down. Take the default arguments, for example.

"If EXPR is omitted, splits the $_ string. If PATTERN is also omitted, splits on whitespace (after skipping any leading whitespace)."

Then, later in the documentation:

"As a special case, specifying a PATTERN of space (' ') will split on white space just as "split" with no arguments does. … A "split" with no arguments really does a "split(' ', $_)" internally."

So, how many whitespace characters is that? A single space as a delimiter implies a single space is used for the split, but it actually does 'one or more whitespace':

   $_ = '    This    is    some    text';
   @a = split;
   $" = '|';
   print "@a\n";

Produces

   This|is|some|text

So leading whitespace is ignored, and one or more whitespace is used as a delimiter. There is a (documented) subtle difference with \s+:

   $_ = '    This    is    some    text';
   @a = split /\s+/;
   $" = '|';
   print "@a\n";

Produces:

   |This|is|some|text

Notice that the first element of the resulting list is empty, which was not previously the case.

A few questions arise from this. First, what is this ' ' syntax all about? Don't we need a regular expression match?

   $_ = 'xxxThisxxisxxsomexxtext';
   @a = split 'x';
   $" = '|';
   print "@a\n";

Produces:

    |||This||is||some||te|t

So it does work, except not exactly the same as a single space, it does not match 'one or more' (x+), so the space is magic. To be fair the documentation does say that ' ' is a special case. But the documentation does not show the syntax of a string literal, it specifically shows that an RE delimited with / / is required. Single quotes works with regular expressions, and with multiple characters (without a leading 'm'). But double quotes or other characters do not work unless preceded with 'm'.

Second question. What does whitespace mean? Is ' ' the same as \s in this case? Normally, of course, it is not, but in this case it is! ' ' is very special.

Inside-out accessor methods - version 2

2006-04-27T01:02:00.000+01:00

This solution uses a hash of hash-references, which means we no longer need the nasty eval. With thanks to the perl monks.


my (%speed, %reg, %owner, %mileage);
my %hashrefs = ( speed   => \%speed,
                 reg     => \%reg,
                 owner   => \%owner,
                 mileage => \%mileage);
                   
sub set {
   my ($self, $attr, $value) = @_;
   my $key = refaddr $self;
     
   if ( !exists $hashrefs{$attr} ) {
      carp "Invalid attribute name $attr";
   }
   else {
      $hashrefs{$attr}{$key} = $value;
   }
}

UNIX commands to Perl

2006-04-26T23:58:00.000+01:00

This is a rough conversion table from UNIX commands to Perl.
Please note that there are not always direct single equivalents.


UNIX         Perl                         Origin
.            do                           built-in
awk          perl ;-) (often 'split')     built-in
basename     File::Basename::basename     Base module
cat          while(<>){print}             built-in
             ExtUtils::Command::cat       Base module
cd           chdir                        built-in
chmod        chmod                        built-in
chown        chown                        built-in
cp           File::Copy                   Base module
             ExtUtils::Command::cp        Base module
date         localtime                    built-in
             POSIX::strftime              Base module
declare      see typedef 
df           Filesys::Df                  CPAN
diff         File::Compare                Base module
dirname      File::Basename::dirname      Base modules
echo         print                        built-in
egrep        while(<>){print if /re/}     built-in
eval         eval                         built-in
exec         exec                         built-in
             pipe (co-processes)          built-in
export       Assign to %ENV               Hash variable
             Env::C                       CPAN
find         File::Find                   Base module
ftp          Net::Ftp                     Base module
function     sub                          built-in
grep         see egrep    
integer      int                          built-in
kill         kill                         built-in
ln -s        link                         built-in
ls           glob                         built-in
             opendir/readdir/closedir     built-in
             stat/lstat                   built-in
mkdir        mkdir                        built-in
mkpath       ExtUtils::Command::mkpath    Base module
mv           rename                       built-in
             ExtUtils::Command::mv        Base module
od           ord                          built-in
             printf                       built-in
print        print                        built-in
printf       printf                       built-in
rand         rand                         built-in
rm           unlink                       built-in
             ExtUtils::Command::rm        Base module
rm –f        ExtUtils::Command::rm_rf     Base module
sed          s/// (usually)               built-in
sleep        sleep                        built-in
             alarm                        built-in
sort         sort                         built-in
source       do                           built-in
times        times                        built-in
touch        open()/close()               built-in
             ExtUtils::Command::touch     Base module
trap         %SIG                         Hash
             sigtrap                      pragma
typeset      my                           built-in
typeset –I   int                          built-in
typeset –l   lc                           built-in
typeset –u   uc                           built-in
typeset -Z   sprintf                      built-in

Sad....

2006-04-11T08:25:00.000+01:00

Walking across London I spotted a poster which screamed:
100 Gig tickets to be won!
"Must be a big stadium", I thought....

Accessor methods - update

2006-04-11T08:24:00.000+01:00

I have a much better solution, with the help of the perlmonks. I'll post it when I get chance

Accessor methods for inside-out objects

2006-03-23T15:54:00.000Z

I have just been looking at Class::Accessor, recommended by Simon Cozens at http://www.perl.com/pub/a/2006/01/26/more_advanced_perl.html. It automagically generates accessor methods, but appears to rely on a %fields type approach, and will not work on inside-out objects.
I got to thinking. Actually generalised accessors for inside-out objects are not that difficult, not need some ev[ai]l doings.


   my %speed;
   my %reg;
   my %owner;
   my %mileage;

   sub set {
      my ($self, $attr, $value) = @_;
      my $key = refaddr $self;

      my $hashref;
      eval "\$hashref = \\\%$attr";

      if ( !defined $hashref ) {
         carp 'Invalid attribute name';
         return;
      }
        
      $hashref->{$key} = $value;
   }

   sub get {
      my ($self, $attr, $value) = @_;
      my $key = refaddr $self;

      my $hashref;
      eval "\$hashref = \\\%$attr";

      if ( !defined $hashref ) {
         carp 'Invalid attribute name';
         return;
      }
        
      return $hashref->{$key};
   }

BTW: I also found how to do formatting: <pre>...</pre>

More of the same..

2006-03-08T15:57:00.000Z

I havn't posted for a while. It's not that nothing has happened, its just that too much has! I'm afraid that the blog suffers when I'm busy. I have been madly updating our 'Advanced Perl with CGI and Web Applications' course, and I am quite pleased with the result. As always I reckon I learnt at least as much as the delegates will (although maybe on more obscure subjects).
For example, did you know you can use __DIE__ and __WARN__ signal handlers to trap Carp::croak and Carp::carp calls?

I returned home from teaching in the USA last Saturday to my latest multiple order from Amazon:
Object Oriented Perl by Damian Conway. A little out of date (2000) but good nontheless, written in typical Damian style. Hacking the Advanced Perl course (two chapters on OO) and reading this book has totally changed my views on Perl as an OO language. The book does not contain anything on inside-out objects, but you can see that Damian was almost there. We now have a chapter on this simple but effective means of encapsulation in the Advanced course.

Number two book is the "Pickaxe book", Programming Ruby. About time I read it, more on that later (hopefully).

Number three is "PHP and MySQL Web Development". A huge tome which (unlike OOP and Ruby) does not look like 'fun'.
Last but not least another O'Reilly pocket book, this time the Linux one. All these pocket books, I need huge pockets. Not to mention another bookshelf....

What a refreshing change

2006-01-16T17:46:00.000Z

I'm on the West coast of Ireland, Galway, teaching. I don't think anyone will be offended if I say that Galway is a little remote.
The air and the Guiness are all refreshing, but what stopped me in my tracks is the co-operation between businesses. One major international company booked me to give a course, but then said to others in Galway, "Hey, we have a technical course running this week, anyone else interested?" , and so along comes a few guys from other companies that could not justify the course on their own. Apparently this happens all the time.
These companies are major internationals acting like sensibile human beings. Now why can't that happen everywhere?

IFS and read

2006-01-04T14:19:00.000Z

I find myself continually underestimating the Korn shell. I always thought that Bash was neat because we could read into an array with read –a, but of course we can do the same in ksh with read –A. The effect is to split up the input into fields around $IFS. But did you know we can set IFS just for a read statement? Look carefully:

IFS=','
while IFS=':' read -A line
do
end=$(( ${#line[*]} - 1 ))
if [[ ${line[$end]} != /sbin/nologin ]]
then
echo "${line[*]}"
fi
done < /etc/passwd

(display all the lines in /etc/passwd whose last field is not /sbin/nologin, changing the field delimiter from colon to comma).

The first IFS=',' sets IFS for the echo expansion and is not overridden by the IFS on the read line. That IFS only applies for the read, nothing else.

By the way, did you know that the expansion (including $*) uses the first character from IFS, so order matters? So IFS=',.:' gives different results to IFS=':,.'.

All this applies to the standard POSIX shell, as well as Korn shell and Bash, and I was alerted to these features by Classic Shell Scripting (see a previous post).

Finally, a significant bug in pdksh has been fixed in the version with Fedora core 4 (1993-12-28 q). In early versions a pipe generated a child process on each side, so piping into a while loop was not useful, since the while loop ran in a different process to the rest of the script, and so variables could not be saved. Now it all works, consider:

#!/bin/ksh

ps -ef | while read uid pid ppid c stime tty time cmd
do
if [[ $cmd == *xinetd* ]]
then
break
fi
done

echo "Pid of xinetd is: $pid"

Unfortunately this will still not work in Bash, for the same reason as it did not work in the old pdksh. In Bash we can use process substitution instead, and while we are about it we may as well use an ERE (because we can):

while read uid pid ppid c stime tty time cmd
do
if [[ $cmd =~ '^xinetd +' ]]
then
break
fi
done < <(ps -ef)

echo "Pid of xinetd is: $pid"

The man pages for the new pdksh say process substitution works in Korn shell as well, and I am told this works on Solaris, but it does not appear to work on Linux.

Toys for boys

2005-12-30T12:03:00.000Z

The weather has been kind over Christmas, particularly clear skys for my *new* Meade ETX 105 - that's a telescope by the way. Not exactly the largest - only 4" and yes, size does matter, but I couldn't really justify a larger one given that it usually rains on the rare occasions when I am actually at home.
Good views of Mars in November and early December, now good views of Saturn, and various nebula like M42 and M37. OK, so what's that got to do with programming and stuff?
Well, most modern telescopes have an electronic control called a "goto" system - you get the idea. This is normally controlled by a hand-held device, but can be controlled by a PC using an Open Source object model named ASCOM - Astronomy Common Object Module.
http://ascom-standards.org/faq.html
Currently it appears to be written using COM, which of course is proprietry and realistically only runs on one of the many Microsoft operating systems. OK, you can get it to run on others, but would you want to, really?
This is one I will be looking at further.

Switching off terminal echo in Perl

2005-12-21T14:38:00.001Z

A delegate recently asked me how to switch echo off for password prompting. I couldn't remember, I thought it was Term::ReadLine. Actually it is Term::ReadKey - as expected the Perl Cookbook reminded me. Term::ReadKey is not a base module, but is present on ActiveState and Fedora installations, and easily installed from CPAN. Here is a simple script that works on both Linux and Windows (despite what the documentation says) :

use Term::ReadKey;

print 'Password: ';
ReadMode 'noecho';
my $password = ReadLine; # Corrected
ReadMode 'normal';
chomp $password;

print "Password was: $password\n";

Another delegate said something like "I guess another module will replace characters typed with asterisks". Well, that is not so simple. After some experimentation I came up with the following script, which has to use raw terminal input and take iinto account differences between Windows and Unix/Linux:

use Term::ReadKey;
use strict;

local $| = 1;
print 'Password: ';

my $password = '';
my $retn = $^O eq 'MSWin32'?"\r":"\n";

ReadMode 'raw';

while (1)
{
my $key;
# Loop waiting for a key to be pressed
while (!defined $key)
{
$key = ReadKey -1;
}

last if $key eq $retn;
$password .= $key;

print '*';
}

ReadMode 'normal';
print "\n";

print "Password was: $password\n";

Check the Term::ReadKey documentation for the arguments. Note that I have to wait for a key to be pressed this time, and write an unbuffered '*' (local $| = 1 makes STDOUT unbuffered).

Perl subroutine parameter passing - again

2005-11-16T08:06:00.000Z

One of the reasons Damian Conway does not like prototypes is that (his words) they do not give the results expected. Well, what results do you expect when you don't use them?

It is sometimes said that parameters are passed into Perl subroutines by copy, and some say they are passed by reference. In fact they are passed by magic, and as we all know, any technology sufficiently advanced is indistinguishable from Perl.

What do you expect to be printed?

sub mysub
{
@_ = qw(one two three)
}

@array = qw(The quick brown fox);
mysub (@array);
print "@array\n";

We get 'The quick brown fox '. The array is unchanged, so you might think the array is passed by value. The call:

mysub (qw(The quick brown fox));

also works fine, so no problem. Let's change the subroutine to use a slice instead:

sub mysub
{
@_[0,1,2] = qw(one two three)
}

Now we get 'one two three fox', the array is changed! Changing the elements changes the caller's array, changing the whole array does nothing! Now the line:

mysub (qw(The quick brown fox));

fails with "Modification of a read-only value attempted…"

This is a run-time error, whereas if you use a prototype:

sub mysub (\@)

we get a compile-time error:

"… must be an array (not list)…"

If I preferred run-time over compile-time errors I would be using the Korn shell.

Once the prototype has been specified it does not matter if we alter the caller's array in total or use a slice – the caller's code is always the same.

OK, I admit that altering @_ direct is damn awful code, and it breaks another of Damian's rules. He's not all bad ;-)

I don't believe that an inconsistency in one part of Perl (prototypes) is a justification to switch to an inconsistency in another (pass by magic). Just admit the inconsistencies and celebrate them – or move to Perl 6.

Perl 6 at EuroOSCON

2005-10-20T06:52:00.000+01:00

This is a brief summary of my notes from the Perl 6 and pugs sessions. I have only included the stuff that was new compared to the Perl 6 Appendix in the 'Perl Programming' course. I might have repeated myself.

use strict and use warnings are on by default;

Quotes are only required around a hash key when {} are used.

$hash{'key'} becomes %hash

Barewords are not allowed. Ever. Even for file handles.

fail{…} warn{…} die{…} throw a 'not yet implemented' type warning – the Perl 6 developers must be using that a lot ;-)

Many improvements to interpolation:

"{ executable code like a do block} "

Subroutine calls are allowed: "&mysub(args)"

Interpolate an array: "@array[]"

Interpolate a hash: "%hash{}"

sprintf is probably obsolete, printf definitely is:

print $var.as('%6.2f');

print $hash.as("%-20s: %2.6f", "\n");

Control can be made over exactly what is interpolated:

qq:c(0)/ / # don't interpolate

qq:s(1)/ / # only interpolate scalars

qq:s(1):a(1)/ / # interpolate scalars and arrays

hummmmmmm

Here document syntax is changed:

my $var = q:to /END/ # s(1):a(1):h(1) can be added

…

END

Not sure where the semi-colon goes

Ranges have a new feature:

x..^y means from x up to y-1

x^..^y means from x+1 to y-1

To open a file with an automatic chomp::

my $fh = open $filename

This can be overridden.

while (<$fh>) {…}

becomes:

while = $fh {…}

while (<>){…}

becomes

while = $ARGS{…} or while =<> {…}

=<> is known as the 'fish' operator

There is now a 'prompt' verb

Most built-ins that operator on scalars and arrays are now methods:

$var.substr(…) # returns the substring

$var .= $var.substr(…) # changes the substring

No need for Data::Dumper, instead:

$var.perl()

And there is much more, Damian could not complete his talk because of time, and it is not downloadable. I'll keep you posted as I play with pugs.

EuroOSCON - whatever. Is Perl 6 too complex?

2005-10-19T19:52:00.000+01:00

I asked the main man, yer actual Larry Wall, this question. Poor guy - he is a very nice chap - thought he was just signing a book for me. There was no one else around, so I consulted the oracle (I won't use an uppercase O, you'll get the wrong idea).

Q. Will Perl 6 still be suitable for guys who just want a better language that ksh or .bat files?
A. The easy things will still be easy

Q. Will Perl 6 still be suitable for sys.admin's?
A. Hey! I'm still a sys.admin.
Me: No you are not, you are a language demi-god.
A. Shrug. Sys.Admins will still be able to use it.
Q. Are you saying that because that is what I want to hear?
A. Sys.Admins will still be able to use it.

Thank-you very much.
We also discussed training strategies for Perl 6, and even for Perl 5, and I wasn't that hard on him. Larry Wall is a very nice chap (oh, I said that).

EuroOSCON 6 - Perl 6 Update

2005-10-19T19:25:00.000+01:00

Well I did type all this in, but the network broke when I sent it, and was lost. You will have to wait.

EuroOSCON 5 - Installation Installation Installation

2005-10-19T18:35:00.000+01:00

I saw a demo today of a brilliant web-based Perl debugger called Devel::ebug::HTML. It uses the browser, and appears to have everything you want from a visual debugger.
BUT...
Trying to install it is a nighmare. There is no README file, so no list of dependancies. I start downloading and installing them by hand and choke when I come to Catalyst. So I switch off my firewall (oooooooh) and run cpan, and nothing installs. Tried ppm - the same. Everything downloads OK, but every single one fails on the make - it should be using nmake on windows.
Doing this manually would take for ever, so I give up.

Other Open Source products seem to suffer from the same thing, and it is about time the community did something about it. OK, that means you and me. Something is being done about it, there are many people working on this. I appreciate the complexity, but I don't think the underlying issue is technical. It is trying to bring order to the bazaar, and I am not sure that is possible without turning it into M&S.

EuroOSCON 4 - 'Ere we go, 'Ere we go, 'Ere we go!

2005-10-18T16:33:00.000+01:00

Various speakers and sessions attended (more tonight), here is a summary:

O'Reilly

Book sales year on year current figures:
C# down 3%
Python up 30%
VB down 1%
Perl up 4%
Ruby up > %1000
Linux(generic) down 4%
Read Hat down 24%
Other Linux/Unix up 2,807%

Now does that tell several stories?

Waugh

Microsoft dialog boxes: Q. "Would you like tea or coffee?

The other talks were just saying what a wonderful thing Open Source is, and how it is the future. Yeh Yeh Yeh.

Perl 6

Lots of detail, I have to sort out my notes in detail - another post tomorrow.
Damian and Larry's presentation was titled "Perl 6: End-game"
Damian Conway is a trainer, and is giving 2-day Perl 6 courses NOW, and expects to be giving full blown Perl 6 courses next summer.

Have more sessions this evening, tomorrow is quieter.

EuroOSCON 3 - Perl Best Practices (2) - prototypes

2005-10-18T05:52:00.000+01:00

Like the true geek I am, this has kept me awake all night.

Damian Conway is dead against prototypes, and says they should all die a horrible death in a pit filled with spikes.
And snakes (I forgot about the snakes).

Prototypes on $@% are rubbish – I agree, but \$\@\% have some advantages.

Take the swap_arrays code in his book (p.196), let's think of some simple user mistookes:

Without prototypes:

swap_arrays(@sheep, @goats); # Missed out the \

Without prototypes, provided you are using strict, you get:

Can't use string ("Blackface") as an ARRAY ref while "strict refs" in use at …

("Blackface" is a sheep, we know about these things where I do live)

BUT the error is given as being in the subroutine, not at the call (where the actual error is). perl also stops right there, it does not detect that the next argument is wrong as well – we need to do another run to find that.

Mostly if we supply the wrong type, use strict 'refs' will save our bacon (or mutton). BUT:

swap_arrays([qw(one two)], [qw(three four)]);

does not give us an error, surprisingly. OK, at the end of the day we should not be calling a subroutine we do not understand, but if a user can, she will. If she thinks that the two lists are flattened (Damian's assumption with prototypes), then she clearly does not understand the thing anyhow.

With prototypes:

swap_arrays(\@sheep, \@goats); # Added a \

With a prototype we get an error for each argument at fault:

Type of arg 1 to main::swap_arrays must be array (not reference constructor)

Type of arg 2 to main::swap_arrays must be array (not reference constructor)

Execution of C:\proto.pl aborted due to compilation errors.

This time the error is reported correctly.

Also, with:

swap_arrays([qw(one two)], [qw(three four)]);

and a prototype we get a similar error message, again identified with the correct line number.

And another thing

And I nearly forgot, empty prototypes are the only way I know (other than hand-coding) of preventing any arguments being passed, as in:

sub void_sub () { ,,, }

Calling this subroutine would give an error at the point of the erroneous call.

The Volcano beats Joe

Let's face it though, Damian is right. If you want a standard there cannot be exceptions. The problem with prototypes is that everyone thinks they are like C/C++ prototypes, but most people don't even understand THEM. Take:

size_t myfunc (char string[4]);

int another ();

Neither do what most people think they do, and are misleading. So how do you expect to understand Perl prototypes on the same (false) basis? KISS.

EuroOSCON 2 - Perl Best Practices

2005-10-17T16:39:00.000+01:00

Damian Conway's talk on this subject woke me up.
A lot of great ideas and tips, methinks I need an extra slide at the end of each chapter on style, or something like it. Do you think people will mine a six day course? That's the problem, I keep picking up more stuff to put into the course!

At least I found something to take out - subroutine prototypes.
A packed room, with lots of experts, and it appears I was the only person who understood prototypes (except Damian, of course). That is scary, and probably indicates we sould not spend so much time on them in our course. Damian hates them, I just find them 'flaky' personally - but no more flaky than lots of other parts of Perl. Then again, it is Damian's mission to destroy flaky Perl, and we have to start somewhere...

I got a chance for a brief chat with Mr. Conway (he comes from that country which just lost the ashes), but I have more questions for him. Here is a reminder for me:

Why use -t *STDIN instead of STDIN?
Shouldn't IO::Prompt offer the locale specific Y/N?
An idea for a pragma - RegExp::NoCapture - make all parenth. groups non-capturing. Could REs be optimised better?

Some good speakers tonight - Damian Conway (again) and Larry Wall - really looking forward to that.

BTW - am I grateful for downloading FireFox? The W/Lan here does not work with MS Internet Explorer. Apparently it is the same problem as I mentioned with ActiveState (see below), Or so they say.....

EuroOSCON 1 - DBI

2005-10-17T11:42:00.000+01:00

This conference is so cool, it is Arctic.

Went to the Database Tutorial this morning - another chapter to re-write! Not that there is anything wrong with the Perl Programming Appendix on DBI, but I just learnt so many cool things, that the chapter will probably double in size.
So what did find out?
First, my Perl knowledge is not so bad ;-) It is always worrying going up against a bunch of enthusiastic geeks, but hey, I don't think I let the side down. That was a confidence boost (even lecturers need that sometimes)
Next: dbi:DBM: is a prototyping database bundled with DBI, looks great for quick training demos
Next: queries can be done against a dbh, but this is not efficient (we don't mention that)
Next: if $sth->disconnect is not done, you can get a memory leak
Next: placeholders are very cool if you know how to use them (I feel lots of example slides coming on)
Next: I had missed dbh attributes before (set in the connect)
Next: Lots of things about RaiseError
Next: $sth metadata hashes are tied, so using these invoke method calls
Next: and other stuff as well

A lot I already knew, like American jokes don't work on a European audience.

Fedora 4 /etc/profile

2005-10-14T08:14:00.000+01:00

Found a buglet in this file this week.
The /etc/profile file is the first file executed by a Korn shell session on login, but it has been hijacked by Bash. On Linux the Korn shell is not used much, except by training companies of course. I doubt that /etc/profile is ever tested with ksh, so it is not surprising there is a bug, however it is a shame that the mistake is such a simple one - doesn't give one much confidence.

On sign-on to ksh we get a syntax error in /etc/profile, on the line:

if [ $EUID = 0 ]; then

The EUID shell variable is set by Bash, and is a read-only value giving the effective user-id (if you want to know the difference between effective and real user-ids, come on QAs Unix Programming course). The code is checking for a 'root' user. On Korn shell this variable is not set.

The fix is quite easy, quote the variable:

if [ "$EUID" = 0 ]; then

ALWAYS quote variables in single [. One of the effects of using [[ is that quotes are not necessary around a variable, and if the authors had come on QAs Mastering BASH Shell Scripts course, they would have known that.

Sneaky bat files

2005-10-04T16:42:00.000+01:00

The ActiveState Perl implementation contains some examples of imbedding Perl inside a Windows .bat file. Of course if we wanted to do that sort of thing in a Unix shell we would probably use a 'here' document, but bat files do not have that level of sophistication.

The structure of these bat files is as follows:

@rem = '--*-Perl-*--

perl -x -S "%0" %*

goto endofperl
@rem ';

#!/usr/local/bin/perl -w
#line 9
... Perl code ...
__END__
:endofperl

The "%0" is the name of the current bat file, and %* is a list of the arguments passed into the bat file (like $0 and $* in the Korn shell and Bash).

So we are executing the bat file as if it was a perl script! This is where the –x option to perl comes in. Check perlrun, -x means that the perl code starts at the #! line, and preceding lines are to be ignored. The –S option means that %path% is used to find the perl script. There is a problem with this.

The quotes around "%0" would appear to protect us from imbedded spaces in the filename, but they do not. Paths with imbedded spaces do not work with this method. Strangely, this works if the quotes are omitted. The implication is that most of the ActivePerl bat files do not work with imbedded spaces in path names.

(The pugs Windows installation has a similar problem)

The –S option is not really needed. This is because to run the bat file we need to specify a hierarchic or full pathname anyway (Explorer makes the script's directory its current directory).

The sneaky part is the use of @rem.

The bat file starts with a non-echo'ed remark:

@rem = '--*-Perl-*--
… bat file stuff …
@rem ';

The idea here is that, by a happy coincidence, @rem comment prefix in a bat file is a valid perl command, to assign to an array called @rem. The assignment ends on the final comment, with a close quote (notice that the first @rem has no close quote).
I can only guess that this is in the bat files for historical reasons, since the –x option to perl makes this trickery unnecessary. The goto (whatever that is) is required to leap over the perl code.

The #line directive sets the line number when error reporting, or for the die statement. This means that we can set the line numbers relative to the start of our bat file, rather than the start of the perl code.

Finally, the __END__ label tells perl that this is the end of the script, and ignores any junk that follows.

A link of interest

2005-10-04T12:10:00.000+01:00

Will Linux Benefit from Microsoft's SNAFU in Massachusetts? by Tom Adelstein -- Microsoft had everything to lose in Massachusetts and did.

Unix Shell Scripts

2005-09-23T14:34:00.000+01:00

A weekend between giving courses, thankfully back to talking Perl next week.
I have been teaching our Korn shell course this week and I enjoyed it, but....
Why do so many people think that Korn shell and Bash are the same?
Bash is hugely more powerful than ksh, with Extended Regular Expressions in version 3, and possibly associative arrays soon (looking at the source, it seems most of the code is already there). There are still things missing, for example ERE matching is supported, so wouldn't a substition statement be a small further step?
Yes I know, danger of feature bloat. With a little more work Bash could be the new Perl 5 ;-) There might actually be a place for that once (if) Perl 6 takes hold (double smiley).

Funny how conventional programmers tend to look down on shell scripting. And funny how system administrators, and their ilk, are expected to write scripts with no training on good programming techniques and practice. Companies spend huge amounts making sure their applications are fast and efficient, then some poor erk runs a badly written script and screws the entire system. Is it the erk's fault? Maybe, but more likely the person who put them in that position.

Here is my contention: anyone can write a shell script, but it takes skill and knowledge to write a good one.

I recommend: Classic Shell Scripting by Robbins and Beebe (O'Reilly) May 2005. The best book on the subject I have come across.

Redemption

2005-09-08T23:50:00.000+01:00

Microsoft, good ol' Microsoft, have a bunch of (useful) Perl scripts on:
http://www.microsoft.com/technet/scriptcenter/scripts/perl/prlindex.mspx
They only use Windows Management Instrumentation (WMI), and they are a bit thin of explaination of the code, but it's a start. One day, maybe, I will get to write my "Perl Programming for Microsoft Windows" course.

Bye bye IE, hello Firefox

2005-09-07T00:13:00.000+01:00

After downloading ActiveState Perl 5.8.7 I had problems displaying the html Perl documentation. Turns out (http://bugs.activestate.com/show_bug.cgi?id=39240) that the Internet Explorer patches with XP SP2 "locks-down" some features, like actually displaying web pages. Sorry, what did you say a browser is for?

I'm all for security, but preventing access to web pages on the local machine, even when it is not connected to the web? Give me a break!

So, hello Firefox! I used it months ago, but never got around to installing it on my nice new machine. Wel now I had a real need, and it works a treat! I am just ashamed I took so long.

So Mister Microsoft, if you want to know why I am not using your product, it is simply because it does not work - nothing personal ;-)

Win32::StreamNames

2005-08-31T16:32:00.000+01:00

A truly excelent new module has just appeared on CPAN, named Win32::StreamNames. It returns a list of file stream names for a specified file.

OK, I'll come clean. Yes it's one of mine. I was proudly showing off my blanket trick (see below) to another QA instructor when he said "I suppose you use a stream". Well, aside from a small river, a stream to me is an IO channel, or maybe a Unix device driver STREAM. This is neither. Enhanced at NTFS5 at Windows NT 5, correction, Windows 2000 ;-), a Windows File object points to Stream control blocks, which can include a Named stream. The names are of the form file-name:stream-name:$DATA. They are not normally visible to Windows Explorer, the dir command, Perl globbing, or readdir (using Win32 API). BackupRead Win32 API can retrieve the stream names, although it is not easy to use. Hence the module.

See it at your local CPAN mirror today! http://cpan.search.org

Blanket

2005-08-19T07:49:00.000+01:00

While on the train, I decided to ammuse myself by writing a version of Damian Conway's 'bleach' program. Basically it "encrypts" a perl script with whitespace, so when it is viewed it looks blank.
The method I used (I couldn't remember Damian's exact code) is to take the ordinal number of each character in the character set, and write that number of space characters to the file.

For example, 'A' is decimal 65 in the character set (ISO Latin 1 and ASCII), so for that character I write a line contining 65 spaces. To get the number I use the Perl built-in function 'ord'. To decode I just have to get the length of each line and feed that into the 'chr' function to get back the original.

I showed this to my class as an ammusement, and one delegate (thank-you Daniel) played around with it. He discovered that if you zip the blanked file you get a very small archive. Smaller (about two-thirds) the size of the original, zipped. Must be the way the zip algorithm works with repeated characters.

I have just thought of another tweak I can do .............

pdksh integer arrays

2005-06-04T10:56:00.000+01:00

Just found a bug in pdksh version 5.2.14 – it cannot handle integers arrays correctly (pdksh is the Public Domain Korn Shell). I first discovered this with a lottery script, but actually it is very easy to reproduce:
typeset –i array
array[0]=42
array[1]=43
array[2]=44
array[3]=45
echo "${array[@]}"

gives: 45 45 45 45

This works correctly if typeset –i is not used. Bash, with declare –ia array, works fine, as you might expect.

Unix for Perl Programmers

2005-05-27T20:30:00.000+01:00

This is an introduction to using the Unix API from Perl.

It concentrates on POSIX compliant system calls, and introduces System V Release 4 Unix. I look at the fundamental concepts of a process and its interaction with the Unix kernel, the difference between library calls and system calls is discussed, and I give an overview of error handling. Later postings will look at API calls in detail.

I have assumed a degree of familiarity with Perl that can be obtained by attending the QA Perl Programming course.

Unix compatibility

Unix has a long and chequered history, with a number of standards which often conflict. An attempt to standardise application code interfaces into Posix 1 (IEEE Std 1003.1, now the X-Open SUS) has not been universally adopted.
Use defensive programming techniques where possible, for example do not hard-code system limits, but call functions at run-time to find what these are (see later).
As Perl programmers we are protected from many of the differences between operating systems, but we are not immune from them. Most Perl statements are portable, but the lower we delve into the innards of Unix, the less portable we become. The differences are often expressed in C terms, even in the Perl documentation, so a knowledge of that language is very beneficial.
Use the built-in Perl variable $^O to mark blocks of code you know are specific to certain versions of Unix.
In this article I often give example code and function prototypes. Arguments and their types have been described in a generic way where possible, however you should not assume that they exactly match the systems you use, or the version of Perl.

Perl with Unix: call interfaces

The kernel is the heart of a Unix system. Its job is primarily one of resource management; it looks after the physical resources (hardware) of the computer system and shares them amongst all of the processes that wish to use them.
The kernel can be thought of as providing a number of services to programs, such as a file system for data and program storage, management of the RAM in the computer and a means of accessing devices attached to the computer. The details of the actual hardware are hidden from programs by the kernel; it presents a "virtual-machine" interface to the programs. The virtual-machine interface is consistent across almost all versions of Unix, regardless of hardware details.
Programmers will occasionally require the kernel to perform some function for them, such as performing input or output, or starting a new process. Such requests are made through a number of well-known special function calls known as system calls. The set of system calls defines the interface to the kernel for programs and is kept consistent as much as possible across different variants of Unix. Nevertheless, such low level access is by definition dependant on the operating system, and often requires detailed knowledge. To enable portability, and to supply easy-to-use interfaces, a higher level language like C provides its own RunTime Library (RTL). This takes an ISO standard call and converts it into an operating system specific call, and maybe adds other features as well. Other languages and products have their own RTLs, for example an RDBMS will have a RunTime Library to enable embedded SQL to run.

Perl is written in C, so uses the C RTL, and many of the built-in function calls reflect that. Sometimes these are too general, so we have available lower-level interfaces to make direct calls to kernel APIs, but remember that these were designed to be called from C, not Perl. Perl modules often make use of low-level calls, although they sometimes "cheat" by using imbedded C.

System Calls and Library Functions

Most Unix programs will need to make use of one or more library functions or system calls. While they may both appear to be the same from the program’s point of view, there are some important differences between library functions and system calls.

System calls are the means by which a process can request services from the kernel. Services include accessing and manipulating files and devices, and process-related operations. System calls are documented in section 2 of the Unix reference manual, and are platform specific. Some Perl built-ins make system calls, and appear to be the same as the kernel call. The read() function is an example. The Perl version will make a call to the C library function fread(3) eventually, but along the way adds value. It can handle utf8 files, for example. If we want the kernel function read(2) we need to call the Perl function sysread (there is also a syswrite and a sysopen). Here are some examples:

Perl Kernel call Library Function

$$ getpid
fork fork
open fopen
read fread
sysopen open
sysread read

Library functions are simply C functions written by someone else and supplied via libraries of functions. These libraries are linked to perl either statically (before the program runs) or dynamically (at run time). Using a library function does not necessarily require any help from the kernel.
C Runtime Library functions are documented in section 3 of the Unix reference manual, but Perl does not necessarily use them. The Library functions themselves can have portability problems, so Perl often implements its own. See perldoc perlclib.

The major portion of the work carried out by the various bodies that have developed standards for Unix, such as the IEEE (POSIX) and the Open Group (X/Open Portability Guide, Unix 1170, etc.) have been to standardise the names, arguments and results of library functions and system calls. The result is that it the various versions of Unix that exist today should all contain a compatible set of these, forming a standard API supporting the development of applications that can be ported with minimal efforts.

Perl system interfaces

There are a couple of different methods that may be used to imbed C calls into Perl. The first is the syscall interface. This allows a call to be made direct to a C function, assuming the function's prototype is defined in a header file. The h2ph utility will convert the header file to a Perl .ph file, which can then be loaded with use or require. Unfortunately the h2ph utility cannot cope with all structures and calls, so sometimes the resulting .ph files have to be "tweaked" manually to get them to compile. The most difficult part of using syscall is in converting the variable types between Perl and C, using pack and unpack. A detailed knowledge of C is required for this, as well as an understanding of the function you are about to call.
The XS interface is much easier to use, but still needs an understanding of C. Again we convert the header file, but this time using h2xs. We imbed C code in an XS code block, then build a new module using the XS compiler (called xsubpp). This time the conversion between C and Perl variables is done automatically.
There is a simple XS tutorial available in perldoc, called perlXStut, and the full XS documentation is in perlxs.

Converting between Perl and C

Perl variables are held in an "internal" format, and do not usually have a 1:1 mapping with C primitive types. However, many low-level functions make C system calls, and so we often need to supply a C struct from Perl.

We convert from Perl to C using the built-in function pack:

C scalar = pack TEMPLATE, LIST

where TEMPLATE indicates the type of each field (see below).

To convert from C to Perl we (unsurprisingly) use unpack:

Perl list = unpack TEMPLATE, SCALAR

There are over 30 different template characters, some common ones are:

A A text (ASCII) string, will be space padded.
Z A null terminated (ASCIZ) string, will be null padded.
c A signed char value.
C An unsigned char value. Only does bytes. See U for Unicode.
s A signed short value.
S An unsigned short value (16 bits).
i A signed integer value.
I An unsigned integer value.
l A signed long value.
L An unsigned long value (32 bits).
f A single-precision float in the native format.
d A double-precision float in the native format.
D A long double-precision float in the native format.
x A null byte.

for a full list see perldoc -f pack.

The following example is required to call fcntl for a file lock:

$wlock = pack 'sslli',(F_WRLCK, SEEK_SET, $rec, $rsize, 0);
fctnl( ... ) # Some C call

Unpack the returned flock structure:

($locked, $pid) = (unpack 'sslli', $qlock)[2,4];

The above examples show an interesting portability issue, they do not work on Linux 2.4. This is because the Linux implementation of the perl fcntl function does not call fcntl(2), but a lower level function, fcntl64. This means that the normal 'l', which is 32-bits, has to be doubled up:

my $wlock = pack 'sslllli', (F_WRLCK, SEEK_SET, 0,
$rec, 0, $rsize, 0);

(It took me ages to figure out that Linux perl uses fcntl64, I eventually ran strace(1) on my perl script to discover why it would not work).

POSIX module

The POSIX module is an important tool for Perl programmers on Unix. The interfaces it provides are mostly portable, although you must still be aware of the limitations of your particular system. The POSIX 1 standard (there are actually 13 parts to POSIX) defines a C language interface, and so some functions are not applicable to Perl. Perl abstracts messy tasks like dynamic memory allocation, so strictly speaking the POSIX module does not, and cannot, provide true POSIX compliance, only POSIX functionality. Being a C interface does mean that knowledge of C is a definite advantage when understanding the documentation.
Some POSIX functionality is provided by standard built-in Perl functions and variables, and it is generally better to use those rather than the POSIX module versions, since Perl is optimised with them, but there is not a huge difference.
To use the POSIX module effectively requires some C knowledge, but you do not need to be an expert.

Later postings will explore the POSIX module further.

Error Handling

Always check a function call for errors. As a rule, a return value of FALSE (0 or undef) indicates that an error occurred that prevented the system call from completing successfully. The exact details of the return values of the system calls are found on the relevant manual pages. Return values should always be checked, since undetected error conditions may cause serious problems that are more difficult to locate later in a program.

Some low-level functions return a value which could be zero, yet do not wish this to indicate an error. For example the tell function returns the current file position, which could be zero (beginning of file). In this case the text value '0 but true' is returned instead of just zero. This evaluates to TRUE in Boolean context, but 0 in numeric, yet is immune to usual warnings about use of non-numeric text.
By convention error messages should be reported to STDERR, using any of the methods shown below.

print STDERR "Oops: $!\n" # not fatal, not trappable

die "A death: $!" # fatal, but trappable

warn "Will Robinson!" # not fatal, trappable

The warn and die functions output the perl script line number, and both calls may be trapped using signal handling. The usual way to trap die is by using an eval block (exception handling).
If a system call fails, the global variable $! will contain a system-defined error number, in numeric context, and the error text in string context. The variable $! is only set when an error occurs. It is not cleared on a successful system call. It is only safe to rely on this value when a system call has failed.

Errno module

Unix recognises a standard set of error codes, not all system calls can return all errors. Which errors are returned by which functions are described in the manual pages.

Some useful error codes and constants (conditionally exported by Errno.pm):

EPERM Operation not permitted
ENOENT No such file or directory
ESRCH No such process
EIO I/O error
ENOEXEC Exec format error
ECHILD No child processes
EACCES Permission denied
EEXIST File already exists
ENOTDIR Not a directory

Note that some errors such as EAGAIN and EINTR may not be errors but indicate that a system call must be attempted again (EAGAIN because conditions were not right for the call to complete and EINTR because the call was interrupted by some other system activity).
The package Errno defines the symbolic constants that represent system error conditions. It also defines the hash %!, which has a key for each error constant, and its value is TRUE if the error has occurred. Not all error constants are portable, so you should not assume any particular code exists in %!.

Example - Error Handling

use Errno; # No need to import anything

my $file = 'input.txt';

if ( !open (HANDLE, "$file") )
{
if ($!{ENOENT})
{
print STDERR "$file does not exist\n";
...
}
elsif ($!{EACCES})
{
print STDERR "$file: Permission denied\n"
...
}
else
{
die "Unable to open $file: $!";
}
}

This example shows how Errno can be used to handle different error conditions.
If the call failed, then after an error message is printed, the program terminates with die. This is safer than exit, particularly in a module, since it may be trapped in the calling routine.
Errno can also import POSIX codes with:
use Errno ':POSIX';
An alternative to the Errno module is the POSIX module:
use POSIX ':errno_h';
which exports the error number constants, but not %!.

Command Line Arguments

When a program is executed, it is passed a variable-length list of arguments. The shell arranges for these arguments to be the command name and arguments typed at the command line.
The argument list is accessible to the program by the array @ARGV, so the number of arguments passed can be obtained by using the array in scalar context. Unlike C, the first element is the first argument, not the program name (that is available from $0).
There is a system-defined maximum size of the argument list. Normally, it is not allowed to be larger than 5KB or 10KB.

Option Processing

In the past, Unix commands were written by different contributors, often employing slightly different methods of dealing with the command-line arguments of their programs. By convention, Unix program arguments are either options, which are usually single letters preceded by the "-" character, or arguments to the command itself, usually pathnames. In recent versions of Unix, a definition has been developed of a "standard" command interface, which defines a basic format for command lines. New programs should, if at all possible, be written to conform to this format.
The format is fairly flexible, but does impose a few more restrictions:
Options must be a single character.
All options and their associated arguments must appear before the main arguments.
Multiple option characters may be grouped behind a single "-", except where an option requires an argument.
The standard module Getopt::Std offers two routines, getopt and getopts, the later being the more useful. getopts reads the command line argument list array @ARGV, and extracts valid switches into side-effect variables, or into a predefined hash. If an option is to have an associated argument, the option letter must be followed by a colon (:) character.
Valid options are removed from @ARGV. The end of the options is signified by encountering a non-option argument (ie.. one which does not start with a "-") or the "special" option "--", which is also removed from @ARGV.
If getopts encounters an invalid option then it prints a message on the standard error stream.
Further features are available in the extended version Getopts::Long.

Examples - Options and Arguments

These two code fragments show examples of both methods of syntax.

Using global variables

use strict;
use Getopt::Std;

our ($opt_a, $opt_d, $opt_l);
getopts('ad:l');
print "a: $opt_a, d: $opt_d, l: $opt_l\n";
print "Remaining arguments: @ARGV\n\n";

Using a hash

use strict;

use Getopt::Std;
my %options;

getopts('ad:l', \%options);

while (my ($key, $value) = each %options)
{
print "Switch: $key, value: $value\n"
}

print "Remaining arguments: @ARGV\n";

When using globals, with use strict, we must pre-defined them using our. When using a hash we pass a reference.
In both cases the value is 1, TRUE, if the switch was set. When a value is required it will be set in the side effect variable or the hash.

The Environment block

Every process has access to a variable-length list of pointers to strings, known as environment variables.
Environment variables can be set up from the shell with the export or setenv (csh) commands. They are useful for communicating information on a global level to multiple commands, as they can be accessed easily from within programs and shell scripts.
To see the current environment from the shell, use the env command.
The entire environment is made available to Perl through the hash variable %ENV. The keys are the environment variable names, with values.

print "Home directory is ",
exists($ENV{'HOME'})?$ENV{'HOME'}:"unknown", "\n";

TMTOWTDI

if ( exists $ENV{'HOME'} )
{
print "Home directory is $ENV{'HOME'}\n"
}
else
{
print "Home directory is unknown\n"
}

The example retrieves the value of the HOME environment variable (ie.. the pathname of the user’s home directory).
If HOME is set in the environment, its value is returned, otherwise the key will not exist, and a warning will be given. Therefore it may be advisable to check with exists first.
Incidentally, the HOME environment variable is the default directory for the built-in chdir function (like a shell cd command). If HOME is not set, then LOGDIR is used.

Limits

Since Unix is supported on a wide range of platforms, it is reasonable to assume that there will be differences in detail among them. For example some may allow long filenames where others may still impose the old limit of 14characters. Some may support sophisticated signal-driven job control while others may not.
In developing Unix programs that are intended to work across a wide range of Unix implementations, it is desirable to be able to represent such properties in a platform independent manner, perhaps discovering details about a particular system at run time.
POSIX defines a number of symbolic constants, specified in the POSIX module that govern the details of data types, their sizes and max/min values.
Run time values can be checked using one of three functions - sysconf allows general properties such as the number of system clock ticks per second (useful in certain timing calculations), the maximum number of files that a process can hold open at one time, and the maximum number of child processes that a process can create. Properties that are more closely related to files and directories, such as the length of a filename or pathname, are discovered using the system call pathconf or fpathconf.
Many of these “runtime” limits are also mentioned in the POSIX module, but the values here are absolute minimums. For example, a system may set the maximum number of open files per process to be any value greater than _POSIX_OPEN_MAX, usually defined as 16.

POSIX::sysconf

General system limits and properties can be queried using the sysconf function. The argument is an integer symbolic constant representing the value to be queried. The return value is the current setting of the value, or -1.
If the name parameter represents a property that is valid, but not supported, then errno will be clear even though the function returns -1. An invalid name causes the function to return -1 and set errno to EINVAL.

$max_open = sysconf(_SC_OPEN_MAX);

if (!defined $max_open)
{
print STDERR "Cannot determine _SC_OPEN_MAX: $!"
}
else
{
print "Max open files: $max_open\n";
}

To check on values that can be queried using sysconf(), consult the manual page or the include files or .

POSIX::pathconf

Some limits may have different values on different logical filesystems on the same overall Unix system. These are mostly related to the properties of the filesystem, such as the maximum length of a filename, or a pathname.
To query such a value, use the POSIX::pathconf function. It is similar in its operation to sysconf, except that it requires either a pathname or a file descriptor representing an already open file to be passed in addition to the name of the limit being queried. This indicates on which filesystem the limit is being checked.

$max_name_len = pathconf('.', _PC_NAME_MAX);

if (!defined $max_name_len)
{
print STDERR "Cannot determine _PC_NAME_MAX: $!"
}
else
{
print "Max name length: $max_name_len\n";
}

The return values from pathconf are the same as those from sysconf.
fpathconf has the same functionality as pathconf, except is takes as its first argument a file descriptor instead of a file name. File descriptors will be described later.

Dates and Times

Unix systems count time as a number of seconds from a fixed starting point, called the Epoch, that is defined as midnight (00:00:00) on January 1st, 1970.
This is how time is represented internally in the operating system, applications use a number of functions to translate this into the appropriate representation, taking into account timezones and daylight saving time.
To find the number of seconds since epoch, use the time() built-in function. This can be useful for timing operations to the nearest second.
On some systems it may be possible to use an alternative function, gettimeofday(). This function was originally part of the BSD Unix family, it is often useful because it provides a higher resolution of the time measured. To use gettimeofday() you are required to pass in an indication of a timezone as the second argument. It is acceptable to use NULL here, signifying the current timezone.

For measuring time in better granularity than one second, you may use either the Time::HiRes module from CPAN, or if you have gettimeofday(2), you may be able to use the syscall interface of Perl, see the perlfaq8 manpage for details.

Displaying Dates and Times

Although the central time representation in Unix is a count of seconds, there are many ways in which the date and time can be displayed. A collection of functions allow the programmer to convert the date from the seconds count into various formats, including formatted strings and a structured “broken-down time” representation.

The most commonly used function is localtime(), which takes the basic second count and converts it to a string, in the correct timezone and ready for printing: By default the current time is returned, but output from time() can be used as an argument.

The Time module offers interfaces to localtime and gmtime where the time elements are named.

“Broken Down Time” Array

Sometime it is desirable to obtain partial date information, for example the month or year. To do this, it is best to use an alternative representation known as “broken down time”, as returned by localtime in list context.

Index name Purpose Range Notes
0 sec Seconds after minute 0-61 2 "leap seconds"
1 min Minutes after hour 0-59
2 hour Hours 0-23
3 mday Day of the month 1-31
4 mon Month of the year 0-11 July is 6 (not 7)
5 year Years since 1900 2003 is 103
6 wday Days since Sunday 0-6
7 yday Days since 1 Jan 0-365
8 isdst In Daylight Saving Time -1/0/1

The elements in the list allow the programmer to extract any pertinent pieces of date or time information to allow more specialised calculations to be performed. The sec range requires some explanation: normally the upper limit is 59, but up to 2 extra "leap seconds" can sometimes be added.
To translate from the number of seconds since epoch to broken down time, use localtime or gmtime:
@dt = gmtime ( $secs );
@dt = localtime ($secs );
If the argument is omitted the current time is used. The difference is that localtime will return the time correct in the current timezone, gmtime returns the time correct in GMT (basically this is the same as UTC).
Alternatively, the POSIX function strftime allows the programmer to apply printf()-like formatting so that the exact format of the date can be specified when it is printed. Consult the manual page for details of the formatting options - they are considerable…
It is possible to translate from broken down time back to seconds since epoch. To do this use the POSIX function mktime.

Summary

Many standard Unix system APIs can be called from Perl, some are built-in and some come from modules, notably POSIX.

The original APIs were designed to be called from C, so knowledge of that language is useful. If you wish to know more about the C interfaces, come on QA's Unix Programming Course.

I hope to post examples of other Unix API, for example IO, Sockets, and System V IPC.

Doh!

2005-04-29T14:37:00.000+01:00

It all started when Ryan Air cancelled my flight to Dublin on a Sunday evening. That meant I had to travel on the Monday, and had half an afternoon with nothing to do. "Upgrade my laptop to Fedora Core 3!", I thought. The Devil makes work for idle hands.

Everything went wonderfully until I got to disk 4 of 4, which would not read. I had, of course, skipped the media check as being too time consuming. Mistake number one.

There is no exit from a Linux install, so I had to remove my laptop battery to try to reboot. Except it wouldn't, not even Windows XP (it was a dual boot machine).

So I tried to rescue the hard-disk by reinstalling GRUB (the Linux boot loader), no dice.

So I tried to rescue the hard-disk by booting Windows XP in rescue mode, except I had forgotten Administrator's password (it had been set under pressure a couple of years previously when attached to a client's network, and hadn't been used since).

So I tried to download a password cracker, except my company had blocked such sites.

So I asked someone else to get one for me, except I had not bothered to bring my floppy disk drive to Dublin. Double doh!

So I reinstalled Red Hat 9 and recovered the Linux partitions, but still could not boot Windows XP (although I could "see" them from Linux).

Saturday, back home and on holiday (joke!). Found the Toshiba recovery disk for my machine – wonderful! Except that it blasted everything in sight, including all my Linux and Windows XP partitions.

So I reinstalled Windows XP, and started reinstalling products. (Great assistance from out support guys, I was reading emails within minutes of installing MS Office).

Before I got too far I decided to reinstall Fedora (with a new disk 4). All went OK except it would not boot – Linux or Windows! Triple doh?!

So I tried again, this time the install software crashed with an assertion failure. After much gnashing of teeth I googled the error. Turned out to be a "known" bug in the utility 'parted' (Bugzilla Bug 138419, if anyone is interested). It appears that this bug was the cause of the original corruption. I read up on the bug, and guessed that PartitionMagic might fix it from Windows, except I couldn't boot.

So I found an old MS-DOS 6 boot disk, and did a swift fdisk /mbr. That enabled me to boot Windows XP and run PartitionMagic.

Reinstalled Fedora – third time lucky!

So now it is Friday of my one week's holiday and I have reinstalled most of my software and restored my backups. I lost some Perl 6 work I had been doing, but I should be able to regenerate that fairly easily.

Lesson's learnt:

ALWAYS do a media check before a Linux install.
Do not throw away Windows, it can get you out of a hole.
Linux is not bug free (!).
If you are going to trash your machine, do it when you have a week to spare.

Pugs

2005-04-06T08:25:00.000+01:00

Just downloaded the latest version: http://search.cpan.org/~autrijus/Perl6-Pugs-6.0.14/
Will I ever get time to have a proper play with pugs?
What's pugs? It is a prototype Perl 6 written in Haskell. On CPAN it is rated ***** (the highest) and deserves more.
It is slow, and does not support all of Perl 6, but is only meant as a kinda 'proof of concept' tool. It is also great for sado's like me who can't wait for Perl 6.

Perl 6

2005-04-06T07:53:00.000+01:00

This is an overview of Perl 6. To avoid a posting of biblical proportions it lacks any technical detail - that will follow in further postings.

Perl 6 is the next generation of the highly successfully Perl programming language. This posting discusses why Perl 6 is necessary,has had to be developed, and gives an overview of changes.

Why?

Perl was originally written to replace the scripting language awk when Larry Wall, the author of Perl, became frustrated with awk’s limitations. Perl’s strength lay in text handling, and was an obvious choice for manipulating HTML. It became known as “the duct tape of the web”, although its design pre-dated the web boom. Perl 5, released in 1994, gave it the flexibility and power to rival conventional 4GLs, and boosted it beyond mere scripting and report generation. Successive releases of Perl added database support, references, OO, imbedded low-level code, and a huge range of add-on modules.

But all is not well, ten years is a long time in IT. Essentially designed to slot between C and Unix shell programs, Perl is showing its age. The syntax of Perl 5 is undeniably situated firmly in Unix-land, yet the product has an important niche in proprietary operating systems, such as Microsoft Windows.

Unix was originally a process-based system, and Perl was designed to work within that. Over recent years the advantages of multi-threading beyond "just" GUI systems have become clear, and a number of threading models are in common use. Perl has had several tries at multi-threading implementations but none have been completely satisfactory. Retrofitting multi-threading is difficult and painful.

In Perl 5 Object Orientation capabilities are tacked on as an after-thought. Frankly this is not a problem for Administrators and for small projects, but it is not scalable and lacks many features expected by today's (mostly OO) programmers. For example: a Perl 5 class has no data layout imposed or inherited (inheritance is really only through methods), no data hiding or protection is available (the whole object is visible), and there is no prototype checking for method calls. In Perl 5, multi-threading and OO do not mix; if an object is passed between threads it looses its class.

Perl 5 language can make it difficult to learn. That Perl has been able to get away with this is a tribute to its power and usefulness. Part of the problem has been a basic rule that, so far as possible, upgrades should be backward compatible. In other words, new features could not replace the old ones, only supplement them. "There's more than one way to do it" can be powerful but is confusing when trying to learn the right way to do it.

Perl has begun to loose its dominance on the web with the growth of ASP and the popularity of alternatives like JavaScript and PHP. On the application side Perl is threatened with competition from well-designed rapid development tools and the almost universal adoption of Object Orientation. On the scripting side we see the growth of modern feature-rich tools such as Bash, and the up-coming Microsoft Shell for Longhorn. Directly competing with Perl, and gaining popularity, are the OO scripting languages Ruby and Python. In the long term these could sideline Perl and destine it to the Great Recycle Bin in the sky.

The source code of the Perl interpreter has been highly hacked over the years to the extent that changes are now difficult to implement without breaking apparently unrelated code. This is a common problem with software that has had to adapt to meet changing needs and expectations.

If the language and culture is to survive a radical change is needed.

Enter Perl 6. Perl 6 is being designed and specified - that in itself is a novelty - by a confederation of programmers from the Perl community, shepherded by Larry Wall.

What's new?

Perl 6 is designed for change, and an expected life of over twenty years. There is no desire for backward compatibility. compatibility, Perl 5 scripts will rarely, if ever, run directly on Perl 6. Fortunately conversion routines and transition aids are being produced.

There is a price to pay for Perl 6 in the investment made in Perl 5 skills. Those who are comfortable with Perl 5 are naturally aggrieved to find they are beginners again. It should be made clear that Perl 6 is still undeniably Perl, after all it was written by Perl people who have probably invested more in Perl skills anyone else. There are a huge number of new features, but many of them are optional. As with any other product, users of Perl 6 will learn and use just those features needed to get the job done.

A technical overview of changes in Perl 6 is given as an Appendix to the QA Perl 5 Programming course.

Parrot

In Perl 6, the language syntax and implementation internals have been separated, in a similar way to the Common Language Runtime in Microsoft's .Net product, and Sun's Java runtime. The language independent runtime component is called Parrot. Parrot is downloadable now, and you can write Parrot Assembler (PASM) if you really must. A higher level interface is Parrot Intermediate Representation, or PIR. PIR is expected to take the place of Perl 5's XS interface for low-level code.

Although its aim was to drive Perl 6 code, it was realised quite early on that other similar languages could be plugged into Parrot. Implementations of Ruby, Python, Java, PHP, and others, exist or are being developed. More can be added "merely" by generating the writing the PASM or PIR interface. Part of the attraction of this approach is that objects created for Parrot may be shared, regardless of the language. The Database Interface (DBI) module from Perl 5 is being rewritten for Parrot, so it should be usable from any Parrot language. Parrot also enables general functionality such as multi-threading and garbage collection to be abstracted by the language.

While there is no specific aim to compete, there is an interesting overlap between Parrot and Microsoft's .Net, more specifically between Parrot and Mono, which is the portable implementation of .Net, now owned by Novell. For example, both intend to run Python and PHP, and C# on Parrot is a possibility. Will Parrot and .Net compete head-to–head? Possibly, but it is doubtful that Parrot will run C# as fast as .Net, or that .Net will run Perl 6 in a satisfactory way.

Perl 6 OO

Perl 6 is Object Oriented from the ground up and for many Perl programmers that will be enough reason to use it. The syntax to define a class is similar to other languages, and variables are objects. For example, to find the length of a variable we now call a method on it, rather than a library routine. Classes in Perl 6 replace packages in Perl 5, and data layout may be imposed and inherited. Private, public and protected attributes and methods are supported, as well as delegation, roles and so on.

Multi-threading and OO should co-exist happily in Perl 6, but implementation details are still under wraps. The new garbage collector should remove memory leak problems that plague Perl 5 multi-threading.

In Perl 6, methods and subroutines have been considerably improved, allowing all the features normally expected, including named and typed arguments, and overloading.

The inclusion of Object Orientation is not to everyone’s taste, and for many users of the language it will be an irrelevance. Not every job needs OO and knowledge of OO will not be a pre-requisite to enable effective use of Perl 6.

Basic syntax and operator changes

There are a number of changes to the basic Perl syntax, which will be immediately obvious to a Perl 5 programmer, such as accessing an array element or hash value. Many of the syntax elements and operators are different, and some are less than obvious. This is not the place to list them, partly because they have not yet been finalised.

A number of new operators are available in Perl 6, particularly for handling lists. Some allow operations on all elements in a list and others give the ability to join and match lists in a number of different ways.

The reasoning behind the operator changes is to make Perl 6 easier. New Perl programmers will be able grasp its concepts more readily. Experienced Perl 5 programmers will have to learn new ways, but at least they already know the concepts and can concentrate on learning the new syntax.

Regular Expressions – Rules

A Regular Expression (RE) describes text of interest, usually for extraction or editing. It consists of a "meta-language", and the one chosen for Perl 5 is a well known dialect based on POSIX and used by many other languages and tools, including Microsoft .Net.

Perl 6 has modified the language with the intention of clarifying it. Very few of the familiar meta-characters are changed in Perl 6, but the most significant (the one everyone is shouting about) is the use of white space. In Perl 6 white space is used as in any other part of the program - it is not significant unless quoted.

Changes such as new quantifier delimiters are part of assertion syntax. Assertions are an extension of the concept of character classes that either match (true) or do not (false). Assertions return rules that can be stored meta-characters, a single or list of literals, array elements or hash keys, and even code. Many of these features can be achieved in Perl 5, but rules bring them together. The Perl 6 language itself will be expressed in rules.

Traits and Attributes

A feature of most scripting languages is that variables are type-less until a value is assigned to them. Later in its life the variable (and possibly its value) might change type. Perl 5 has always had that feature and, by default, so does Perl 6. For some operations typed variables are required and, although some checking is possible in Perl 5, Perl 6 extends this with optional built-in traits applied at compile-time. Aside from the additional help these supply to program logic, they also help the optimiser.

Attributes enable a value to be overridden depending on its context at run-time. This is a familiar theme in Perl 5, but Perl 6 has introduced many more contexts and ways to force them.

When?

Good question. We were promised a development release in Quarter 3, 2005, but funding problems have caused slippage. At the time of writing a pessimistic guess is Quarter 1 2006 for the development release and Quarter 1 2007 for a production release. What state the development release will be in is anyone's guess, but for sure it's website will get a huge number of downloads from day 1. The community will make it work, and work well.

Is Perl 5 dead?

No, far from it. Partly because of the uncertain Perl 6 timetable, Perl 5 maintenance continues, and releases are planned for several years yet. Unlike commercial organisations, the Perl community has no interest in pressuring people to upgrade, Perl 4 maintenance continued for some time after 5 was released.

Ponie is rather contrived from Perl On New Internal Engine. It is the interface required to run Perl 5 upon Parrot. It should enable some of the advantages of Parrot, like operating system abstraction, to be fitted to Perl 5. Versions of Perl 5 running with Ponie on Parrot should be available at Perl 5.12. It is intended that eventually all Perl 5 development will move to Ponie on Parrot.

Perl 5 modules from CPAN should be able to run on Perl 6 via Ponie. This will reduce the problem of a lack of Perl 6 modules in the early days.

Conclusion

Perl 6 represents a change of emphasis for the Perl language. It should be easier to learn and use for today's generation of programmers. Existing Perl programmers will hate some changes but like others. Some will stay on Perl 5, but will be fighting a rear-guard action as the benefits of Perl 6 become more apparent.

QA (www.qa.com) intends to produce a Perl 6 for Perl 5 Programmers course as soon as a viable version of Perl 6 is available. Pure Perl 6 courses will then follow. It is intended to continue updating the Perl 5 Programming course to keep it in line with Perl 5 maintenance releases. And of course when Perl 7 is released…

From Larry Wall:
"Perl 6 is merely the prototype for Perl 7.:-)"

References

A technical overview of changes in Perl 6 is given as an Appendix to the QA Perl 5 Programming course.

The Perl 6 development web site is dev.perl.org/perl6. Larry Wall's explanations are given in Apocalypses at dev.perl.org/perl6/apocalypse, with further information and examples in Exegeses, dev.perl.org/perl6/exegesis. These are a little out of date, but the Synopses at dev.perl.org/perl6/synopsis were updated in December 2004.

A fortnightly digest of events in Perl 6 development is posted onto the O'Reilly Perl website, www.perl.com.