Korn Shell programming and being "dotted"

While teaching a Korn shell course I was asked "where do you stand on exit verses return"? To be honest I had not realised you could use return outside a function in Korn shell.
Well you can, but not in Bash, and the POSIX standard says that the effect is undefined.

Why would you want to? The effect is exactly the same as exit, unless you have been executed through the dot command, "sourced", or "dotted". The argument goes that if you use exit then when your script is invoked using 'dot' then it would exit the caller, whereas return does not.
I find the argument fundamentally flawed. It is folly to assume that it is safe to execute any old script in this way, because it breaks a fundamental rule of programming - encapsulation.
"Dotted" files do not run in their own namespace or environment. Any change, for example a cd command, will alter the caller. Therefore writing a script to be safe would also involve restoring any changes.
Now, what about variables and functions? ANY declared in the called script will overwrite those in the calling program. If you are unaware of the script you are using ("I don't know if it has exit in it") then you will be unaware of its variable names and functions. It could do absolutly anything to your environment (I use the term loosly). An important principle of encapsulation is that you have your own namespace, of course with 'dot' you do not.

It works both ways. Take the following senario:

typeset -i x
. myscript

If myscript has a varible called 'x', and it is used for non-numeric text, then the typeset will have the effect of altering that text to "0". What if x contains a filename in the script? This could completely alter or invalidate the action of myscript.
The dot command is designed specifically so that the called code WILL change your current process, but the code has to be written specifically for that job. Having blind faith that a script will not trash you current session, and will even work in your environment, is taking a big risk.
I suggest, in ksh93, that you put this at the head of your scripts:

if [[ $0 != ${.sh.file} ]]
then
    echo "Do not . this file!" >&2
    
    return 1
fi

Are we loosing C skills? Should we care?

A sign of the times. In the last three weeks I have taught two Python courses and one Perl. Last week a C programming course was running with just one delegate. I have not taught C for several years despite being the only trainer for C in the company (I do not include ObjectiveC). This week I will be teaching UNIX programming using C for the first time for four years.

So what? Software moves on, no one uses C anymore. Really?

Perl, PHP, Python, Ruby, are all written in C. Most of Windows and Linux is written in C. Sure, there are versions like Jython, but the use of that variant is relatively small. Programmers of my generation are gradually "logging-out", what happens then? The number of new C programmers seems very low. That will be fine for while, but we are loosing the skills to operate at the lower levels.

Will the systems of the future be able to build on the C base we have, or will everyone be using high level tools? Does it matter? I think it does, the foundations of software should be solid, not top-heavy. I don't think you can really understand UNIX or Linux unless you understand C. Who will pickup the baton?

C-shell: basic features compared to ksh

Sign-on files

The following files, if they exist, are read at sign-on:

/etc/csh.login instead of /etc/profile - interactive (login) sessions only
~/.cshrc instead of $ENV
/etc/csh.cshrc if supported, not all c shells use this
~/.login instead of .profile - interactive (login) sessions only

On sign-off the optional file ~/.logout is executed (no equivalent in ksh).
In-line scripts

To execute a script in-line, that is without creating a sub-shell, use the source command (instead of the . command), as in bash. For example:

source ~/.login

Filename Completion

An interactive C shell can complete a partially typed filename or user name. By default this feature is switched off, to turn it on:

set filec

When the user types characters followed by , the shell attempts to fill in the remaining characters of a matching filename, assuming the characters typed are enough to identify a single file.

If the filename prefix is followed by EOF (usually d), the shell lists all filenames that
match. It then prompts once again, supplying the incomplete command line typed so far.

Ending the name with a tilde (~) indicates that a username is required instead.
Aliases

The syntax for alias in the C shell is slightly different to other shells:

alias new-alias existing-command

i.e. there is no assignment character (=).

Functions

No. There are none.

History mechanism

The history list (and the history command) exists as in other shells, however the syntax for retrieving lines from it is different to ksh. The following table refers to lines in the history list:

!n The nth line in the history list
!-n The nth previous line
!! The last line
!prefix The most recent line with the specified prefix
^xx^zz The last line, with the string xx replaced by zz
!n:s/xx/zz/ The nth line, with the string xx replaced by zz
!* All the arguments of the last command
!$ The last argument of the last command (instead of $_)
!^ The first argument of the last command
!:n The nth argument of the last command

This is also supported by bash.

Variables

Pre-defined variables are different from other shells, the most noticeable feature being that they are in lower and upper case. The list of variables is large, here are a few:

The exit status of the previous command is $status (instead of $?).

The command line prompt is $prompt (instead of $PS1). The contents of which may contain a large range of format characters. It is too huge to list here, see the man pages for details. Some versions may also support $prompt2 and $prompt3.

Command line arguments are set in the array $argv (see below).

The current working directory is in $cwd (not $PWD).

Lowercase versions of path, home, and term also exist. The lowercase version of path is set from PATH, but is only used by the shell. This does not mean that path can be altered without affecting PATH, since the two are kept synchronised. A similar scheme exists with term, home, and others. Only the upper-case versions are exported. Note that path is a list, i.e. the directory names are separated by white-space, not colons.

Setting values

Local variables may be assigned values using set (which is more like BASIC than C), for example:

set var = "She sold sea-shells"

Note that spaces are optional around the assignment symbol (=) – hooray! The = is mandatory.

To create a variable in the environment block (export), use setenv:

setenv var "She sold sea-shells"

They may be removed using unsetenv.

There must be no assignment symbol (=) with setenv.
If you have an existing local variable of the same name then that is not overwritten, a new one is created. For example:

$ set var=red
$ setenv var blue
$ echo $var
red
$ csh # Create a child
$ echo $var
blue
$ exit

In the example above, if we had just done

    setenv var 

then the child process would see an empty variable.

Arrays

Like ksh and bash, csh arrays cannot be exported. However unlike ksh and bash csh does not allow you to try. Attempting to setenv an array will give an error.

An array is initialised from a list (as in bash and ksh93), where the elements are space separated and delimited by parentheses, for example:

set arry = (HPUX AIX DGUX Dynix Tru64 DRSNX SunOS Linux) 

The old ksh88 method of initialising an array with set –A is not supported.

Access an element of the array using the index, counting from 1 (which is weird, since in C we count from zero).

echo $arry[3]
DGUX

We can also access a range of elements:

echo $arry[2-4]
AIX DGUX Dynix

and the whole array using an index of * :

echo $arry[*]
HPUX AIX DGUX Dynix Tru64 DRSNX SunOS Linux

This is the same as printing the array variable (echo $arry).

The number of elements in the array is held in a variable with the same name as the array, prefixed with a #, for example:

echo $#arry
8

so getting the last element is simple:

echo $arry[$#arry]
Linux

With other shells we would have to use braces ({…}), which are not needed in this context, but may still be used to delimit a variable name:

set money = "dollar"
echo "Give me your ${money}s"

Command-line arguments

The list of command line arguments is held in the array argv. To get the complete list of arguments, use $argv[*] (instead of $* or $@). The variables $0..$n are still available, but may alternatively be obtained using $argv[n]. A C programmer would also expect a variable called argc, but the number of arguments is held in #argv instead. See also Arrays.

The command shift is supported as in ksh and bash.

Modifiers

The C shell also supports modifier codes. These are used by suffixing a variable name with a colon (:), followed by the code letter:

h remove a trailing path name (similar to dirname)
t remove all leading path names (similar to basename)
r remove a file "extension" (suffixed preceded by a dot).
e remove the filename preceding a file "extension"

When used on a whole list or array, they should be preceded by a g (gh, gt, gr, ge) otherwise they will only be applied to the first element. The exception is the modifier code q, which places the extracted data in quotes (thus avoiding $*/$@ differences).

For example:

set file = /home/user1/Seas/North.c

echo $file:h
/home/user1/Seas

echo $file:t
North.c

echo $file:r
/home/user1/Seas/North

echo $file:e
c 

Other variable operations:

Shell Options
There are no shell options like ksh and bash have. Instead everything is controlled by shell variables, and many have the same names as options in ksh, like noclobber, ignoreeof, noglob, etc. See the man pages for a complete list.

To set a shell option you normally only have to create the variable, you often do not need to give it a value, for example:

set ignoreeof


Variable types

No, not in csh. (ksh typeset) For example, to convert to uppercase we have to use:

set var = `echo $var | sed \ y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/`

Reading from stdin

To read a variable from the keyboard (stdin), use the < var =" $<">to continue…"
$< Command substitution

Command substitution uses back-ticks ($(…) is not supported), for example:

set DIR = `ls`

Note that this will return a list, and make DIR into an array. This reduces the need for slicing, for example:

set now_date = `date '+%a %H %M %S'`

returns a four element array, where:

$now_date[1] is the weekday
$now_date[2] is the hour
$now_date[3] is the minute
$now_date[4] is the second

Arithmetic

Arithmetic commands are prefixed with the @ symbol, which is actually a built-in command and so must be followed by a space.

The usual C/C++/Java arithmetic operators, including post-fix increment and decrement (but not pre-fix), are supported, for example:

set x # variable must already exist
set j # variable must already exist
@ x++ # Increment $x
@ j = ($j + 42) * $x # Brackets are supported

# To continue over > 1 line, escape end-of-line

@ now_secs = ($now_date[2] * $hour_secs) + \
($now_date[3] * 60) + $now_date[4]


Conditionals

As well as the usual conditional statements (see below) the C shell supports a short cut for testing if a variable exists, the $? prefix. Confusingly it returns 0 if the variable does not exist, and 1 if it does. For example:

echo $?non_existent_variable
0

echo $?existing_variable
1

Relational operators

!= not equal
== equal (note : two = characters)
> greater than
>= greater than or equal to
< pattern1 ="~" file ="~" dirname =" $SYS_LPT/$usernm" userdir =" true" dirname =" '$SYS_JOB_'$type" result =" `eval" dirname =" '$SYS_'$type">Loops

As in most shells, there are two basic loops, the foreach loop, for list processing, and the while loop. The syntax is familiar but note, like Perl, the foreach loop does not contain an in:

foreach variable-name ( list )

# loop body

end


while boolean-expression

# loop body

end

The commands break and continue have the same meaning as in most languages, break exits the loop prematurely, and continue executes the next iteration at once.


Unconditional flow

The command onintr is similar to trap in ksh – it executes specific code on a signal, but it cannot pick specific signals to trap. There are three forms;

onintr use default signal handling
onintr - ignore signals
onintr label jump to the specified label and continue execution from there. The syntax of a label is: label:

Apparently there is also a goto command.

Redirection

Standard channel (stdin / stdout / stderr) redirection is similar, but not the same, as other shells. Channel numbers are not used,

command > filename redirect stdout
command >> filename append stdout
command <>& filename redirect stdout and stderr to the same file
command >>& filename append stdout and stderr to the same file

The only way to direct stdout and stderr separately is by invoking a subshell, for example:

(command > out_file) >& err_file

Pipes are supported as normal, however co-processes are not.

The set command sets shell variables, so to set option noclobber:

set noclobber

That is, no –o. To override noclobber, append a !, as in:

command >! filename

Background jobs

Very similar to ksh – in fact Korn "stole" the idea from csh. The jobs and bg commands are supported.

Quote from man csh

"Although robust enough for general use, adventures into the esoteric periphery of the C shell may reveal unexpected quirks". You have been warned!

That Python next/send demo

import glob
import os

def get_dir(path):
 
    while True:
  
        pattern = path + '/*'
        count = 1
        for file in glob.iglob(pattern):
            if os.path.isdir(file):
                print count
                count += 1
                path = yield file
                if path: break
      
        if not path: break  
          

gen = get_dir('C:/QA/Python')

print "about to next"
print next(gen)
print next(gen)
print gen.send('C:/QA')
print next(gen)

Gives:

about to next
1
C:/QA/Python\AdvancedPython
2
C:/QA/Python\Appendicies
1
C:/QA\Android
2
C:/QA\blanket

A little Python introspection

In Python we can load a module and assign an alias:

import module-name as alias-name

I was asked how a programmer can find the mapping between an alias and the real module. That was rather beyond the scope of the course, but here goes.

The locals() built-in, and it's sister, globals(), were mentioned in the course almost as a 'by the way'. In fact both are very useful for introspection. In this case locals() will give us the name of the module aliases. It also gives us all the other names that are local, but we are only interested in modules. We can use inspect.ismodule() to get only those names that refer to modules. Watch it: locals() returns a dictionary where the keys are the names, but the values are the objects themselves - handle with care. Fortunately, stringifying the object gives us a text string like this:

<module 'GetProcs' from 'C:\Python27\lib\site-packages\GetProcs.pyd'>

and it is simple regular expression to extract the names.

We still have issues. Most obvious is that any modules used for the introspection (inspect and re) are included in the data. In Python 3 we also get issues trying to read the locals dictionary because it changes during a loop (items() returns an iterator in Py3). We can solve both by putting the code into a different module (although there are other solutions). But then, how can we look at our callers namespace? Simple: sys._getframe(1), which exposes just about everything, warts and all. The locals dictionary is avalable through f_locals.

So, here is my module, which I named Spection:

import inspect,sys,re

def look(): 
    for name,val in sys._getframe(1).f_locals.items():
        if inspect.ismodule(val):
            fullnm = str(val)
                if not '(built-in)' in fullnm and \
                   not __name__     in fullnm:
                    m = re.search(r"'(.+)'.*'(.+)'",fullnm)
                    module,path = m.groups()
                    print "%-12s maps to %s" % (name,path)

The user's code looks like this:

import Spection
import glob as wildcard
import ConfigParser as parser
import GetProcs as ps

Spection.look()

And the output looks like:

ps           maps to C:\Python27\lib\site-packages\GetProcs.pyd
parser       maps to C:\Python27\Lib\ConfigParser.pyc
wildcard     maps to C:\Python27\Lib\glob.pyc

Perl - the way ahead

Wow! I've just been knocked off my feet.

I'm at YAPC::Europe (Yet Another Perl Conference) in Riga, and just listened to the vision for Perl 5's future. Jesse Vincent is the Perl 5 Pumpking, essentially the project manager. His vision is to evolve Perl 5 into a modern language (he did not use those words, but that's the gist).

Old Perl is 5.14, the current stable release. The idea is to introduce changes, like a fixed smartmatch, and deprecate the stuff that was preventing Perl 5 from moving forward because of the need for backward compatibility. Backward compatibility in the future will be provided by stating which version of Perl we should use - but still complete backward compatibility will not be guaranteed.

By moving platform specific features out of the base and into modules (as in Python), the base gets to be small and easier to maintain, and maybe even faster. Deprecating features then removing them should have the same effect.

What a shame this was not decided on ten years ago, still, better late than never. There will be issues with CPAN. Modules are usually only maintained for one version, the current stable one if you are lucky. Backward compatibility will be an issue with modules. The 'use version' directive (hopefully) will be lexical, so modules will be able to run on a later release (mostly, maybe). But of course they won't be able to run on an earlier release - but we have that issue right now. CPAN will need to search/tag/group modules by base release, that should not be hard, but it needs to be done before the issues arise at 5.16.

So where does that leave Perl 6? Perl 6 is not dead, but rather it takes the pressure off it. Most people (me included) looked to Perl 6 for the future, and all the smart stuff that's in Python that I can't do in Perl. No longer. Perl 5 will move forward, alongside its sister language. The possibility of Perl 5 overtaking Perl 6 was not discussed.

The times, they are a-changing.

Installing Python 3.2 Tkinter on Linux

A quick note for the install. Tkinter is not installed by default with Python 3.2 on Linux unless the tk development pack is installed. Where that is documented I could not say.

On Red Hat derived distributions this would be using (under user root):
yum install tk-devel
Make sure you do this before building Python. It is required for idle, which is named idle3 (and that is not obvious either).

A Christmas present from the monks

After what seems an age I finally got promotion to Prior, next stop (in 3000 XP points time) is Monsignor. My best node was Unexpected Output with a score of (strangely) 42.

It's weird, I get such a buzz from helping people with Perl it is almost adictive.

Python msvcrt.locking trials and tribulations

It should be easy. All I wanted to do is to replace my C demos which use flock with a nice homely Python version on Windows. I wanted to stick with the standard library so I could just slot it into the existsing course (so I couldn't use the Locking module on PyPi).

A simple demo to start with:
1. Process 1: lock a region, write a record, pause
2. Process 2: attempt to lock the same region
3. Allow Process 1 to release the lock
4. Process 2 continues

Converting the code was easy (I thought), but no matter what I did I always found the region was locked for the second process - the lock was not being released.

I blame the documentation, and possibly the implementation.

What is not obvious is that the current file position must be reset to the original locking position before the lock gets released. The write (of course) advances the current file position, so by the time we do the unlock we are unlocking a region for which we don't have the lock in the first place! Wouldn't it be better if msvcrt.locking returned a lock object, saving the file position? Anyway, here is my completed demo:

"""
   lock_w.py

   Run two copies, each in its own terminal session.
   Allow one to write a number of records, then switch
   to the other showing that it blocks on the same record.
   Switch back and release the lock, and show that the
   blocked process proceeds.

   Also run with lock_r to demonstrate interaction with
   read locks.

   Default filename is rlock.dat

   Clive Darke QA
"""

import msvcrt
import os
import sys
import time
from datetime import datetime

REC_LIM   = 20

if len(sys.argv) < 2:
    pFilename = "rlock.dat"
else:
    pFilename = sys.argv[1]

fh = open(pFilename, "w")

# Do only once
Record = dict()
Record['Pid'] = os.getpid()

for i in range(REC_LIM):
    Record['Text'] = "Record number %02d" % i
    Record['Time'] = datetime.now().strftime("%H:%M:%S")

    # Get the size of the area to be locked
    line = str(Record) + "\n"
    
    # Get the current start position
    start_pos  = fh.tell()          
    
    print "Getting lock for",Record['Text']
    msvcrt.locking(fh.fileno(), msvcrt.LK_RLCK, len(line)+1)
    fh.write(line)       # This advances the current position
    
    raw_input("Written record %02d, clear the lock?" % i)
    
    # Save the end position
    end_pos = fh.tell()
    
    # Reset the current position before releasing the lock
    fh.seek(start_pos)
    msvcrt.locking(fh.fileno(), msvcrt.LK_UNLCK, len(line)+1)
    
    # Go back to the end of the written record
    fh.seek(end_pos)
    
    print      

fh.close()

Update

I try to keep these posts technical, but I have not blogged in a while so I'll break with tradition.

I have been teaching a lot of Python. Mostly this has been Python 2, but some clients want Py3. Unfortunately not everyone understands how Open Source language releases work, and some internally assumed that no-one would want Python 2 training as soon as Python 3 was released.

I have to confess that, having done a lot of Python recently, moving back to Perl is rather a drudge. Python is not perfect, but I now find the syntax of Perl unnecessarily fussy, and Perl 6 is even worse. I mean fussy in the sense that Perl code is to Python as a doily is to a beer-mat.

Talking of Perl 6, with the release of Rakudo Star I thought (hoped) that we would get a flood of requests for courses, but we have not had one. Rakudo Star is, I guess, still too early, and I suppose early adopters are happy to learn it themselves.

Meanwhile, roughly half of the delegates that I am teaching Python have come from Perl. There does not appear to be a consistent reason for this, and often it is not the practitioner's decision anyway. It appears to be the perception of where Perl is in the scheme of things. Its all about image and marketting. What can Perl do about that? Perl 6, but Python is catching up.

Here is a simple example. One of the nice things about Perl is the way that lists can be used:

($one, $two, @fred) = qw(The quick brown fox);

Cannot do that in Python 2, but in Python 3:

one, two, *fred = ('The', 'quick', 'brown', 'fox')

The * indicates a greedy list (and you though Python didn't have sigils?). No qw() equivalent yet though, and Perl array and hash slices are still more powerful. Unless you know better...

Now I learn that Civilisation V is using Lua as its scripting language because Python (used in Civ. IV) is too slow. Is Lua next on my list?

Sony Vaio Windows 7 "No Internet access"

This error has been driving me crazy. I got it when trying to connect to certain WiFi's, but not all. Surfing the web gave various solutions, none of which worked. I eventually cracked it by accident, so I'm posting it in the hope I might prevent someone else from going insane. It is far too late for me.

Sony bundles all sorts of clever programs on its laptops. It's mostly concerned with multi-media, but one of them concerns us - VAIO Control Center (sic). So go to start/"All Programs" and select it. Now select "Network Connections", then "VAIO Smart Network".
Here be Dragons.

VAIO Smart Network creates "profiles". Click on "Advanced", recognise that stupid dialogue displayed at start-up? Click "Settings" (bet you never thought of going there).
Now select "Profile Settings" from the left panel. Yes, I know this navigation is tortuous, but we are nearly there. From here you can edit one of your profiles, or create a new one. When a new profile is created from the desktop dialogue when it first connects, it defaults everything to the previous settings. It was carrying over a DNS setting I had from another connection, and that was preventing me from connecting. Select the "IP and DNS" tab and ensure that "Obtain an IP address automatically" and "Obtain DNS server address automatically" are both selected (it was the later which screwed me).

The profile is selected by the name of the network. One thing that Sony engineers did not take into account in their design is a name collision. That is, two distinctly different networks having the same name. Many of our training centres have WiFi's, and they all are named the same, however they have different DNS addresses. That allowed me to connect to one training centre successfully, but then fail at all the others because the profile remembered the DNS address from the previous site. Going through the Microsoft Windows network settings did not detect where this was being held, because Sony were causing the problem not Microsoft (for once).

Hope this helps.

A view from EuroPython 2010

I have just returned from EuroPython which was, like last year, in Birmingham, UK. This post will not go into too much technical detail, email if you need more. I'm guessing that the reader does not want to know how Python implements IEEE 754 floating point format (that's even mentioned in the Programming Foundations course, do try to keep up!). It requires about 4000 lines of C code to convert between float and text - no, you didn't want to know that.
Neither am I going to give a blow-by-blow account of each talk - they are available on the europython 2010 website.
There were just under 400 delegates this year - slightly down on last year. The organisers should not be concerned though: early indications from rival conference YAPC Europe (Yet Another Perl Conference) is that attendance is approximately half of last year.
Organisation was slightly better than 2009, a number of lessons appear to have been learnt. Talk streams were still patchy, but I guess that's inevitable.


Multiprocessing, the GIL, and threading
The conference was opened by Russel Winder who impressed last year as well. The theme was very familiar to me, it told the story that Intel, and others, have been banging on about for a couple of years now - the fact that multi-core CPUs are here to stay and currently the only way to get the increase in speed beloved of developers. Russel went much further though, with my brain screaming ME TOO! Where Intel's solutions are distinctly small scale with core numbers in single figures, Russel spoke of much larger clusters (note the term). His contention was that current hardware solutions with cache are not scalable beyond 16, and neither are software solutions using threads. Message passing systems, like MPI, are a more likely future than threading systems like TBB or OpenMP. And no one in their right mind would be programming native threads. Here, here. A thousand times, here, here. All these lessons were learnt on mainframes in the 1970s: those that ignore history are destined to repeat it. It is a great shame that Russel's talk was rushed.

That did not stop discussion in the conference about the vagaries of the Global Interpreter Lock (GIL), the much maligned excuse for not doing multithreading in Python. Hey guys: multithreading is hard, error prone, and not necessarily all that faster. Let's just accept that and move on. There will be a reworked GIL in Python 3.2, work which has come out of Google's "Unladen Swallow" project. It will be interesting to see how much that helps (?) and what excuses people will use to avoid multithreading in the future.

I was browsing one of the book stalls, looking at an Advanced Python book when it hit me how much Python has moved forward in the past couple of years. The section on Multiprocessing mentioned an out-of-date module, not Subprocess and Multiprocessing. I dismissed the book as "out-of-date" even though the first edition was 2008 - QA's own Python courses have always covered those two modules. Then I realised that we have only had our own courses for about a year, although I have been tinkering with Python for (wow!) over ten years.


Python's progress
Version 2.7 of Python was released on 4th July. OK, I'll update the QAPYTH2 course material as soon as someone gives me the time. (Actually many of the significant changes in 2.7 are back-ports from Python 3.1, so I can flitch the material from my Python 3 course. Just don't tell the boss). Python 2.7 is the last Python 2 major release. Version 3.1 is now the only Python development stream, although 2.7 will continue to be maintained for around five years. When I started writing our own Python course material in late 2008 I was interrupted by the Python 3.0 release literally as I was writing the PowerPoint slides. I decided that a new course on Python 2 was daft, and switched to Python 3. It just so happened that I thought that Python 3 was a vast improvement for the language as well (don't cry for me, Perl 6). Six months later at EuroPython 2009 I realised I might have made a mistake, so I back-ported the course material to Python 2. I ended-up with two courses and "let the market decide". One year on and we have taught many more Python 3 courses than Python 2. and we are in a great position to move forward, unencumbered by legacy Python. Maybe it wasn't such a bad idea to target Python 3, but it was scary to be ahead of the curve.

Python 3.2 should be out around the end of the year. One speaker gave January 2011 and another said "before the end of this year". Take your pick. Christmas? Python 3.3 was mentioned a few times, it should include Google's "Unladen Swallow" (yes Ian, it is from "Monty Python and the Holy Grail") with a target of a 5 times performance improvement (European or Asian?). No dates yet.

My impression is that many developers still have not realised the benefits of Python 3, and have no plans to move over. They will.

It looks like Python will overtake Visual Basic in the Tiobe stakes in the next few months (it passed Perl a couple of years ago). Are developers realising the benefits of Python? They are.


The cheese shop (the Python module repository: PyPi)
(Recent course delegate: "but the Monty Python cheese shop had no cheese in it!". Me: "neither has PyPi")

One of the main benefits of Perl (what?) is CPAN. Dave Cross (well-known Perlmonger) has said that this was the main reason for not moving to Python. Well Dave, Python just had its 10,000th module uploaded (round of applause). Actually I'm not so sure that is such a good thing, duplication and crud makes navigation difficult. Still, it's an indication of popularity and progress.


One of the main benefits of Python 2 against Python 3 is the cheese chop. Here too Python 3 is catching up. The module numbers are still only a few hundred, but there major modules are appearing all the time, for example NumPy (numerical processing) was just released for Python 3.


Wot I learnt
Quite a lot, as always. I learnt the expression for such as __path__ is "dunder path". Nice one. I also learnt about the importance of the new unittest module (that'll have to go in) and changes in the way import works for Python 3.2. I discovered how technicians lie about language comparisons to their managers to justify using a cool product (actually, I already knew that). Speakers were at times less than accurate with their comparisons against Perl.

I learnt how HTML5 is going to make Microsoft Silverlight obsolete. Well the speaker did not actually say that, but I can dream.

There were some fascinating statistics, like there are more people in India with access to a mobile 'phone than to a flush toilet (please don't take the comparison further). I would love to use that in a course if only I could find the derivation.

The portability issues of using ActiveX are well-known to anyone using Microsoft's remote desktop (there are no portability issues - it's not portable), but apparently not to the South Korean government.

The spec. for HTML1 was three pages, for HTML5 is 900. That's progress.

Oracle had a stand at the conference, and they were giving out DVD's with developer's resource on for, let's see, Linux, PHP, Ruby, Python, and Oracle VM. Hummm, spot the missing language. Me: "What about Perl then?"; Oracle chappie: "oh, that's only used by system administrators for scripting". Go figure.

And you can write cool games in Python, drive neat little robots, automate PowerPoint slides (I gotta get me some of that, and the robot stuff). And the Python Software Foundation are as cheerfully disorganised as everyone else. As with all conferences, it is always nice to have opinions confirmed, and to be able to say from time to time "actually, I knew that".

Oh, and I saw a great example for "If then else for the lazy" in our UNIX fundamentals course:
./configure && make


Finally...
Python is self-assured, confident, and looking forward. The community is proud of its product, and itself. So they should be.

Python exercise - alternative solution

I promised to post an alternative solution to a Python exercise for an on-site today. For other interested parties, the code lists unused ports from /etc/services:

import sys

# set the file name depending on the operating system
if sys.platform == 'win32':
    file = r'C:\WINDOWS\system32\drivers\etc\services'
else:
    file = '/etc/services'

# Create an empty dictionary
ports = dict()

# Iterate through the file, one line at a time
for line in open(file):

   # Ignore lines starting with '#' and those containing only whitespace
   if line[0:1] != '#' and not line.isspace():

       # Extract the second field (seperated by \s+)
       pp = line.split(None, 1)[1]

       # Extract the port number from port/protocol
       port = pp.split ('/', 1)[0]

       # Convert to int, then store as a dictionary key
       port = int(port)
       ports[port] = None
   
       # Give up after port 200
       if port > 200: break

# Print any port numbers not present as a dictionary key
for num in xrange(1,201):
    if not num in ports:
        print "Unused port", num

Here is a smaller solution using Regular Expressions:

import sys,re

file = r'C:\WINDOWS\system32\drivers\etc\services' \
    if sys.platform == 'win32' else '/etc/services'

found = set()
for line in open(file):       
    m = re.search(r'^[^#].*\s(\d+)/(tcpudp)\s',line)
    if m:
       port = int(m.groups()[0])
       if port > 200: break
       found.add(port)
     
print set(range(1,201)) - found   

On the takeover of Sun by Oracle

I guess I might as well comment - everyone else and his dog is doing so. A collegue said "A sad, sad day for FOSS". I know what he means, but actually I disagree. Take-overs and mergers will always happen.

It is occasions like this that demonstrate the power of FOSS. FOSS gives us choice, and we can choose to stay with MySQL or migrate to PostgreSQL, which all the smart people were using anyway. IMHO, technically PostgreSQL knocks MySQL into a cocked hat, and the Oracle takeover of MySQL could give PostgreSQL the increase in popularity it deserves. Other databases are available.

Sun blew hot and cold on FOSS anyway, I could never figure out what their policy was. Java probably has too much momentum of its own for Oracle to screw it up, although if anyone can.... Never underestimate the ability of corporates to kill things off by hubris (I used to work for Computer Associates).

There will always be other languages, other databases, other operating systems.

Other hardware?

Back in the 1980s a collegue (Haydn Moston - are you still around?) told me that CISC (Complex Instruction Set Computer) chips could never last beyond 2000, and RISC (Reduced Instruction Set Computer) chips were the only way to go. The i386 familiy is CISC, and Sun Sparc is the most well-known RISC.

Intel are running out of steam and having to use multi-core, ten years out was actually not bad Haydn. I'm not sure that the switch to multi-core is connected specifically with CISC, but the possible loss of RISC machines is my biggest worry with the Oracle takeover. The number of mainstream instruction sets out there is disapointingly small. How I hate monocultures.

File notification events on Windows

One of my most popular posts is example code for Inotify on Linux. I have also been asked for similar code for Windows, so yer tis:

There are two different interfaces available on Win32, I prefer ReadDirectoryChangesW because it is easier to control:

// ------------------------------------------------------------------
// Clive Darke QA Training
// ReadDirectoryChangesW example
// ------------------------------------------------------------------

#define _WIN32_WINNT 0x0400      // <<<<<<<<>
#include <iostream>
#include <windows.h>

void DisplayLastError( LPSTR lpszText );

// ------------------------------------------------------------------

int main ( int argc, char *argv[] )
{
   HANDLE hDir;
   DWORD dwReturned;
   BOOL bResult;
   FILE_NOTIFY_INFORMATION *pNotify;
   
   if ( argc < 2 )
   {
      cerr << "You must supply a directory name" << endl;
      return 1;
   }

   // Note FILE_FLAG_BACKUP_SEMANTICS, which is the strange
   // attribute required to get a handle to a directory.   

   hDir = CreateFile (
          argv[1],                            // pointer to the file name
          FILE_LIST_DIRECTORY,                // access (read-write) mode
          FILE_SHARE_READ|FILE_SHARE_DELETE,  // share mode
          NULL,                               // security descriptor
          OPEN_EXISTING,                      // how to create
          FILE_FLAG_BACKUP_SEMANTICS,         // file attributes
          NULL                                // file with attributes to copy
        );

   char Buffer[MAX_PATH] = {0};

   while (TRUE )
   {
      char szAction[42];
      char szFilename[MAX_PATH];

      bResult = ReadDirectoryChangesW (hDir, &Buffer, sizeof(Buffer),
      TRUE, FILE_NOTIFY_CHANGE_FILE_NAME, &dwReturned, NULL, NULL);
    
   if ( !bResult )
      break;
    
   pNotify = (FILE_NOTIFY_INFORMATION *) Buffer;

   switch (pNotify->Action)
   {
    case FILE_ACTION_ADDED :    {
     strcpy (szAction, "added");
     break;
    case FILE_ACTION_REMOVED :
     strcpy (szAction, "removed");
     break;
    case FILE_ACTION_MODIFIED :
     strcpy (szAction, "modified");
     break;
    case FILE_ACTION_RENAMED_OLD_NAME :
     strcpy (szAction, "renamed");
     break;
    case FILE_ACTION_RENAMED_NEW_NAME :
     strcpy (szAction, "renamed");
     break;
          default:
     strcpy (szAction, "Unknown action");
    }

      wcstombs( szFilename, pNotify->FileName, MAX_PATH);
     cout << "File " << dwerror =" GetLastError();" lpmessagebuffer =" NULL;">

ASSOC on Windows 7

A while back I posted some suggestions for .bat files to run cmd.exe sessions for Python. I use these a lot for PHP and Perl as well. However there is an issue with them on Windows 7. "Out of the box", the ASSOC command gives "Access is denied", even though the user is an Administrator.

My first thought was to turn off the dreaded User Account Control settings, but that changed nothing. The eventual solution was to go into regedit and grant my user full control on registry key HKEY_CLASSES_ROOT, which is where the file associations are held. Fortunately Windows security for keys (and directories) is based on inheritence, and so keys within that hive will have the same permissions by default.

I can see why file association sould be seen as a security risk, so why don't Microsoft finally admit that the filename is not a good way to identify the file type, and use magic numbers like everyone else? Microsoft Office files, and other Microsoft proprietry formats, all contain magic numbers, and if you place one of these files onto Linux then it identifies them correctly without the need for file extensions.

UNIX's #! line for script files is a consequence of the magic number system. It is simple and incredibly flexible. When C# was developed the format of the .exe files (PE) had to change to support it. If they implemented #! then Microsoft would not have had that problem.

When I wrote the yash shell I simply read the commands from STDIN. On UNIX I needed no special code to support yash scripts - just the #! line. But on Windows I had to associate a file extension.

If I want to run Python 2 and Python 3 on the same machine on UNIX or Linux then I just have different #! lines at the top of the script. On Windows, because they have the same file extension, I have to redo the file associations every time I want to run one of them. Dire.

A winning feature of Perl 6

So what features do Python programmers love to say about their code? First off they like to crow about how clean their code looks without sigils, those nasty $, @, %, &, * that Perl programmers have to put up with. Ah, but that means you cannot do any interpolation in Python strings, because Python does not know the difference between bare text and a variable name.

Anyway, Python uses _ and __ (double underscore) to change the behaviour of variables, if these are not sigils then what is? Python uses an @ to prefix a decorator. Sounds like a sigil to me.

Perl 6 mandates that variables are declared using 'my' (or some other scope indicator), but Python does not require that. This means that in Python you cannot declare a variable for scope inside a local block, like an 'if' or loop.

Python does not use braces to delimit a block. Nice, unless you need to declare an empty block, so you need a 'pass' statement (reminds me of NEXT-SENTENCE).

Most of that is trivial to be honest, none is important enough to make a strong case either way.

So what is? Threading.

A recuring theme at the EuroPython conference was the fact that CPython is not truely multi-threaded because of the GIL (Global Interpreter Lock). Ruby has the Global VM Lock. There are various implementations which bypass this issue, Jython and Iron Python do not have a problem with multi-threading but both are several versions behind and I won't be holding my breath for a Python 3 version. Google is apparently working on a CPython version which eliminates the GIL, but again that is some way off.

In a classic "sour grapes", the Python community say that threading is too difficult anyway, so use processes instead. Very few people at the conference were swallowing that line.

Multithreading should be a selling point of Perl 6 and Parrot, but unfortunately is not yet complete and will not be in Rakudo Star. That is a great pity.

Perl 6 gets a release date!

Big news from the current Perl conference YAPC Europe 2009 here in Lisbon is that Rakudo * (star) will be released in spring 2010. Rakudo is Perl 6 on Parrot, which is an equivalent to the JVM or CLI.

Is that an official Perl 6 release? Well, it is the nearest to official that you will see. The Perl 6 team are keen to point out that there will be no such thing as a official version, just different implementations. I guess they are looking at Python for their inspiration there.

This could be make-or-break for Perl. It shouldn't be, Perl 5 will continue no matter what, but the new stuff is what will make Perl cool again in the eyes of the fashion conscious. We all know that fashion is fickle, illogical, and driven more by hormones than intellect. We also know how important it is in decision making. Much as it would like to, the Perl community cannot pretend that Perl 6/Rakudo is too far away to worry about. It is six months away.

My worry is that expectations will be high - the release will be incomplete and they are very open about that. The Rakudo "pumpking" (project manager), Patrick Michaud, sees this as a test release to encourage people to write applications to find bugs and feed back into the development. Trouble is that companies have been burned too often in the past to invest in applications on an incomplete and possibly flaky platform.

The dilemma is that if something is not released soon then Perl will have missed the boat - which is already upping anchor and about to steam away.

Update: read the comments from Pm - they are very good.

Perl 6 Questions and further information

This is following a recent Perl 6 seminar I gave.

Perl 5 style is still supported.

Named parameters go into %_
Unnamed parameters go into @_

Subroutines can be predefined with:

sub foo {…}

… is known as the "yadayadayada" operator.

Lexically scoped subroutines:

my sub mysub () is context { …}

Optional parameters: ? now comes after the parameter name.

sub mysub ($value, @harry, %ash, $opt?) {…} 

Named parameters: prefix is now : , instead of + and are optional unless they have a ! suffix.

sub named ($value, :@harry!, :%ash, :$other) {…}

...

named(42, other = 'fred', harry = @files);

Slurpy parameters, like perl 5 lists and arrays, are prefixed *

C style prototypes
Are not supported. It appears that when a subroutine is called before it is defined the parameter resolution still works correctly. However a runtime error is reported, not a compile time error (as it would with Perl 5 prototypes).

Dynamic types
These are called "traits". See Synopsis 12: Objects.

Zip operator
Has changed, it is now 'Z'. Behaviour with different length arrays has also changed between pugs and Rakudo, the result uses the shortest:

my @array = qw(The quick brown fox);
my @brray = qw(Now is the time for all);

my @all = @array Z @brray;
say @all.elems();
say @all[];

@all = @brray Z @array;
say @all.elems();
say @all[];

Gives:

8
TheNowquickisbrownthefoxtime
8
NowTheisquickthebrowntimefox

Important further change

When testing these changes in Rakudo I ran up against a change I had almost forgotten. When calling a subroutine, no whitespace is allowed between the sub name and the open parentheses. That slowed me up until I remembered. A backward step IMHO.

MySQL VARCHAR limits

A delegate recently queried the size limit on a VARCHAR field in MySQL. Our course documentation states a maximum of 255 characters: that is correct. For larger fields use TEXT.