Monday, June 23, 2008
The trials of a CPAN author
And that's the trouble with being a developer – people will actually try and use the stuff. So I get queries, and I get bug reports.
Generally the email queries are sensible, and I have had a few over the past week. First I, or rather a user, had an issue with Microsoft Visual Studio 2005, which is a version I haven't got around to using. It seems that good ol' Microsoft decided to fix the "DLL hell" problem by introducing a 'manifest' file into its link/load phase. So now we get "manifest hell" instead - the issues are well documented on the web. Of course the Perl MakeMaker does not generate a manifest file for its DLLs, and so my module wouldn't load and run. After some blind alleys and investigation I found a patch for MakeMaker, via perlmonks which I passed on. Of course that meant I had to document the thing as well. Meanwhile I found a small memory leak. OK, so that was worth it.
Then I got a bug report saying that the installation failed if the user did not have a C/C++ compiler. OK, I would have thought that if the product included an XS file, and the documentation showed compilation steps, then that would have been obvious. Clearly not, so more lines to the doc.
A user sent a "it would be great if…" feature request. "Good idea", I thought. "Simple patch", I thought. Only one of those thoughts was correct. Well you see it all worked on my Windows machines, it's just that it failed everywhere else. It wasn't until one of the cpan testers sent me a copy of his environment block that I cracked it. Their environment blocks were bigger than a hard limit I set – size matters. I hate hard limits and try and avoid them, but in this case I have to set one to a file mapping object. It's just that I was not handling the overflow case correctly.
There is nothing like a sustained bout of debugging to improve the diagnostics available in code. And of course I found another (unrelated) buglett as well – a possible race condition.
Then another bug report, this time someone is trying to compile the module under cygwin. Cygwin is a UNIX emulation environment, and this module is called Win32::EnvProcess – the clue is in the name. I check the OS name in the tests, but the thing won't compile and link anywhere except Windows. Of course not, RTFM. Alright, so Cygwin runs under Windows but it doesn't give access to the Windows kernel, it pretends it is UNIX.
Don't get me started about Strawberry Perl, just don't. Alright then. Strawberry Perl is a laudable package of open source products to enable you to install Perl without any dreaded Micro$osft licences. Good, great stuff, more power to them. Except that the Perl installation tools, like MakeMaker, doesn't really support it. So more magic in my Makefile.PL, already burdened by this stupid manifest issue. UNIX? Pah, that's easy.
Version 0.05 of Win32::EnvProcess has just been uploaded. God bless her, and all who sail in her.
Friday, May 09, 2008
Win32::StreamNames - new version!
The bug was an unexpected (and undocumented) behaviour of the Win32 API BackupRead when it finds an empty stream, and I discovered the fix just by experimentation. Good old Microsoft, guarenteed to keep life interesting.
Thanks to Frederic Medico for reporting the error.
Thursday, May 01, 2008
Yet another time calculation in Perl
This will probably change because the specification is a bit "wooly".
#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
sub str2time
{
my $in = shift;
my ($hour, $min, $sec) = split (/:/, $in);
my ($mday,$mon,$year) = (localtime(time))[3,4,5];
my $retn;
# timelocal throws an exception on an invalid date/time
eval {
$retn = timelocal($sec,$min,$hour,$mday,$mon,$year);
};
return $retn; # undef on error
}
print "Please enter the first time in HH:MM:SS format: ";
my $intime = <STDIN>;
chomp $intime;
my $time1 = str2time($intime);
die "Invalid time: $intime\n" if !defined $time1;
print "Please enter the 2nd time in HH:MM:SS format: ";
$intime = <STDIN>;
chomp $intime;
my $time2 = str2time($intime);
die "Invalid time: $intime\n" if !defined $time1;
my $diff;
if ( $time1 > $time2 ) {
$diff = $time1 - $time2
}
else {
$diff = $time2 - $time1
}
my $mins = int($diff / 60);
my $hours = int($mins / 60);
$mins = $mins - ($hours * 60);
my $secs = $diff - ($mins * 60);
printf "Difference: %02d:%02d:%02d\n", $hours,$mins,$secs;
Friday, April 18, 2008
Away from the dark side - Python
So these are the impressions of a novice - and someone coming from Perl at that. I will probably come-up with justifications and solutions for the shortfalls as I learn, they are probably a mark or my ignorance rather than anything wrong with Python. Having said that, I was surprised how many things are missing.
The good
1. Forced code formatting
2. No stupid sigils
3. OO built-in
4. Default variables in a function are not global
5. Documentation
6. OS specifics are removed from the core
7. Excellent string handling
8. Intuitive file IO
9. Method calls on built-in types instead of weird system variables
10. No GOTO
The bad (IMHO)
1. No constants
Even True and False can be altered!
2. No scoping within loops and 'if' statements
All variables outside functions are globals
3. No interpolation - an effect of the lack of sigils I guess
4. Comparisons between numbers and strings give unexpected results, and NO WARNINGS! A poor and surprising feature
5. I know ++, --, +=, etc. are arguably poor coding, but I missed them.
Still, 10 good and only 5 bad.
Wednesday, February 13, 2008
Win32::EnvProcess
This module enables the user to alter or query an unrelated process's
environment variables.
Windows allows a process with sufficient privilege to run code in another
process by attaching a DLL. This is known as "DLL injection", and is used here.
SYNOPSIS
use Win32::EnvProcess qw(:all);
use Win32::EnvProcess qw(SetEnvProcess);
my $result = SetEnvProcess($pid, env_var_name, [value], ...);
use Win32::EnvProcess qw(GetEnvProcess);
my @values = GetEnvProcess($pid, env_var_name, [...]);
use Win32::EnvProcess qw(DelEnvProcess);
my $result = DelEnvProcess($pid, env_var_name, [...]);
use Win32::EnvProcess qw(GetPids);
my @pids = GetPids($exe_name);
This is another case where a question on perlmonks generated the interest. How do I get the child to create/alter an environment variables in the parent? Had this been on UNIX then the answer would be simple - you can't without having co-operating proceses. On Windows however DLL injection makes this possible.
Enjoy!
I have done a brief investigation on how this might be achieved on Linux, but I don't think it can fly. The API ptrace(2) is a start, but the problem is in creating a thread in the host. Addresses in the host can be changed, but the environment block is not at a fixed location so far as I know, and without a symbol table I'm not sure how you would find it.
Wednesday, January 30, 2008
Win32:IdentifyFile
This one was written because of a conversation on perlmonks Chatterbox. A module author was bemoaning the fact that Windows files do not have an inode number. I waded in to say that they do, kinda. However it's not easy to use, and requires a C API to be called. Hence the module, written using XS.
There are two functions, IdentifyFile() and CloseIdentifyFile().
IdentifyFile() returns 3 components to uniquely identify the file or directory. In list context these are returned as a 3 item list, in scalar context they are joined together as a single string using ':'. It is therefore simple to compare two identities in perl, using 'eq'.
Just getting this information could result in a race condition. The file could be deleted, possibly by another process, between getting the information and using it in a test. Worse, another file might be created with the same file index meanwhile. To prevent this scenario, files (or directories) are opened internally by IdentifyFile(), and kept open until CloseIdentifyFile() is called (files are not physically deleted until all open file handles are closed).
I wonder how many people think of that when using an inode number on UNIX?
Wednesday, January 23, 2008
UNIX command equivalents in Perl
For the very latest list see my perlmonks post: UNIX command equivalents in Perl
Monday, January 21, 2008
First impressions in India
My hotel is a Marriott, not everyone's cup of tea, but I think they are excellent. In my previous job we always stayed in them, nowadays I rarely get the chance; usually the hotel is um, economic. Anyway, the luxury of the hotel is in stark contrast with the surroundings. Quite a culture shock, it is like living in a travelogue.
Stereotypical taxi from the airport, including the dangly bits on the mirror. Streets being brushed by doubled-up women holding rushes, and this at 1:30 AM. The reflective safety jackets were particularly incongruous. And yes, there was even a cow in the road.
The guards, sniffer-dog, and metal detectors at the hotel entrance were a bit of a shock. The helpful and polite staff are overwhelming. I think I have said "thank-you" more than any other phrase since I got here.
Walking out of the hotel on Sunday morning I was surrounded by tricycle drivers wanting to take me for a tour. When I politely declined they disappeared, and I was not bothered again. Just outside the hotel the poverty is heart-rending, yet the busy road ignores them. I have never seen anyone riding side-saddle on a motorbike while wearing a sari before.
Friday, January 04, 2008
(Windows) What is the difference between a thread's HANDLE and its ID?
The most obvious difference between a HANDLE and a TID is that a HANDLE is specific to a process, whereas a TID is system wide.
A HANDLE is actually an unsigned int (whereas a UNIX file descriptor is a signed int) and is an index into the process's Handle Table, which is maintained by kernel.
A TID is a type of Client ID, the other type of Client ID being a PID (Process ID). TIDs and PIDs are generated in the same namespace and used by the kernel to identify these objects system wide.
So, why would you need the TID? You don't. At least, not often. A TID may be passed between processes, and another process can then get a HANDLE to that thread using the OpenThread() API.
Yes, you can manipulate threads in other processes - the caveat being that security may stop you, particularly in Windows Vista. Once you have a HANDLE to another thread you can do all those wonderful things, Suspend, Resume, Wait, GetExitCode, as you can if the thread is in the same process. One of the useful APIs in this context is PostThreadMessage(). This sentence is mis-leading, PostThreadMessage takes a TID, see comment from Hibou57 below.
BTW: make sure that HANDLE is closed, the thread cannot truly die until all HANDLEs to it are closed - that is how GetExitCodeThread() works. And you thought zombies only occurred on UNIX?
It is possible to use DuplicateHandle() to get a handle opened by another process, but that is far more painful than the MSDN implies.
Wednesday, December 05, 2007
Linux inotify example
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>
#include <sys/inotify.h>
void get_event (int fd, const char * target);
void handle_error (int error);
/* ----------------------------------------------------------------- */
int main (int argc, char *argv[])
{
char target[FILENAME_MAX];
int result;
int fd;
int wd; /* watch descriptor */
if (argc < 2) {
fprintf (stderr, "Watching the current directory\n");
strcpy (target, ".");
}
else {
fprintf (stderr, "Watching %s\n", argv[1]);
strcpy (target, argv[1]);
}
fd = inotify_init();
if (fd < 0) {
handle_error (errno);
return 1;
}
wd = inotify_add_watch (fd, target, IN_ALL_EVENTS);
if (wd < 0) {
handle_error (errno);
return 1;
}
while (1) {
get_event(fd, target);
}
return 0;
}
/* ----------------------------------------------------------------- */
/* Allow for 1024 simultanious events */
#define BUFF_SIZE ((sizeof(struct inotify_event)+FILENAME_MAX)*1024)
void get_event (int fd, const char * target)
{
ssize_t len, i = 0;
char action[81+FILENAME_MAX] = {0};
char buff[BUFF_SIZE] = {0};
len = read (fd, buff, BUFF_SIZE);
while (i < len) {
struct inotify_event *pevent = (struct inotify_event *)&buff[i];
char action[81+FILENAME_MAX] = {0};
if (pevent->len)
strcpy (action, pevent->name);
else
strcpy (action, target);
if (pevent->mask & IN_ACCESS)
strcat(action, " was read");
if (pevent->mask & IN_ATTRIB)
strcat(action, " Metadata changed");
if (pevent->mask & IN_CLOSE_WRITE)
strcat(action, " opened for writing was closed");
if (pevent->mask & IN_CLOSE_NOWRITE)
strcat(action, " not opened for writing was closed");
if (pevent->mask & IN_CREATE)
strcat(action, " created in watched directory");
if (pevent->mask & IN_DELETE)
strcat(action, " deleted from watched directory");
if (pevent->mask & IN_DELETE_SELF)
strcat(action, "Watched file/directory was itself deleted");
if (pevent->mask & IN_MODIFY)
strcat(action, " was modified");
if (pevent->mask & IN_MOVE_SELF)
strcat(action, "Watched file/directory was itself moved");
if (pevent->mask & IN_MOVED_FROM)
strcat(action, " moved out of watched directory");
if (pevent->mask & IN_MOVED_TO)
strcat(action, " moved into watched directory");
if (pevent->mask & IN_OPEN)
strcat(action, " was opened");
/*
printf ("wd=%d mask=%d cookie=%d len=%d dir=%s\n",
pevent->wd, pevent->mask, pevent->cookie, pevent->len,
(pevent->mask & IN_ISDIR)?"yes":"no");
if (pevent->len) printf ("name=%s\n", pevent->name);
*/
printf ("%s\n", action);
i += sizeof(struct inotify_event) + pevent->len;
}
} /* get_event */
/* ----------------------------------------------------------------- */
void handle_error (int error)
{
fprintf (stderr, "Error: %s\n", strerror(error));
} /* handle_error */
/* ----------------------------------------------------------------- */
Monday, September 10, 2007
Just for fun
You have 20 votes left today.
You gained 15 experience points, isn't that nice?
Congratulations, you have been promoted to Chaplain!
You have 598 points until level 12 - Deacon.
That is from 272 writeups in about a year-and-a-half.
I wish there was a site like perlmonks for PHP!
Friday, August 31, 2007
Parallelism
The "free lunch" for developers really is over, and the industry will not know what hit it. Software design and building is about to change, and it will be painful. Strangly, a programmer who had been asleep for 20 years and has suddenly awoke would not see anything new. That programmer might well be better equipped for the multi-core world than those that "kept current". Why? Because the skills practiced then will be even more important now, and the sloppy programming that has become so commonplace will no longer do the job.
Revealing comment:
The lead Intel instructor (an authority on parallel processing) was asked how people who have built everything on OO would cope with parallel processing. The answer was: "They are screwed!".
Now you might say that we have been doing multi-threading for years. Yes, but that is not parallel processing. Multi-threading is usually about splitting I/O tasks: GUI windows, overlapped comms and disk I/O are typical. Parallel processing is about splitting (decomposing) processor tasks, which is a whole new barrel of fish. At least, it is for most people.
This revolution has started, but is not yet in full flow, and won't be for a couple of years. The reason is that there is well enough work for an extra CPU with other processes, like services, and the multi-threading we have right now. The real need willl be with quad and 8 (oct?)-core processors.
Watch this space
Tuesday, July 10, 2007
The real jungle
I found Singapore to be hot and steamy, and I am talking about the climate. The place has still not shaken off its colonial past. Its own character does come through sometimes, and more power to it. The people seem to be friendly and open without the hang-ups of many large cities. Of course I was only there for a week and I only saw a small part of it.
As usual I looked at scripts that delegates brought in. I have also had a couple of scripts from other people recently. I see very common bad practices regularly:
No use warnings
No use strict
All production scripts should have these set.
Calling subroutines using the & prefix. Why do people do this? Because that is the way we used to do it in perl 4, and people do not update their skills. The & prefix ignores prototype checking, and passes the current value of @_ into the call if no other argument is specified. Don't do it!
Calling external programs instead of Perl built-ins. I have seen `date` used instead of localtime(), `cd $dir`, `pwd`, `mkdir`, `rm`, `egrep`, and so on. This is not just lazy, it is grossly inefficient. It is also a security risk, by the way. I published a list of UNIX commands and perl equivalents, I will try and update it soon.
Monday, June 25, 2007
The threading jungle
Multi-threading has been around a long time. I first heard the term in the mid-1980s when discussing how ICL were going to host UNIX on VME. The first real implementation I saw was on Apollo NCS, a system that had many innovative features. ICL, VME, and Apollo have all gone, victims of acquisitions. Threads remain.
Multi-threading hit the mainstream with Windows NT, the system was built on it. The consensus around us UNIX hackers was that Microsoft had to encourage threads because their processes were so inefficient - no fork/exec you see. Probably utter twaddle; you know what UNIX hackers are.
Threads brought synchronisation problems. Of course they did. System programmers were no strangers to this type of issue, just the context of threads was new(ish). Way back when, on ICL VME, all we had were "test and set" and "decrement and set" instructions that were atomic. When I were a lad we had to build our own primitives, there was a particularly handy instruction that let you not only raise an interrupt (event/signal, whatever) but allowed you to send a two-word data block with it. Kids of today … don't know they're born… forty miles 't pit… etc.
I got to write real synchronisation primitives when I did the VME port of the MIMER RDBMS in the early '80s. There is nothing like having to write the damn things to understand how they work. I did not design them, I hastily add - that was done by some egg-head at
First rule when writing co-operating threads: things only go wrong at the worse possible moment. And that is usually after the thing has gone into production.
The Windows APIs are fairly easy to use, certainly compared to knitting your own. The main wrinkles in using them are a few peculiarities with the C/C++ runtime-library. I find it amazing that so many system engineers insist in using CreateThread instead of _beginthreadex. Back in Visual Studio 2.0 this was understandable, even I did it. The documentation was, um, "challenged". Now, with the MSDN, there is no excuse, the problems are well documented. Still people insist on using the wrong API. My theory, for what it is worth, is that a call to _beginthreadex, with its associated casts, looks messy on the screen, surrounded as it is by unrivalled elegance (sic). CreateThread looks good, _beginthreadex is ugly. For wotsit's sake! A thing of beauty with a memory leak is bad code, and I don't care what it looks like. Humph.
Still, you thought that was ugly? Try finding the thread-safe RTL functions in pthreads. Threads had to be retro-fitted to UNIX. Many fought against it, even the hero Torvold in the clone wars, but resistance was futile. Microsoft made life easy for us developers by having a whole special runtime-library just for multi-threading, ungrateful pups that we are. The pthreads implementations on UNIX have no such luxury. Instead there is a whole mess of different functions that are re-entrant with the _t suffix. Bye-bye portability. Finding which functions these are is a bit hit and miss, sometimes they are not in the man pages, and the standard only lists minimum requirements. Try teaching Condition Variables to a class already spaced out by mutxes. The word "predicate" makes their eyes glaze over. The fixed grin is the give-away: "Gone, solid gone", as Baloo would say. I'm sure Kipling was a coder, who else would write a whole poem about a conditional statement?
Wednesday, May 30, 2007
What I did on my holiday
2. Read "Parallel Programming in C with MPI and OpenMP" by Michael J. Quinn
Very American (the first computer was ENIAC - really?) No mention of the ICL DAP (Distributed Array Processor) - one of the first commercially available parallel processors.
Nice to see terms like "Functional Decomposition" from the 1970s recycled to be used in a parallel computing methodology.
It really bugs me when authors come up with "Creating Arrays at Run Time" and give code like this:
int *A;
A = (int *) malloc (n * sizeof(int));
When there is a perfectly good feature in C99 to do it:
The author uses for loops to access arrays quite often, but there is no mention of optimisers, and the effect of optimisations on parallel processing. Ho hum.
Thursday, April 05, 2007
Where have you been?
What's happened?
Well for starters the company I work for got taken over, and that caused a lot of dust - mostly settled now.
I have been getting into the new Korn shell 93, and I hope to post something on that soon.
Just today I got my "Happy Monkday" from perlmonks - I have been a monk for a year. Good fun!
Oh where is Perl 5.10? Soon? Please!
Thursday, August 31, 2006
Notifications
(CPAN) Win32::ChangeNotify - does it support (Win32 API) ReadDirectoryChangesW and overlapped IO? If not, how does it avoid missing changes?
(CPAN)Linux::Inotify (and Inotify2) support the Linux kernel API inotify which (perhaps) replaces dnotify. Supported from kernel 2.6.13.
Neither interface is on Fedora core 4, which is 2.6.11.
Further digging required.
Monday, July 24, 2006
Fun with Perl module loading
A. As soon as it is encountered!
For example, take the following code:
use AModule;
use BModule;
BEGIN {
print __PACKAGE__." BEGIN block\n";
}
use AnOther;
$syntax_error = 42
print "Starting main program\n";
print "Ending main program\n";
The order of execution is:
AModule BEGIN block
AModule main body
BModule BEGIN block
BModule main body
main BEGIN block
AnOther BEGIN block
AnOther main body
syntax error at main.pl line 18, near "print"
Execution of main.pl aborted due to compilation errors.
So, the main program's BEGIN block is not necessarily the last one executed. Granted, normally it is, since we usually use all our modules before the BEGIN block in main, but we don't have to.
Note also that the blocks are executed even though we have a syntax error, and the program fails to compile.
Q. What use is that?
A. We can alter the way subsequent modules are loaded
Usually that will be by altering @INC:
BEGIN {
if ( defined $ENV{TESTLIB} ) {
unshift @INC, $ENV{TESTLIB}
}
}
Now all subsequent module loads will search the directory in the environment variable first.
Q. @INC is just a list of directories, Right?
A. Wrong! It can also contain code references.
References to subroutines found by the module loader will be executed as they are found:
BEGIN {
print __PACKAGE__." BEGIN block\n";
my $code = sub { print "\@INC code\n" };
unshift @INC,$code;
}
use AnOther;
Gives:
main BEGIN block
@INC code
AnOther BEGIN block
Q. Is that it?
A. Of course not.
The subroutine is passed two arguments, the code reference itself and the name of the module it is trying to load. This enables us to track every module loaded. So:
BEGIN {
print __PACKAGE__." BEGIN block\n";
my $code = sub {
my (undef, $loading) = @_;
my ($package, $filename, $line) = caller;
print "Loading $loading from $package\n"
};
unshift @INC,$code;
}
use AModule;
use BModule;
use AnOther;
Gives:
main BEGIN block
Loading AModule.pm from main
AModule BEGIN block
AModule main body
Loading BModule.pm from main
BModule BEGIN block
BModule main body
Loading AnOther.pm from main
AnOther BEGIN block<>
Of course, further diagnostics could be added, like filename, line, and date/time. I'll leave that to you.
Q. So, @INC is more powerful than its sister %INC, which is just used for diagnostics.
A. Err, no. %INC is used by perl to see if a module is already loaded.
Q. And how is that useful?
A. We can force a module reload by removing its entry from %INC. Consider:
In one part of our code we want to force a different version of a module to be loaded (don't ask why).use strict;
use warnings; # DON'T use -w
use A;
use MyB;
use C;
A::mysub(); # Original modules used
delete $INC{'A.pm'}; # Force perl to reload
unshift @INC,'mydir'; # Change @INC
{
no warnings 'redefine'; # No 'redefined' warnings
require 'A.pm'; # Reload module
}
A::mysub(); # Module in 'mydir' used
Work it out yourself!
Thursday, May 04, 2006
Splitting whitespace
"If EXPR is omitted, splits the $_ string. If PATTERN is also omitted, splits on whitespace (after skipping any leading whitespace)."
Then, later in the documentation:
"As a special case, specifying a PATTERN of space (' ') will split on white space just as "split" with no arguments does. … A "split" with no arguments really does a "split(' ', $_)" internally."
So, how many whitespace characters is that? A single space as a delimiter implies a single space is used for the split, but it actually does 'one or more whitespace':
$_ = ' This is some text';Produces
@a = split;
$" = '|';
print "@a\n";
This|is|some|textSo leading whitespace is ignored, and one or more whitespace is used as a delimiter. There is a (documented) subtle difference with \s+:
$_ = ' This is some text';Produces:
@a = split /\s+/;
$" = '|';
print "@a\n";
|This|is|some|textNotice that the first element of the resulting list is empty, which was not previously the case.
A few questions arise from this. First, what is this ' ' syntax all about? Don't we need a regular expression match?
$_ = 'xxxThisxxisxxsomexxtext';Produces:
@a = split 'x';
$" = '|';
print "@a\n";
|||This||is||some||te|tSo it does work, except not exactly the same as a single space, it does not match 'one or more' (x+), so the space is magic. To be fair the documentation does say that ' ' is a special case. But the documentation does not show the syntax of a string literal, it specifically shows that an RE delimited with / / is required. Single quotes works with regular expressions, and with multiple characters (without a leading 'm'). But double quotes or other characters do not work unless preceded with 'm'.
Second question. What does whitespace mean? Is ' ' the same as \s in this case? Normally, of course, it is not, but in this case it is! ' ' is very special.
Thursday, April 27, 2006
Inside-out accessor methods - version 2
my (%speed, %reg, %owner, %mileage);
my %hashrefs = ( speed => \%speed,
reg => \%reg,
owner => \%owner,
mileage => \%mileage);
sub set {
my ($self, $attr, $value) = @_;
my $key = refaddr $self;
if ( !exists $hashrefs{$attr} ) {
carp "Invalid attribute name $attr";
}
else {
$hashrefs{$attr}{$key} = $value;
}
}