...and master of none

Java and the Intrinsic Lock

2011-10-19T23:39:00.005-06:00

I was working on converting a Java application to a multi-threaded application today and wanted to use the simplest means possible. This involves the following steps:

Make the class implement Runnable or descend from the Thread class
Put the previously sequential code into threads
Use the "synchronized" keyword on methods that should be critical sections
Join the threads before working on the results

Now, there are variations on this theme. Because each object in Java has an intrinsic lock, you can use any old Object as a mutex by calling synchronized(object){} around the critical section you want to protect.
so this:


private synchronized foo(){
  // critical section stuff
}

is the same as this:


private foo(){
  synchronized(this){
    // critical section stuff
  }
}

And when trying to synchronize multiple threads, both are wrong!!

The problem is that each object has an intrinsic lock, not each class. This is easiest to see in the second case. Because each thread is a different object, or the different thread objects contain their own instance of our Runnable class, every single lock is different. The first case has the same problem, not because the method is different in each case, but because the method is logically different.

So, what should we do?
In each case, we need to make the intrinsic lock static (or global to all instances of the class). In the first case, this can be accomplished by making the method static:


static synchronized foo(){
  // critical section stuff
}

in the second case, we need some other object that is static to use as a lock.


static object myLock = new Object();
private foo(){
  synchronize(myLock){
    // critical section stuff
  }
}

Why would we use the second method? Probably because we want more fine grained protection over the locking than at the procedure level, or because we are referencing properties that should not be made static inside the function.

Threaded Search

2011-10-04T15:45:00.002-06:00

I spent a little bit of time threadifying the classic mergesort.This sort works by using recursively dividing the array to be sorted down to some depth, and then eventually merging the sorted members of the array. This sort also has the desirable characteristic that it sorts in-place. There is no additional memory location required. This is a *good thing* when you have hundreds of thousands of elements in your array. In this particular implementation, an insertion sort is used when the recursive division has less than 100 elements. Multi-threading *could* occur at the first n levels of bifurcation, at the bottom level only, or at some intermediary number of levels. Limiting the number of threads and making them a small multiple of the number of CPU cores causes the threading overhead and context switching to be limited as well. The optimal multiplier appears to be quite low, only about 2 threads per CPU core. For these reasons, I choose to thread only at one depth level.Let's look at the code:


#include 
#include 
#include 
#include 
#include 
#include "mergesort.h"

#define TRUE 1
#define FALSE 0

#define THREADED

// function prototypes
void merge(int A[], int p, int q, int r);
void insertion_sort(int A[], int p, int r);

const int INSERTION_SORT_CUTOFF = 100; //based on trial and error
const int THREAD_DEPTH = 1; //2^(n+1) threads created at this level

struct sortstruct {
	int p;
	int r;
	int d;	// depth
	int *A;
};

/*
 * insertion_sort(int A[], int p, int r):
 *
 * description: Sort the section of the array A[p..r].
 */
void insertion_sort(int A[], int p, int r)
{
	int j;

	for (j=p+1; j<=r; j++) {
		int key = A[j];
		int i = j-1;
		while ((i > p-1) && (A[i] > key)) {	
			A[i+1] = A[i];
			i--;
		}
		A[i+1] = key;
	}
}

void * mergesort_function(void* arg){
	struct sortstruct * s;
	s = (struct sortstruct *) arg;
	printf("In mergesort thread (%d, %d) \n", s->p, s->r);
	mergesort(s->A, s->p, s->r, s->d);
	return NULL;
}

/*
 * mergesort(int A[], int p, int r):
 *
 * description: Sort the section of the array A[p..r].
 * track depth d
 */
void mergesort(int A[], int p, int r, int d)
{
	if (r-p+1 <= INSERTION_SORT_CUTOFF)  {
		insertion_sort(A,p,r);
	} else {
		int q = (p+r)/2;

#ifdef THREADED
		if(d <= THREAD_DEPTH){
			int status;
			pthread_t thread1, thread2;
			struct sortstruct *s1, *s2;

			s1 =(struct sortstruct *) malloc(sizeof(struct sortstruct));
			s1->A=A; s1->p=p; s1->r=q; s1->d=d+1;
			status = pthread_create(&thread1, NULL, mergesort_function, (void*) s1);
			if(status) exit(status);

			s2 = (struct sortstruct*)malloc(sizeof(struct sortstruct));
			s2->A=A; s2->p=q+1; s2->r=r; s2->d=d+1;
			status = pthread_create(&thread2, NULL, mergesort_function, (void*) s2);
			if(status) exit(status);

			pthread_join(thread1, NULL);
			pthread_join(thread2, NULL);

			free(s1);
			free(s2);
		}else{
#endif
			mergesort(A, p, q, d+1);
			mergesort(A, q+1, r, d+1);
#ifdef THREADED
		}
#endif
		merge(A, p, q, r);
	}
}

/*
 * merge(int A[], int p, int q, int r):
 *
 * description: Merge two sorted sequences A[p..q] and A[q+1..r] 
 *              and place merged output back in array A. Uses extra
 *              space proportional to A[p..r].
 */     
void merge(int A[], int p, int q, int r) 
{
	int *B = (int *) malloc(sizeof(int) * (r-p+1));

	int i = p;
	int j = q+1;
	int k = 0;
	int l;

	// as long as both lists have unexamined elements
	// this loop keeps executing.
	while ((i <= q) && (j <= r)) {
		if (A[i] < A[j]) {
			B[k] = A[i];
			i++;
		} else {
			B[k] = A[j];
			j++;
		}
		k++;
	}

	// now only at most one list has unprocessed elements.

	if (i <= q) { 
		// copy remaining elements from the first list
		for (l=i; l<=q; l++) {
			B[k] = A[l];
			k++;
		}
	} else {
		// copy remaining elements from the second list
		for (l=j; l<=r; l++) {
			B[k] = A[l];
			k++;
		}
	}

	// copy merged output from array B back to array A
	k=0;
	for (l=p; l<=r; l++) {
		A[l] = B[k];
		k++;
	}

	free(B);
}
,

We changed the signature of the Mergesort function to include a parameter "d", for depth. This requires the initial call to pass the number 0 to prime the recursive function.There is a wrapper function (mergesort_function) that is the threaded function call. We used this call notation


nice -n -20 ./mergesort 10000000 2

Here are the observed results:

Number of Elements	Unthreaded Time (s)	Threaded Time
1000000	4.330000	4.040000
1000000	4.230000	4.150000
1000000	4.150000	4.040000
1000000	4.220000	4.060000
average	4.232500	4.072500

There appears to be a slight advantage on my 2 CPU virtual machine when using the multi-threaded application. However, it does not approach anything like the multiplier that we would expect for an application that is using two cpus instead of one. This is probably due to the way that vmWare is managing the CPUs. I'd be interested to see how it works in a native machine.

The most efficient algorithm to scan a bitmap

2011-09-28T01:05:00.002-06:00

Yesterday in class, we were learning about the linux scheduler. How you can have several different algorithms for scheduling, etc. All very nice. The actual scheduler used in linux 2.6 has 140 FIFO queues; one for each possible priority. The scheduler takes the first job from the highest priority queue. The queues have a 140 bit bitmap. Because the bitmap gets scanned every single time there is a new quantum, the algorithm for scanning the bitmap needs to be very fast.

Our professor asked how we would go about scanning the bitmap. As an example, he put up an algorithm that would compare a single bit, then shift the bits left and keep track of the number of shifts. Obviously room for improvement there. I suggested that the algorithm should be just an integer base 2 log function. To which the professor asked me how I would implement it. I wanted to say: "in hardware", but he was already moving forward. But really, the question of "what is the most significant bit?" is exactly the same question as "what is the integer base 2 log?", one of these two problems has been solved already, in hardware. And actually, I'd solved a problem similar to this years ago, again using a little bit of assembler.

Then he presented how linux does it: by using a binary search algorithm on the highest integer, then on the second integer, etc, until all 140 bits have been found. To me this looks like a O(log n) algorithm because that's what a binary search does.

However, big O notation assumes that we are dealing with large numbers, and 140 is just not that large. A bit scan function is a 0(n) algorithm. However, if your lower order values are small enough, and the problem small enough, then you can consistently beat an 0(log n) algorithm.

Here is my approach:

I am going to specialize on the processor. I am working with a 64-bit Intel processor. I would expect any kernel code to have libraries that are platform specific. I am just implementing one for my platform. In this case, it is not that big of a deal, because I just have to implement a intLog2 function that takes a 64-bit int and returns one as well. Second, I am going to use an assembly code operation that performs a reverse bit scan to do my log function (thank you Intel). On a 64-bit machine, that means I only have 3 operations to perform on that 140-bit bitmap. I am also going to use the operations registers to perform the function. Finally, I am going to wrap that assembly operation in a #define so that it is executed inline.

Performing the scan on all three elements of the bitmap should simply take the most significant long (I'm calling it b[0], and verify that it has any values at all, if it does, return the scanned value, same for b[1] and b[2]. If none of them have any values, then simply return -1 (for no values in the queue).

Ok, here is the code:


/*
 ============================================================================
 Name        : ScanBitmap.c
 Author      : Aaron Wells
 Copyright   : Freely Distributable
 Description : C, Ansi-style
 ============================================================================
 */

#include 
#include 

/*
 * Integer Base 2 Log Function
 * The integer base 2 log is simply the number of the high bit that is set
 * This is accomplished by using the Intel bsr instruction (Bit Scan Reverse)
 * Command is available on 386 or better CPU
 * The values should be loaded and retrieved from registers
 */
#define intlog2(x)       \
		({ long __value, __arg = (x);   \
		__asm__ ("bsr %1,%0": "=r" (__value): "r" (__arg));  \
		__value; })

// 3 longs is 3 * 64 bits = 192 bits (52 extra)
#define BITMAP_SIZE 3

/* 
 * This is my "algorithm", not much to it, is there??
 */
static long scanHiBit(unsigned long b[BITMAP_SIZE]){
	if(b[0]) return intlog2(b[0]) + 128;
	if(b[1]) return intlog2(b[1]) + 64;
	if(b[2]) return intlog2(b[2]);
	return -1;
}

static void printBitmap(unsigned long b[BITMAP_SIZE]){
	printf("%lx %lx %lx", b[0], b[1], b[2]);
}

int main(void) {
	unsigned long b1[BITMAP_SIZE] = {-1, -1, -1};
	unsigned long b2[BITMAP_SIZE] = { 0, -1, -1};
	unsigned long b3[BITMAP_SIZE] = { 0,  0, -1};
	unsigned long b4[BITMAP_SIZE] = { 0,  0,  1};	// almost worst case
	unsigned long b5[BITMAP_SIZE] = { 0,  0,  0};	// worst case

	printBitmap(b1);
	printf(" :%ld\n",scanHiBit(b1));

	printBitmap(b2);
	printf(" :%ld\n",scanHiBit(b2));

	printBitmap(b3);
	printf(" :%ld\n",scanHiBit(b3));

	printBitmap(b4);
	printf(" :%ld\n",scanHiBit(b4));

	printBitmap(b5);
	printf(" :%ld\n",scanHiBit(b5));

	return EXIT_SUCCESS;
}

There is always room for improvement, and this code is no different. To really do this right, you would pass in the pointer to the start of the bitmap and do the whole thing in assembler. Not very portable, but very fast.

By the way, there would be no reason that you couldn't do a binary search as a fall-back algorithm for CPUs that don't have a reverse scan instruction.

As a follow-up, I may compare my approach (I can't take credit for the algorithm because it is basically a built-in instruction) to the scheduler scanner in linux... another day.

file_operations structure changes

2011-09-22T20:55:00.002-06:00

Working on a device driver for Linux 3.0.4. In class we are modifying some existing code:

static struct file_operations simple_fops = {
    .read =       simple_read,
    .write =      simple_write,
    .ioctl =      NULL,
    .open =       simple_open,
    .release =    simple_release,
};

However, this doesn't compile with the latest kernel source. The error is:
error: unknown field ‘ioctl’ specified in initializer
The new structure, as defined in <source>/include/linux/fs.h is:

struct file_operations {
        struct module *owner;
        loff_t (*llseek) (struct file *, loff_t, int);
        ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
        ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
        ssize_t (*aio_read) (struct kiocb *, const struct iovec *, unsigned long, loff_t);
        ssize_t (*aio_write) (struct kiocb *, const struct iovec *, unsigned long, loff_t);
        int (*readdir) (struct file *, void *, filldir_t);
        unsigned int (*poll) (struct file *, struct poll_table_struct *);
        long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long);
        long (*compat_ioctl) (struct file *, unsigned int, unsigned long);
        int (*mmap) (struct file *, struct vm_area_struct *);
        int (*open) (struct inode *, struct file *);
        int (*flush) (struct file *, fl_owner_t id);
        int (*release) (struct inode *, struct file *);
        int (*fsync) (struct file *, int datasync);
        int (*aio_fsync) (struct kiocb *, int datasync);
        int (*fasync) (int, struct file *, int);
        int (*lock) (struct file *, int, struct file_lock *);
        ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int);
        unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long);
        int (*check_flags)(int);
        int (*flock) (struct file *, int, struct file_lock *);
        ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int);
        ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int);
        int (*setlease)(struct file *, long, struct file_lock **);
        long (*fallocate)(struct file *file, int mode, loff_t offset,
                          loff_t len);
};

It appears that *ioctl has been replaced with *unlocked_ioctl and *compat_ioctl. Why? Well, according to this post, the old structure was a security risk, and was deprecated. With the new 3.0 kernel, the call must now be completely missing from the headers. The correct thing to do is to use the unlocked_ioctl element instead of ioctl.

WCF Ria Services result set size limitation

2011-08-09T14:33:00.001-06:00

While creating what should be a simple silverlight WCF RIA services application, the client returned the following error for some datasets:


System.ServiceModel.DomainServices.Client.DomainOperationException: Load operation failed for query 'GetBins'. Value cannot be null.
Parameter name: validationErrors ---> System.ArgumentNullException: Value cannot be null.

By using firebug and limiting the size of the return dataset, it became evident that after a certain size, the dataset wasn't returned at all with the http package.
It turns out that this is a WCF RIA services configuration issue. The default value for MaxItemsInObjectGraph was something way below the total number of records returned.
The solution was to do one of two things: Set MaxItemsInObjectGraph to be some larger number either in the web.config, or in the domain service class definition itself as follows:


[ServiceBehavior(MaxItemsInObjectGraph = 2147483647)] 
public class myDomainService : DomainService { ... }

Add that little gem to your domain services, and your problems melt away...

Silverlight Context Menu

2011-05-27T10:34:00.001-06:00

Silverlight 4. I can't believe how many times I've had to search for this one and remembered what it was. So, it gets added to my permanent list of memories.
The only real trick to get the right namespace is to drag a MenuItem onto the page, then delete it and add this code inside whatever you want to have a context menu.

<toolkit:ContextMenuService.ContextMenu>
  <toolkit:ContextMenu>
    <toolkit:MenuItem Header="Add child"/>
    <toolkit:MenuItem Header="Remove" />
  <toolkit:ContextMenu>

<toolkit:ContextMenuService.ContextMenu>

Sorting the results of a MS SQL table-valued function

2011-05-10T14:20:00.000-06:00

For whatever reason, MS SQL table-valued functions don't get the respect they deserve. Linq to entities pretends that they don't exist at all. I mean, seriously, how hard would it be to allow you to import them in the same way that you can import stored procedures?
Anyway, as a mechanism for abstracting your database schema, I like them. They are pre-compiled, strongly typed, and results cached, so they can get very fast. One thing they don't do well is return sorted values.
The following code won't work well when contained in a table-valued function:
SELECT * FROM Table1 ORDER BY Column1
This requires us to resort to doing something like this:
SELECT TOP 100 PERCENT * FROM Table1 ORDER BY Column1
And even that result can be ignored sometimes. Consider a table that consists of a key, HierarchyId, and a description. We want to unwind the hierarchyId so that we have key, parent key, and description. This is necessary if we are working with the hierarchy outside of SQL because .net doesn't have any idea how to deal with hierarchyId, Geometry, Geography, and other newer (2008+) datatypes. It would sure be nice if Linq to Entities was updated to at least allow us to pass these items around as binary fields, if not the real .net types they are! But, I digress...
The following function won't return the results in the order we'd like it to:

SELECT TOP 100 PERCENT
t1.key, t2.key parentKey, t1.description 
from Table1 t1 join t2 on t1.Hier.GetAncestor(1) = h2.Hier 
join Table1 t3 on h1.Hier.IsDescendantOf(t3.Hier) = 1 
and h3.key = @topKey
ORDER BY t1.Hier

One of the nice things about the HierarchyId data type is that when you sort by it, the parents and their children stay sorted in the right order; but not in this case.
The solution is to use an alternate method of sorting the results:

SELECT
ROW_NUMBER() OVER (ORDER BY h1.Hier) RowNumber,
t1.key, t2.key parentKey, t1.description 
from Table1 t1 join t2 on t1.Hier.GetAncestor(1) = h2.Hier 
join Table1 t3 on h1.Hier.IsDescendantOf(t3.Hier) = 1 
and h3.key = @topKey

This sort is accepted by the function syntax, and will give us exactly the order we expect. And as a bonus, we can restore that order if we choose to sort the results by some other column. Otherwise, just ignore the RowNumber column that is returned.

An infinite date table

2010-09-23T12:26:00.004-06:00

Here is the creation script for a neat function that will generate a table full of every date in the given range.


-- =============================================
-- Author:  Aaron D. Wells
-- Create date: 9/23/2010
-- Description: Get a list of dates in a range
-- =============================================
CREATE FUNCTION fn_Dates (@fromDate date, @toDate date)
RETURNS @dateTable TABLE( [Date] Date NOT NULL, FiscalYear int NOT NULL, FiscalMonth int NOT NULL )
AS 
BEGIN
     WITH CTE_DatesTable([date])
     AS (
     SELECT @fromDate AS [date]
          UNION ALL
          SELECT DATEADD(dd, 1, [date]) FROM CTE_DatesTable
          WHERE DATEADD(dd, 1, [date]) <= @toDate)  
          INSERT @dateTable
               SELECT [date],
                    CASE WHEN MONTH(date) > 9 THEN YEAR(date) + 1 ELSE YEAR(date) END FiscalYear,
                    CASE WHEN MONTH(date) > 9 THEN MONTH(date) - 9 ELSE MONTH(date) + 3 END FiscalMonth
               FROM CTE_DatesTable OPTION (MAXRECURSION 0)
     RETURN 
END

Using the function is pretty simple, just give it a starting and ending date. By the way, the FiscalYear column is used to calculate the fiscal year for the company I was working with when i wrote this. Your mileage may vary.

Generic Dynamic Class Factory (.net)

2010-07-15T15:05:00.000-06:00

This code provides a simple way to dynamically load all the assemblies in a directory (by default the bin directory in your project)

using System.Text;
using System.Reflection;
using System.IO;
public class DynamicFactory {
   List allAssemblies = new List();
   public DynamicFactory() { }
   public void LoadAssemblies() {
       Assembly thisAssembly = Assembly.GetExecutingAssembly();
       Uri thisUri = new Uri(thisAssembly.CodeBase);
       string path = Path.GetDirectoryName(thisUri.LocalPath);
       LoadAssemblies(path);
   }
   public void LoadAssemblies(string path) {
       allAssemblies.Clear();
       foreach (string dll in Directory.GetFiles(path, "*.dll")) {
           allAssemblies.Add(Assembly.LoadFile(dll));
       }
   }
   public IEnumerable EnumerateTypes() {
       foreach (var assembly in allAssemblies) 
           foreach (Type type in assembly.GetExportedTypes())
               if (typeof(T).IsAssignableFrom(type))
                   yield return type;                               
   }
   public IEnumerable EnumerateObjects() {
       foreach (var assembly in allAssemblies)
           foreach (Type type in assembly.GetExportedTypes())
               if (typeof(T).IsAssignableFrom(type))
                   yield return (T)Activator.CreateInstance(type);
   }
}

Software Developers Do Not Build Widgets...

2010-06-25T15:39:00.001-06:00

This post will be quite different from my normal posts. It is not a quick how-to, or a design pattern, but a rebuttal...

I was recently reading the introduction to a book on enterprise architecture. And the author tried to make the point that software development is a cottage industry because the "widgets" we create (software) does not have the associated factories, structures, and automation that making cars or cookies has. He points out that the auto industry spends 80% of its time designing the factory and only 20% designing the cars. At the end of building a particular type of automobile, the factory line is thrown away (or massively retooled) for the next model of car.

That author, and almost every other like-minded author or speaker I've ever heard of is building a case for their "software creation" automation tools, process, or methodology (shudder). And their analogy is completely wrong.

The truth is, we don't build cars or cookies. Every custom developed piece of software is different, every car or cookie is the same. We are not the mindless robots in the production line.

We are the engineers designing both the car and the production line.

In the world of software development, the factory is the application or system. The car (product) is the data that is being processed. Whether for internal or external consumption.

The purpose of of the factory is to take in supplies (raw materials, parts, widgets) and to produce product (more parts, or processed materials, and widgets), our software takes inputs (user inputs, inputs from other systems) and produces product (more data, or outputs to other systems).

The efficiency of the factory is completely dependent upon the architect, the effectivity of the system is dependent upon the software architect and developers.

Our tools resemble those of the design engineer or the automation process engineer. We model flows and structures. We specify tolerances and dimensions and interfaces. We are concerned about parallel and serial processes. We are the engineers that we have euphamistically claimed to be.

So, no, dear author, we don't design and build widgets, we design and build factories. Our factories interact with and are operated by users. Do we need process? Yes. Do we need procedures? Yes. Do we need tools? Yes.

Compare the number of moving parts in enterprise software to the moving parts in the factory that builds cars if you must, but do not compare websites and say that they are as similar as two cars rolling off of an assembly plant

If anything at all, compare building entertainment software development and production to the production of movies, and the development of e-commerce software to the development of a new commercial franchise.

Is it any wonder that in an information age, we would need to build information factories?

SQL random number column

2010-02-15T13:23:00.001-07:00

Here's a quickie. If you need to have a new random number on a column, this works really well.

rand(cast(cast(NEWID() as varbinary) as int)) number

Of course NEWID generates a random guid. Casting it as a varbinary first, then as an int gives you a random number which can then be used as a seed for the random number generator.

Dynamically Generated List of Dates Using Common Table Expressions (CTE)

2010-02-15T09:50:00.006-07:00

Here is a nice use for Common Table Expressions (CTEs) to generate a dates table or other list of stuff. I won't take credit for this technique since I saw it someplace else (but can't remember where). Of course my blog is for stuff I want to remember, so thanks to whoever you are...

declare @loops int = 1000
declare @StartDate date = '1/1/2000';

WITH CountTable( RowNumber, [Month], [Year], FirstDay ) AS
(
 SELECT
  1 RowNumber,
  MONTH(@StartDate) [Month],
  YEAR(@StartDate) [Year],
  -- compute the first day of the month from whatever day was provided
  DATEADD(dd,-(DAY(@StartDate)-1),@StartDate) FirstDay   
 UNION ALL
 
 SELECT
  RowNumber + 1 RowNumber,
  MONTH(DATEADD(MONTH, 1, FirstDay)),
  YEAR(DATEADD(MONTH, 1, FirstDay)),
  DATEADD(MONTH, 1, FirstDay) FirstDay
  
 FROM CountTable t
 WHERE RowNumber < @loops
)

SELECT * 
FROM CountTable
OPTION (maxrecursion 32767)

The results look like this:

 1  1 2000 2000-01-01
 2  2 2000 2000-02-01
 3  3 2000 2000-03-01
 4  4 2000 2000-04-01
 5  5 2000 2000-05-01
 6  6 2000 2000-06-01
 7  7 2000 2000-07-01
 8  8 2000 2000-08-01
 9  9 2000 2000-09-01
10 10 2000 2000-10-01
11 11 2000 2000-11-01
12 12 2000 2000-12-01
13  1 2001 2001-01-01
14  2 2001 2001-02-01

more rows here...

 999 3 2083 2083-03-01
1000 4 2083 2083-04-01

In this query, there are two parts to the CTE, the seed (top part) and the function (bottom part). The seed sets the initial values, and the function uses the seed values to create the next row of values, which are then used to create the following row of values (and so on, and so on...)

There are a couple of things to notice:

The function values must be based on the previous value (notice my recalculation of the next month)
You must use a semicolon before the CTE
the maxrecursion option defaults at 100 iterations and has a maximum value of 32767

Recursively change unix permissions on a directory

2010-02-06T15:59:00.000-07:00

On my freenas server I sometimes need to change the permissions of files that were uploaded via FTP. Here is the command that can be issued to make those changes:

chmod -R 0777 "/mnt/Seagate650/FTP Root"

The quotes are important because my ftp root directory contains a space.

Load a dll into an AppDomain for reflection or execution and Unload it

2010-01-25T15:37:00.009-07:00

.Net will allow you to easily load any dll you want to into the current AppDomain by simply calling Assembly.LoadFrom() with the path of the dll in question. However, this is only available for the current AppDomain, and as a result, the dll remains loaded for the duration of the AppDomain's lifetime.

This may be ok, but if you want to simply load a dll, inspect it, and run some one-time operations on it, that seems a bit wasteful. In web development, it means you must shut down your web server to recompile the dll you're loading.

So, the solution would seem to be to start up a new appDomain, load the dll in question, do your work, and then unload the appDomain.

Gentlemen, start your compilers...

namespace Utilities {
  public class SomeClass{
    public static void RegisterApplication(
      string appPath) {
      if (File.Exists(appPath)) {
        var domainSetup = new AppDomainSetup() {
        ApplicationBase = AppDomain.CurrentDomain
          .BaseDirectory,
        PrivateBinPath = AppDomain.CurrentDomain
          .BaseDirectory + @"\bin"
      };
      var appDomain = AppDomain.CreateDomain(
        "LoaderDomain", null, domainSetup);
      try {
        var loader = (Loader)appDomain
          .CreateInstanceAndUnwrap(
            Assembly.GetExecutingAssembly()
              .FullName,
            typeof(Loader).FullName);
        loader.DoSomething(appPath);
      }
      finally { AppDomain.Unload(appDomain); }
    }
  }

  public class Loader: MarshalByRefObject {
    public void DoSomething(string appPath) {
      var assembly = Assembly.LoadFrom(appPath);
      foreach (var type in assembly.GetTypes()){
      // Do something with the types we find
      // like create some classes with known 
      // interfaces or examine attributes
      }
    }
  }
}

What's going on?

Well, we create an app domain and seed the PrivateBinPath with a \bin subdirectory so that we can load our current dll in an IIS or cassini environment. Then we fire off the known object and have it call into Assembly.LoadFrom(). Assembly.LoadFrom will resolve dependencies using the standard resolution stuff for our other assemblies. Alternately, we could use a custom resolver event handler to get to our dlls.

CSRF (Cross-Site Request Forgery) Handling in MVC

2010-01-18T08:53:00.005-07:00

Phil Haack's talk on asp.net reminded me of the importance of handling CSRF attacks. In MVC, this is simply handled with a one-two punch. In your posting form, you need to include the Html.AntiForgeryToken() in the form as follows:

The other thing is to add an [ValidateAntiForgeryToken] attribute to the targeted posting action in your controller as follows:

[ValidateAntiForgeryToken]
[HttpPost] 
// or MVC 1.0 style [AcceptVerbs(HttpVerbs.Post)]
public virtual ActionResult Delete(FormCollection form, int id) {
  // code here to delete stuff
  return View();
}

Also don't forget that we want to make sure that any state-changing operations are always posts!
My new syntax highlighting is accomplished via a method outlined on Carter Cole's most excellent blog.

Linq to Entities and Custom Database Functions

2009-12-31T11:56:00.024-07:00

I've been using Linq to SQL since it came out. Linq to Entities didn't have any compelling functionality, and it was far less supported in the development community. Nobody seemed to be using it. For whatever reason, when the language group at Microsoft came out with Linq, the data group was slow on the uptake and that left the language group with the need for an actual database query layer, so Linq to SQL was born. There were rumors that Linq to SQL would not continue to evolve, but it certainly worked fine for what I needed.

The 2009 Microsoft PDC had all kinds of interesting things, but absolutely nothing regarding Linq to SQL. The language group became enammored with .NET 4 and the dynamic language features and have apparently orphaned Linq to SQL. Not so the data group. The improvements to Linq to Entities seem to breathe new life into what had previously been a "me too" implementation of Linq.

Certainly the biggest news about Linq to Entities relates to the new flexibility to use POCO objects, database first, or model first for defining your objects. I especially like the ability to use t4 templates for model generation. This development certainly takes away of the allure of NHibernate as an ORM. In fact, I think you can probably start calling Linq to Entities a nearly full-featured ORM in it's new incarnation. One potentially serious oversight is the lack of ability to perform dependancy injection on object creation, but let's hope that is rectified as soon as possible.

However, the one piece that really gets me interested is the ability to call custom database functions from within your Linq statements. This is potentially huge. It is really too bad that Shyam and Chris weren't able to get the demo of this little gem to work in their presentation. There is also not a lot of documentation available for this feature. In fact, I only found one useful article in the blogosphere.

The key thing that is unclear from the documentation and article is the thing that Shyam and Chris had problems with as well. So, here is how you fix the issue:

The namespace attribute in the schema entity in the Linq to Entities edmx file must match the namespace entry of the EdmFunction code attribute in your static method. This namespace is the value in the Model Browser namespace as shown here.

<Schema Namespace="Data.Store" Alias="Self" Provider="System.Data.SqlClient" ProviderManifestToken="2008" xmlns:store="http://schemas.microsoft.com/ado/2007/12/edm/EntityStoreSchemaGenerator" xmlns="http://schemas.microsoft.com/ado/2009/02/edm/ssdl">
  <Function Name="IsAuthorized" ReturnType="bit" Aggregate="false" BuiltIn="false" NiladicFunction="false" IsComposable="true" ParameterTypeSemantics="AllowImplicitConversion" Schema="dbo">
    <Parameter Name="Id" Type="int" Mode="In" />
  </Function>
</Schema>

public static class LinqExtensions {
  [EdmFunction("Data.Store", "IsAuthorized")]
  public static bool IsAuthorized(int? id) {
    throw new Exception("Direct calls not supported");
  }
}

This allows you to write the following Linq statement:

var result = from s in dc.SomeTable
  where LinqExtensions.IsAuthorized(s.Id)
  select s;

Another important thing is that you don't have to manually edit the .edmx file. You can use the Model Browser or the design surface to select "update model from database" from the context menu. From there you can add an existing function to the model, and not have to deal with the XML in the .edmx file

jQuery and MVC Autosuggest while saving the selected Id

2009-04-29T20:31:00.026-06:00

Yea, I know, boring title. But I really want to make sure I can find the code for how to do this one.

jQuery is very cool. All the eye candy is nice, the syntax is excellent, but the real benefit is AJAX...

The autocomplete jQuery library is extremely nice. It allows you to take a Json result from some url and suggest results to a text box. What I want is for the selected value to populate another input field (hidden) so that I have a real foreign key relationship to the item they selected.

here's the html:

<input class="location" id="LocationName">
<input id="LocationId" type="hidden" keyFor="LocationName"/>

Or alternately you can use Html Helpers

<%= Html.TextBox("LocationName", null, new {class = "location"}) %>
<%= Html.Hidden("LocationId", null, new {keyFor = "LocationName"})%>

and here is the supporting javascript:

$(document).ready(function() {
$('.location).autocomplete("/location/JsonFind",
{ dataType: 'json',
parse: parseData,
formatItem: formatItem,
formatResult: formatItem
}).result(autocompleteResult);
});

function parseData(data) {
var rows = new Array();
for (var i = 0; i < data.length; i++) {
rows[i] = { data: data[i], result: data[i].Description, value: data[i].Id };
}
return rows;
}

function formatItem(row, i, n) { return row.Description; }

function autocompleteResult(event, data, formatted) {
var TargetId = "[keyFor=" + event.currentTarget.id + "]";
$(TargetId).val(data.Id);
}

and finally, the code in the Location controller

public ActionResult Find(string q, int limit) {
return Json(Repository.GetAll().Where(l => l.LocationName.StartsWith(q)).Take(limit));
}

What's going on here?

Well, we simply have an input control and a hidden intput control on our html page. The important convention I have is that the hidden input control has a non-standard "locationidfor" tag so that we can identify where to put the selected value.

The "location" controller is providing a method that returns complete Location records from our Linq to SQL table through a repository. The exercise of setting up Linq to SQL is left to the reader. The linq code at the end simply grab all the Locations where the location names start with the text the user typed in, and we only take "limit" elements from the list. Your method will probably be different.

jQuery is the glue that binds the whole thing together. Our applyLocationAutosuggest function may be applied in the page ready just as easily, but here I've encapsulated the functionality so it can be applied to jQuery templates as well!

In its simplest form Autocomplete just needs a list of strings (simple matching), but here we use the formatItem and formatResult parameters to make sure that the user doesn't see the entire Location row, just the text we want. Mix in a check to make sure that there was at least one result. There are also two undocumented parameters we need to use, the dataType parameter is set to 'json' because autocomplete expects plain text by default; parse gives us the opportunity to take our json data and change it into the autocomplete internal array format.

Finally, when the result function allows us to take the selected Json result, decode it into an object, and put the id in the hidden field.

Mission Accomplished.

It's all about presentation...

2009-01-21T13:28:00.005-07:00

My last post, you may notice has code snippets that look somewhat professional. I found a cool online tool called Code2html

It turns this:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net.Mail;
public partial class _Default : System.Web.UI.Page 
{
    protected void Page_Load(object sender, EventArgs e)
    {

    }
    protected void Button1_Click(object sender, EventArgs e)
    {
        MailMessage message = new MailMessage("FROM@domain.foo", "TO@domain.foo", "SYNCHRONOUS MESSAGE", "MESSAGE BODY");
        SmtpClient smtp = new SmtpClient();
        smtp.Send(message);
    }
    protected void Button2_Click(object sender, EventArgs e)
    {
        MailMessage message = new MailMessage("FROM@domain.foo", "TO@domain.foo", "ASYNCHRONOUS MESSAGE", "MESSAGE BODY");
        SmtpClient smtp = new SmtpClient();
        smtp.SendAsync(message, null);
    }
    protected void Button3_Click(object sender, EventArgs e)
    {
        System.Web.Mail.SmtpMail.Send("FROM@domain.foo", "TO@domain.foo", "OLD MESSAGE", "MESSAGE BODY");
    }
}

to this:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net.Mail;
public partial class _Default : System.Web.UI.Page 
{
    protected void Page_Load(object sender, EventArgs e)
    {
    }
    protected void Button1_Click(object sender, EventArgs e)
    {
        MailMessage message = new MailMessage("FROM@domain.foo", "TO@domain.foo", "SYNCHRONOUS MESSAGE", "MESSAGE BODY");
        SmtpClient smtp = new SmtpClient();
        smtp.Send(message);
    }
    protected void Button2_Click(object sender, EventArgs e)
    {
        MailMessage message = new MailMessage("FROM@domain.foo", "TO@domain.foo", "ASYNCHRONOUS MESSAGE", "MESSAGE BODY");
        SmtpClient smtp = new SmtpClient();
        smtp.SendAsync(message, null);
    }
    protected void Button3_Click(object sender, EventArgs e)
    {
        System.Web.Mail.SmtpMail.Send("FROM@domain.foo", "TO@domain.foo", "OLD MESSAGE", "MESSAGE BODY");
    }
}

You had me at ehlo!

2009-01-21T12:46:00.015-07:00

It seems that a lot of websites send users email for lots of reasons. This introduces a bit of a problem when you are developing applications because as developers we don't necessarily want to put an smtp server on our development machine and network administrators are fairly selective about the open SMTP relays that they want on their network.

ASP.net has a solution, and I have a solution.

The asp.net solution is that the web server where your application will eventually live can (will) have a single machine configuration for the smtp settings. Every application can override this setting in the web.config file, but should actually just comment out the <mailsettings> section.

During development, however, this configuration is useful:


<configuration>
 <system.net>
  <mailSettings>
   <smtp from="admin@foo.bar" deliveryMethod="SpecifiedPickupDirectory">
    <specifiedPickupDirectory pickupDirectoryLocation="C:\pickup" />
   </smtp>
  </mailSettings>
 </system.net>
</configuration>

This will just dump the emails in the C:\pickup directory of your development machine. No need to install the SMTP service on your computer at all! This only works with the email classes in System.Net.Mail.

If you use System.Web.Mail.SmtpMail you are out of luck because it defaults to using a network delivery and localhost as the server. That is where my other solution comes in...

I wrote a quickie program a few years ago that I call "Black Hole" because it sucks up everything you send at it and doesn't let anything out. It is a stand alone SMTP server that just can either silently eat messages, or it can log the communication with your smtp client, and optionally dump the email messages do a directory.

You can download it here.

The zip file contains a stand-alone executable (written in Delphi) that will listen on whatever port you tell it and whatever ip addresses you want. Then, just configure your web.config (or app.config for that matter) settings to point to localhost with no credentials needed.

ATLAS AJAX Control Toolkit AutoCompleteExtender with Key Lookup

2008-12-09T13:46:00.006-07:00

I like the ATLAS AJAX Control Toolkit. It integrates with Visual Studio and has useful if not basic javascript functions that can help a website be much more interactive.

The AutoComplete Extender is used to provide auto-complete functionality to a text box. To use it, you simply add an extender and a web service method that provides a list of strings based on the text that the user started typing. So, if you are trying to provide the exact string that the user is typing (ala google search help), you can use it out of the box.

However, sometimes you want more list-ish functions, that is, you want to present the user with a list of valid choices that represent objects, not just the descriptions. One alternative is to use a Listbox with a ListSearch Extender. This extender is used to find an existing value within the listBox. The downside to this approach is that every item you are looking for has to be in the list. This can be rather unwieldy if the list has more than a couple dozen items.

The other alternative is to use the AutoComplete extender, but have it return a text string and a key that may be used to load the found item. This function is available, but not well documented (or at least where I can find it easily), so, in conformity with the purpose of this blog, I'm putting sample code here.

First of all, we need a hidden field and a text box (with the extender). The bolded properties in the extender point to the web method and the javascript methods that we'll need later.

<asp:hiddenfield id="AccountNumber" runat="server">
<asp:TextBox ID="AccountNumberTextBox" runat="server" Width="20em"/>
<cc1:AutoCompleteExtender ID="AccountNumberTextBox_AutoCompleteExtender"
runat="server" CompletionInterval="200" DelimiterCharacters="" Enabled="True"
FirstRowSelected="True" MinimumPrefixLength="1"
ServiceMethod="GetCompletionList" ServicePath=""
TargetControlID="AccountNumberTextBox" UseContextKey="True"
OnClientItemSelected="AutoCompleteSelected" />

The next thing that has to be done is to return a specially encoded list of label/values in the string array from your web service:

[System.Web.Services.WebMethodAttribute(), System.Web.Script.Services.ScriptMethodAttribute()]
public static string[] GetCompletionList(string prefixText, int count, string contextKey)
{
myDataContext dc = new DataContext();
var r = (from le in dc.Entities
where (le.AccountNumber.Contains(prefixText) le.Name.Contains(prefixText))
select le).Take(count);
List results = new List();
foreach (var item in r)
{
results.Add(AjaxControlToolkit.AutoCompleteExtender.CreateAutoCompleteItem(
string.Format("{0} ({1})", item.Name, item.AccountNumber), item.AccountNumber));
}
return results.ToArray();
}

The AjaxControlToolkit.AutoCompleteExtender.CreateAutoCompleteItem takes a description and a key value and munges them into a string that can easily parse out the account number.

With that done, we need to add a javascript block to our ASPX that will populate the hidden field with the key value when it is selected from the list.

This script (and constant) do a couple of things. First, we capture the Id of the hidden control so we can use it later to know where to put the id. Next, we disassemble the special text by using the eventArgs.get_Text() and eventArgs.get_Value() functions. These functions are included by the same library that defined our extender.

Add this script to the header section of your page

<script type="text/javascript">
var acctNo = '<%=AccountNumber.ClientId%>';
function AutoCompleteSelected(source,eventArgs)
{
document.getElementById(acctNo).value = eventArgs.get_value();
}
</script>

IBM x335 and Windows 2008 installation

2008-08-11T09:27:00.000-06:00

I love the fact that IBM stops supporting its hardware long before Microsoft does; this means that lots of top-tier companies have to decommission their *somewhat* old hardware and you can pick up a server cheap (or free) to play with at home.

It may mean, however, that your installation mileage may vary. Such has been the case for my IBM x335 (of which I have three). I've been on a quest to do an installation of Windows Server 2008 on said boxen. (plural of ox is oxen, plural of box is...)

How fickle a lover are antique hardware and modern operating systems? How doest thou vex me, let me count the ways...

Windows 2008 only comes on DVD while an IBM x335 has only a CD and will not boot from USB.
Windows 2008 will not install from any sort of DOS utility disk. So Bart's Network boot disk is out, likewise a network boot to anything based on DOS.
The IBM x335 has an internal IDE that is used by its CD Rom. You can crack the case and use an internal DVD drive from some other computer to install the os. This feels like cheating on an exam.
VMWare ESXi runs well on an IBM x335, and will allow installation of Windows 2008 from a DVD image (.iso file). This is also cheating, and ultimately not as satisfying as beating the beast at it's own game.

So, you aren't reading this to find out the problems that you probably already know, right? So, let's give some credit here to GeekEleet and two articles that helped me to solve this issue:

I also want to thank redmutt.com for their article on Installing Windows 2008 on an IBM xSeries 335 that included the link to the appropriate raid driver.

The key points of the articles are that:

Windows 2008 only installs from a WinPE 2.0 disk
You can make a WinPE 2.0 disk by using the Windows Automated Installation Kit
WinPE 2.0 allows you to map a network drive (containing your Windows 2008 install files) using net use...
You might also be able to make CD installation media using the WAIK and do away with the network install all together.
You might need to update your bios using the IBM UpdateXpress CD 4.05a for System x
You also need the LSI Logic PCI SCSI/FC MPI Minoport driver v1.25.10 for Microsoft Windows Server 2008 that you can load from a floppy, usb drive, or network drive during installation

One further observation is that you can do an in-place upgrade or install of Windows 2003 server if you specify the driver during installation.

New cool tool

2008-03-10T16:12:00.001-06:00

Oh, by the way, I have to give some credit to Notepad++ for that last post. The asp.net was HTML encoded using Notepad++ (TextFX TextFX Convert Encode HTML). I keep Notepad++ on a USB drive using PortableApps. It has lots of cool tools that are handy to keep with you or keep personal like KeyPass portable, but that's another post!

AJAX and IP Addresses

2008-03-10T15:51:00.001-06:00

I realize you are disappointed, but the only clever title I could think of for IP Address reminded me of a stupid joke from kindergarten... Never the less.

Here is a cool formula for a client-validated IP address using the AJAX Control Toolkit and asp.net.

<asp:TextBox ID="TextBox1" runat="server" Text='<%# DataValue %>'></asp:TextBox>
<cc1:MaskedEditExtender ID="MaskedEditExtender1" runat="server" TargetControlID="TextBox1"
Mask="NNNNNNNNNNNNNNN" MaskType="None" ClearMaskOnLostFocus="true" Filtered=". " PromptCharacter="" />
<cc1:MaskedEditValidator ID="MaskedEditValidator1" runat="server" ControlToValidate="TextBox1"
InvalidValueMessage="IP Address required (192.168.1.1)" ControlExtender="MaskedEditExtender1" ValidationExpression="^([01]?\d\d?2[0-4]\d25[0-5])\.([01]?\d\d?2[0-4]\d25[0-5])\.([01]?\d\d?2[0-4]\d25[0-5])\.([01]?\d\d?2[0-4]\d25[0-5])$" />

It's really not that involved.

The first control is a normal text box. I happen to be using the new dynamic data stuff from Microsoft. It's pretty cool, but you could bind or not as you want.

The MaskedEditExtender is important because it limits the characters somebody can type to numbers (the N characters in the mask) and the filtered characters (period and space). With 15 spaces maximum, this means the user can't type in too much stuff

Finally, the MaskedEditValidator control has a regular expression that accepts all the valid number mixes from 0-255 for each digit, and only four octets

Once you add it up, you get a client-side filtered IP address with Javascript validation.

Update:

The regular expression could also be written as:
^([01]?\d\d?2[0-4]\d25[0-5])(\.([01]?\d\d?2[0-4]\d25[0-5])){3}$
It works something like this:

The carat (^) and dollar sign ($) mean that the expression matches stuff that start and end the input.
The first subsection (first set of paranthesis) has three rules: the first "[01]?\d\d?" will match the numbers from 0 to 199 with or without leading zeroes (000 or 0), the second "2[0-4]d" matches numbers from 200 through 249, and the third "25[0-5]" matches values from 250 to 255
the next subsection prepends the previous rule with a period (or dot) exactly three more times.

The end result is that you have exactly four numbers from 0-255 separated by three dots.

Expand your mind

2007-12-17T10:55:00.000-07:00

The only hassle of working with Virtual Machines is being omniscient: how in the world am I supposed to know how big to make the virtual drive when I don't know what stuff I'm going to install on the machine?

Solution: VMWare Converter

This little beauty non only lets you convert physical machines to VMs, or Microsoft VMs to VMWare VMs, it also lets you convert VMWare VM's to another version and resize the disk along the way!

sweet!

If I could get connected, I could tell who you are

2007-12-05T15:29:00.000-07:00

Playing with Visual Studio 2008, .net 3.5 and SQL Linq today and decided to make the thing a real application. That means using a membership database. I really like using sql express for development (and deployment) of little applications. It's a shame that my hosting provider doesn't let you use an express database, but I digress...

Anyway, how do you go about creating membership in an existing database...

You twiddle with the command line parameters until you figure out that this is how:

aspnet_regsql -A all -C "Data Source=.\SQLEXPRESS;Integrated
Security=True;User Instance=True" -d "C:\MyProject\APP_DATA\aspnetdb.mdf"

You've been very helpful, can you just go away now.

2007-06-06T15:59:00.000-06:00

Sometimes when dealing with a typed dataset (a good idea) you have a problem with constraints, or with nullable fields, or with other stuff. Microsoft gives you a less-than-helpful exception that says "Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints."
Right...
Well, if you use:

ds.EnforceConstraints = false;

then you can call this method which will output the table, row, column, and exact constraint that was violated.

public void GetDataSetErrors(DataSet ds){
try { ds.EnforceConstraints = true; }
catch(Exception ex) {
foreach
(DataTable table in ds.Tables) {
foreach (DataRow row in table.GetErrors()){ System.Diagnostics.Trace.WriteLine("Table: "+
table.TableName); System.Diagnostics.Trace.WriteLine(ex.Message
); System.Diagnostics.Trace.WriteLine("Row: "
+row.RowError); foreach (DataColumn col in
row.GetColumnsInError())
{ System.Diagnostics.Trace.WriteLine("Column
" +
col.ColumnName); System.Diagnostics.Trace.WriteLine(
row.GetColumnError(col)); } } } } }

It just makes you feel all warm and fuzzy inside, doesn't it?

Does this dress make me look fat?

2006-11-10T14:41:00.002-07:00

Here's a nice little bit of code that can be used to dynamically theme a website that has multiple domains at the same site.

using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.Web.UI;

namespace URLThemeHandler {
public class ThemeHandler : IHttpModule {
#region IHttpModule Members

public void Dispose() {
}

public void Init(HttpApplication context) {
context.PreRequestHandlerExecute += new EventHandler(context_PreRequestHandlerExecute);
}

void context_PreRequestHandlerExecute(object sender, EventArgs e) {
HttpContext currentContext = HttpContext.Current;
if (!(currentContext.Handler is System.Web.UI.Page)) return;
System.Web.UI.Page myPage = currentContext.Handler as System.Web.UI.Page;
string strHost = currentContext.Request.Url.Host;
if(ThemeExists(strHost))
myPage.Theme = strHost;
}

bool ThemeExists(string theme) {
string strPath = HttpContext.Current.Server.MapPath("~/App_Themes/" + theme);
return System.IO.Directory.Exists(strPath);
}

#endregion
}
}

You can register it in the httpModules section of the Web.config.

Can you hear me NOW?!?!

2006-07-17T18:50:00.000-06:00

My wife is very disgusted with our packet8 telephone. Part of the issues we're facing are that our Belkin router will block the outbound voice (thinking that it is a DOS attack). In the security log, there was this entry:

07/17/2006 18:35:57 **UDP Flood to Host** 192.168.1.10, 8006->>
209.247.23.73, 62602 (from WAN Outbound)

So, thanks to a technical article I found, there is a support page that references this URL: http://192.168.1.1/firewall_spi_h.stm which allows you to turn off the "SPI and Anti-DoS firewall protection:"

It should be easier to turn off a feature like this than an undocumented feature.

It's about time...

2006-07-06T10:41:00.007-06:00

Time and space. Two of my favorite things. Time keeps everything from happening all at once. Space keeps everything from happening to you. Time AND Space keep everything from happening all at once to you!
The problem to be solved today is this: Create a .net object that has it's XML serialization saved to an XML column in a database table. Now, have a trigger (add/update) that takes that XML field and denormalizes its content to some additional columns. In this way, you have the denormalized data that you might want to use for joins and filtering without having to parse the XML field over and over.
You'd create a class something like this:

public class Foo {
Guid _id = Guid.NewGuid(); //This is our object's database identifier
DateTime _updated= DateTime.Now; //This was the last time the object was changed
string _description = string.empty;

public Guid Id{ get{return _id;} set{_id = value;} }
public string Description { 
get{return _description;} 
set{_description = value; Update();} }
public DateTime Updated { get{return _updated;} set{} }
private void Update(){ _updated = DateTime.Now; }
}

You remember how to serialize an object as XML, right? Ok, just for reference, here is how we would get our object as XML text:

XmlSerializer serializer = new XmlSerializer(typeof(Foo));
System.Text.StringBuilder sb = new System.Text.StringBuilder();
TextWriter writer = new StringWriter(sb);
serializer.Serialize(writer, this);
writer.Close();
string XMLText = sb.ToString();

You would also have a table that looks like this:

CREATE TABLE [dbo].[TableXML](
[Id] [uniqueidentifier] ROWGUIDCOL  NOT NULL CONSTRAINT 
[XML] [xml] NOT NULL,
[Description] [nvarchar](max) )

Then you'd add a trigger to the table that looks like this:

CREATE TRIGGER [TableXML_InsUpd] ON  [dbo].[TableXML] AFTER INSERT,UPDATE AS 
BEGIN
SET NOCOUNT ON;
IF UPDATE([XML]) -- Only update the table if the XML field was updated.
BEGIN
UPDATE [MWI].[dbo].[TableXML] SET 
[Description] = INSERTED.XML.value('(//Description)[1]','nVarchar(MAX)'),
[Updated] = INSERTED.XML.value('xs:dateTime((//Updated)[1])','DateTime')
FROM INSERTED WHERE [PromotionXML].PromotionId = INSERTED.PromotionId
END
END

With these things in place, you could serialize the class to XML with code that looks like this:

public static void SaveFooDB(Foo foo) {
// Put the XML representation of the object into the database
SqlConnection conn = GetDataConnection(); //you'll have to write GetDataConnection
SqlCommand cmd = new SqlCommand("foo_InsertOrUpdate", conn);
using (cmd) {
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("@fooXML", promotion.ToString());
cmd.ExecuteNonQuery();
}
}

The stored procedure "foo_InsertOrUpdate" simply checks if the id already exists in the database and updates the existing record if it does, otherwise it is a new record, so it is inserted. Why SQL server doesn't have this sort of query is a mystery to me. Notice, however that we only have to pass one parameter (the serialized XML from our object) and we extract the Id from that XML.

CREATE PROCEDURE [dbo].[foo_InsertOrUpdate]( @FooXML XML ) AS
BEGIN 
DECLARE @IdUniqueIdentifier
select @Id= @FooXml.value('(//Id)[1]','UniqueIdentifier')
IF Exists (Select FooId from [MWI].[dbo].[FooXML] WHERE Id= @Id) 
BEGIN
UPDATE [MWI].[dbo].[FooXML] SET [XML] = @FooXML WHERE Id= @Id
END
ELSE 
BEGIN 
INSERT INTO [MWI].[dbo].[PromotionXML]([PromotionID],[XML])
VALUES (@PromotionID, @PromotionXML)
END
END

Now you might argue that since we have our insert/update procedure, we could do all the work of populating the additional columns here rather than in the trigger. You are right of course IF all of your table updates come through this procedure. Me, I'd rather not assume that everyone will play nicely, especially since it is possible to update the xml directly and in bulk. So, the trigger works for my needs, your mileage may vary.
Now everything works very well until we introduce another DateTime property in our class. If we (for example) bind that value to a DateTimePicker component, the XML extraction formatting will break. What is up with that?
It turns out that the DateTime format in .NET is far more robust than in many other systems (notibly SQL Server's DateTime). In fact, there is a property called "kind" on .net DateTime objects. The DateTimePicker uses the DateTimeKind.Unspecified kind which doesn't have time zone information. We can remedy this little problem by creating a new DateTime object in the set accessor of our new FooDate Property like this:

DateTime _fooDate = DateTime.Today;

public DateTime FooDate{
get { return _fooDate; }
set { _fooDate = new DateTime(value.Year, value.Month, value.Day, 0, 0, 0, 0, DateTimeKind.Local);}
}

Another thing that we can do here is to strip out the time infomation that is often provided by a calendar picker or other unreliable DateTime mechanism. One of my favorite tricks here is to specify the end of the day when people are picking a date range, so that we don't end up with that annoying "pick the next day really mean the end of today. Do something like this:

DateTime _fooEndDate= DateTime.Today;

public DateTime FooEndDate{
get { return _fooEndDate; }
set { _fooEndDate= new DateTime(value.Year, value.Month, value.Day, 23, 59, 59, 0, DateTimeKind.Local);}
}

And finally (wow, this was a long post) you don't want to put in 999 for the milliseconds if you are going to put this value into SQL server, someplace along the way, the milliseconds are rounded up to tomorrow and our SQL column representation becomes different from the XML representation.

U-Turn (Going the other way)

2006-07-05T13:42:00.002-06:00

So, here's a way to get back a delimited list of items from a table. In other words, concatenate values into one column (that you can then take together to pass to my parse table function).

--Setup a sample data to demonstrate the concept
declare @Table1 as Table (id int, Item varchar(100))
insert into @Table1 values (1, 'a')
insert into @Table1 values (1, 'b')
insert into @Table1 values (2, 'c')
-- show the table data
select * from @Table1

--retrieve the data from @Table1 as a concatinated list
Select distinct id, (
Select upper(Item)+' ' as [text()] from @Table1 S 
where id = T1.id 
for xml path('')) Items  from @Table1 T1

A mighty conversion

2006-05-31T13:19:00.000-06:00

I recently ran into a situation where I needed to create an array of integers to a comma-delimited string in .Net. This *should* be easily accomplished by some sort of intrinsic function (like .toString()). But alas, no. So, you can do this (and other sorts of arrays in the following manner:

string strCSV = string.Join(",", Array.ConvertAll<int,string>(SkuList, new Converter<int,string>(Convert.ToString)));

Going the other way you can use something similar to turn the whole thing inside out:

int[] result = Array.ConvertAll<string,int>(strCSV.Split(new char[] { ',' }), new Converter<string,int>(Convert.ToInt32));

X-Men Code 127

2006-04-20T13:35:00.000-06:00

One of the more interesting things about SQL Server 2005 is the support for the XML datatype. Instead of playing around with wierd and funky stored procedures to manipulate XML, you get validation and a bunch of other features just by specifying a field or a parameter as XML.
Now, as nice as it is to be able to serialize a .net object and store the entire thing in one field in the database, I have my feelings that there is significant overhead to querying into that XML on a more than ad-hoc basis. So, you *might* want to add some additional fields to the table containing the xml. For example if you had the following XML:

<OrderID>0a4fc0f8-6106-467f-ae10-45dcbfe4cefa</OrderID>
<CustomerNumber>1234</CustomerNumber>
<Status>Active</Status><Lines>
<OrderLine ItemId="123" Quantity="2" Taxable="false" Price="12.34"
DateChanged="2006-04-20T12:16:20.8942144-06:00"
IsReadOnly="false"></OrderLine>
<OrderLine ItemId="234" Quantity="2" Taxable="false" Price="56.78"
DateChanged="2006-04-20T12:16:20.8942144-06:00"
IsReadOnly="false"></OrderLine>
</Lines>
<DateChanged>2006-04-20T12:16:20.8942144-06:00</DateChanged></Order>

You *might* want to have the top-level elements like "DateChanged" "CustomerNumber" and "OrderID" in redundant fields. Your data table might look like this:

OrderID UniqueIdentifier
DateChanged DateTime
CustomerNumber Int
OrderXML xml

If you did this, you could create an Insert/Update trigger on the table that would automatically update the (non-key) fields (Thanks Scott!) that used the built-in XML features to update the duplicate fields whenever the xml (or any other field) was updated.

ALTER TRIGGER [OrderXML_InsUpd]
ON [dbo].[OrderXML]
AFTER INSERT,UPDATE AS BEGIN
SET NOCOUNT ON;
UPDATE [MWI].[dbo].[OrderXML]
SET
[CustomerNumber] = INSERTED.XML.value('(//CustomerNumber)[1]','Int'),
[DateChanged] = INSERTED.XML.value('xs:dateTime((//DateChanged)1])','DateTime'),
[Status] = INSERTED.XML.value('(//Status)[1]','nVarchar(50)')
FROM INSERTED
WHERE [OrderXML].OrderId = INSERTED.OrderId
END

Anyway, the whole thing is interesting, but pay close attention to the line that moves the date field into the DateTime column of the database table.

[DateChanged] = INSERTED.XML.value('xs:dateTime((//DateChanged)1])','DateTime')

The key to importing a .NET DateTime field into SQL server is to make sure to use the XML to cast the field as dateTime before SQL server gets it. Of course, this may not be necessary if you have a schema associated with the field that is already strongly typed in the XML.

What!?? No clever title?!??

2006-04-06T13:24:00.000-06:00

No clever title for this one. Just a reminder that you can turn on tracing in an asp.net web page by including the following in your page directive:

<%@ page trace="true" %>

And, lest you tire of remembering which specific pages you have this enabled on, you can turn on tracing for an entire application by including the following in the web.config:

<trace enabled="true" localonly="true" tracemode="SortByTime" pageoutput="true" requestlimit="10"><trace enabled="true">

Apple Core...Baltimore...Who's your friend?

2006-04-05T10:49:00.000-06:00

I've always had a soft spot in my heart for Apple. While it wasn't my "first" computer, (that was a Commodore Pet), The Apple II was the computer where I learned about data structures, graphics, and really did some useful programming. In college when I needed to get a computer, I bought a Macintosh IIsi. While my computer programming career has been mostly Windows-centric, I still like the elegance and simplicity of products from Apple; especially when they are free.

After playing with Music Match and other music programs, I've pretty much decided to use iTunes in my home as a music library management and MP3 player. I like it's playlist management features, the fact that it is easy to rip and burn physical CDs and the user interface is simple and elegant. The visualizations are cool too. I don't buy much music through the iTunes store (only one to date), but I do buy music on CD and want to be able to listen to it without physically handling the CDs; or more precisely, I don't want my children handling my CDs.

Enter the problem. ITunes is great as a personal mp3 player for a single person. It can automatically organize your music and it sounds good. If I had an iPod, I'm sure that it would be really nice to be able to listen to the music there too. However, the one person - one computer paradigm that it is built around is really limiting. I'm not talking about circumventing DRM here, just being able to have everyone in my household listen to music while protecting my investment in CDs.

Take, for example, my scenario. I am married with seven children. We have four computers in our home. I have a NT domain and each person can log into any of the computers. That means that if I install iTunes on each computer that there are potentially 36 copies of the music around my network! Ok, that is a worst case, but you can begin to see the issues.

Now, iTunes has music sharing capabilities. Using some really cool network technology, you can make your music available over the network to other people in your local subnet. Great, but the person sharing the music must be logged in and running iTunes for that to work. That is no help to me. If my wife has some music that I want to listen to, she has to be logged in. Worse than that, apple has an artificial limit of five users connected to your music in a 24 hour period. Apparently a family should only have five people in it, each with their own personal computer.

What about using my network to have a common music share? Well, here we begin to make some real progress. If I have a common directory for the music folder, then there is only one copy of the music on the network, but because iTunes keeps its database and configuration file in a user directory that is separate from the music directory, each user has to re-import all the music every time anyone imports a new CD. Still not good.

An ideal solution would be to have iTunes use the registry to point to the configuration file and database file (that could be shared) and then have the music stored on a file share from the server. But that is not what they did. Each user's music database and configuration file live in a hard-coded "iTunes" folder that exists in the users "my Music" directory. You can change the name of the "my Music" directory in the registry, but it must always be in the "my Documents" folder which is unique for each user on each machine. If anyone from apple reads this, they might take this into consideration.

Ok, Solution time.

The first part of the solution is something that I already had in place. That is to create a user policy that mapps everyone's "my Documents" folder to a file share (mine is the p: drive). You can accomplish this using a login.bat file with the following line in it:

net use p: \\server\%USERNAME% /PERSIST:NO

Next, in the policy, remap each users "my Documents" directory to their p: drive.

I'm not going to go into the details of how to create user policies here. There are lots of resources available that cover this.

If I stopped at this point, I would have achieved one really cool feature. All of the documents for every person in my family would follow them around from computer to computer. So Daniel could work on his essay on any computer in the house.

Windows does this by using "offline folders" to synchronize the "my Documents" folder and its contents from one machine to another at logon, on demand, and at logoff. One downside is that there are copies of the files on every computer that the person uses. Most people don't have that many working files, however, so this isn't too big a deal, most of the time; as we shall see later.

At this point, we still have the iTunes problem, however. That is that everyone has their own music, configuration, and database. The music issue can go away if we put all the music on a network file share. This is very important because without this, there will be a copy of each song on every computer. Now I know that hard drive space is cheap, but I have about 5 gb of mp3s, multiply that times 4 computers and 9 people and you'll agree that is a huge waste of space.

Once you have a network share for media, you can configure iTunes to point to that directory.

Final problem. Because iTunes doesn't automatically scan it's music folder for changes, you have to manually "refresh" each person's music database when anyone imports a new CD. This is a pain, because it isn't the most intuitive process. You have to re-import the directory. Well, my 9 year old likes to listen to music, but she really isn't going to run the music import function.

What would be really good is to be able to share the iTunes folder...

Enter the Symbolic Link. Microsoft has provided a feature called a symbolic link. A symbolic link allows you to make one directory available at multiple locations in your directory tree. Unix afficionatos will recognize this feature. Unfortunately, there is no built-in way to actually create these links. Do not confuse these with "Shortcuts". Lets say that I have folders "C:\iTunes", "C:\Aaron", and "C:\Daniel". If I create a symbolic link of the "C:\iTunes" directory under both the Aaron and Daniel folders, I will have paths "C:\iTunes", "C:\Aaron\iTunes", and "C:\Daniel\iTunes" that all have the same space on my disk. Or in other words, if I were to put a coniguration file in any of those directories, they would all be the same file, albeit with three different ways to access that one file.

That is exactly what I did. On my server, I created a symbolic link for the iTunes directory (with the config and database files) in each of the users "my Music" folders. That configuration file says that the "iTunes Music" folder is located in my M: drive. You just have to give all users full access to the directory.

Now, whenever any user adds music, all the users configuration files are updated and there is only one copy of the music.

Master and Commander

2006-02-20T09:33:00.000-07:00

I just thought I'd put my favorite registry hack here. Create a text file with a .reg extension with this text:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\Command]
@="Command &Prompt"
[HKEY_CLASSES_ROOT\Directory\shell\Command\command]
@="cmd.exe"

Once you have the file saved, just double-click it and merge it with the registry. This adds a context menu that can be used in the explorer to open a command prompt window to the highlighted folder.

How to skin a Turkey

2006-01-24T12:49:00.002-07:00

Here is a very useful function created to parse a delimited list in TSQL. It is fairly straight forward. Pass the string to be parsed and the delimiter (defaults to a comma) and it returns a table with a single column containing the strings that were delimited together. Why Microsoft doesn't have something like this built in is beyond me.

CREATE FUNCTION dbo.fn_ParseDelimitedList
(
@DelimitedList varchar(8000),
@Delimiter char(1) = ","
)
RETURNS @TableVar TABLE (Item varchar(100) NOT NULL ) 
AS
BEGIN
DECLARE @IDListPosition int 
DECLARE @IDList varchar(8000)
DECLARE @ArrValue varchar(8000) 
DECLARE @Pattern char(3)
SET @IDList = COALESCE(@DelimitedList, '') 
IF @IDList <> '' 
BEGIN 
-- Add Trailing delimiter to end of list so user doesn't have to 
IF RIGHT(@DelimitedList,1) <> @Delimiter
begin
SET @IDList = @IDList + @Delimiter 
END

SET @Pattern = '%' + @Delimiter + '%'
-- Loop through the comma demlimted string list 
WHILE PATINDEX(@Pattern , @IDList ) <> 0 
BEGIN 
SELECT @IDListPosition = PATINDEX(@Pattern , @IDList) 
SELECT @ArrValue = LEFT(@IDList, @IDListPosition - 1) 
-- Insert parsed ID into TableVar for "where in select"
INSERT INTO @TableVar (Item) VALUES (@ArrValue) 
-- Remove processed string 
SELECT @IDList = STUFF(@IDList, 1, @IDListPosition, '') 
END 
END
RETURN 
END

You liquidated her, eh? Very resourceful...

2005-11-09T13:56:00.000-07:00

Long time, no blog. What can I say? I've been to the "Happiest Place On Earth"

But, now that I'm back, here's a neat little trick that I learned about how to embed and use graphic resources in a Visual Studio .NET application. For my example, I'm assuming that I have an icon file named "Test.ico".

Using visual studio or another tool, you can create a bitmap, jpeg, gif, or Icon.
Put that file in the same directory as the rest of your project.
Then Add the file to the project (Add Existing Item...) and add the file.
Change the Build Action of the file to be "Embedded Resource"
When you want to use the file, simply add the following code (or something similar):

Dim _icon as Icon = New Icon(Me.GetType(),"Test.ico")

Now you have a real icon that can be used for any number of things (application icon, status icon, etc.)

Am I Blue?

2005-10-20T11:31:00.000-06:00

Just a quickie post with a picture for those of you who may not know what I look like. Actually, you still won't know what I look like because this was my Halloween 2005 costume. A "Blue Man"

This whole effect was accomplished using simple make-up items from "The Costume Shop" in Garden City, Idaho.

I prepped myself with a good shave and a haircut
The skin-head wig was prepared using Caster Sealer. This helped it to have a less streaky appearance than it would have had otherwise. This is bascally castor oil, but it makes the latex skin less slippery.
I positioned the skin-head over my hair and ears and then glued it in place with stage glue.
We applied Cobolt Blue Greasepaint to my entire head and neck.
I'm wearing a black turtleneck shirt with black pants, shoes and socks.

The entire process took about an hour.

Total cost: $20

Loss of Focus

2005-10-10T14:55:00.000-06:00

It would appear that the venerable GDI routine "DrawFocusRect" and the MFC routine "DrawFocusRectangle" have lost a bit of functionality in their .net contemporary function "DrawFocusRectangle".
In the old world, you could draw the focus rectangle once to draw it, and then again to erase it because it uses an XOR pen to draw the familiar dashed box. Under the .net version "ControlPaint.DrawFocusRectangle", it doesn't use an XOR pen, so you can't erase it with another call.
There is a function called "DrawReversableFrame", that will erase itself, but it doesn't take a parameter containing the graphics object, so you have to manually transform the coordinates to screen coordinates.
A change like this is really not very understandable because it means that somebody had to create a function from scratch rather than wrapping the windows functions (as they did most of the time).

Brent: Programmer. Gamer. Cheapskate. All around good guy.

2005-10-10T09:48:00.000-06:00

So... not being one that can leave a sore spot alone, when Brent Brown sent me an email Brent his "preferred" blog link so that I could update the link I had created to Stubenville, I naturally couldn't just update my previous blog entry. I needed to take the opportunity to pour lemon juice, salt, and vinegar into the wound until it became a pulp of agonized flesh. It may be because that's the way I am, but it may also be because Brent really needs this kind of opposition in his life. After all, he has a wonderful wife and cute kids, but as far as I can see, nobody that really gives him the harassing that he needs to keep his life in balance. That's where I come in.
I don't know what the big deal is... after all, I liked the Stubenville (not Stupid-ville, that's a whole other wound that needs opening from time to time) thread. However, since he isn't there any more, I suppose that it has a limited shelf life. So, if anyone actualy reads my blog, you should also read his because he has far more time on his hands to talk about cool stuff like the "Using" statement in C# and dot net 2.0.

(Tool)Band Aid

2005-09-15T16:04:00.000-06:00

So, I recently completed writing a Desk band for windows. What is a desk band, you ask? Well, your windows start bar is the home for several desk band objects. These are basically COM objects that support a bunch of interfaces and register themselves in appropriate places so that windows knows to load them.

In short, you register your windows class and it gets put in the toolbar. Nearly the same code can work in multiple places in IE, Explorer, and in the Start Bar. So, I learned a few lessons:

It is far easier to do your initial development so the band will work in the IE toolbar than in the start bar. This lets just run the COM object (and automatically start IE) rather than manually starting up the desk band, and attaching to the Explorer process. When you get most of what you need done, you can test it in the Start bar.
A conditional compilation switch in your code will make sure that you don't accidentally register the release version of your desk band object to run in IE (if you don't intend it to).

So, I based my deskband on the sample SDK code from Microsoft. But it has a few problems.

It registers fine. The normal "regsvr32 myobj.dll" works great, but it doesn't implement any unregister functions. You have to write those yourself. Now, I know that if you use ATL or MFC that you can get that for basically free, but it seems a bit of an overkill if you are doing just a small simple COM object. And besides, you have to do custom registration for it to get loaded automatically anyway.
The window with the text is kind of ugly. I really wanted a desk band that didn't have a big square box.

So, here's what I did to get rid of the box...

First of all, I registered the window class with a background color of NULL. That way, it won't automatically erase the background with blue. I also changed the flags so that there isn't a black border around the window contents.

The next thing I did was implement a handler for the Erase method that windows will call whenever the window needs to be erased. MSDN says that you should do this. If you don't do this, you'll get text that writes on top of itself over and over, works fine until your text changes. Then it looks like some kid scribbled on the screen. But at least you have the same background that the toolbar has. (there are some people that don't use the blue jelly toolbar, they must like grey).

So, how do we keep the nice gradient, or the grey background, or whatever they have as a background while we're drawing on it all the time? StretchBitBlt. Enough said.

... What? That wasn't enough said?

Ok, so here's what I did. realizing that the gradient is vertical, when my erase event gets invoked, I just stretch bit blit whatever is on the left hand side of the draw window all the way across my drawing rect. Then, I text out on top of that. It's easy once you figure it out.

Windows XP ala' Burger King (have it your way)

2005-09-14T10:44:00.000-06:00

I got my Shuttle SN25P XPC on Monday from NewEgg and immediately put it together. The components that I had delivered with it are:

AMD Athlon X2 64 3800+ processor
2 GB Crucial pc-3200 memory (2x 1GB)
Samsung SATA2 250 GB drive
Gigabyte Nvidia GeForce 6600GT 128 MB video card

Then I spent most of the evening installing Windows XP, service packs, and all kinds of hot fixes. Not to mention the drivers needed by my machine (like everything!) Well, to make a long story short, the UN-service packed version of XP doesn't recognize drives over 130 GB or so, so it formatted the drive smaller than it should have been. So now I'm faced with resizing the drive with partition magic or something similar.

I could have saved myself a lot of time using the following utilities and programs:

Unattended XP CD
nLite - Tool for slipstreaming and customizing windows installations
RyanVM's Windows XP Post-SP2 Update Pack - latest hotfixes all together in one install or separately
DriverPacks - up-to date drivers for slipstreaming, by category

The Unattended XP CD link has links to everything else that you need, but also is a good reference by itself. However, for casual use, I recommend just using nLite's options along with your own Windows XP sp2 CD and RyanVM's update pack and the DriverPacks.

And of course, once you have minimized the size of the installation, you can also install it from a USB memory stick using a free utility from HewlettPackard

Most excellent!

Jack of all trades...

2005-09-13T13:36:00.000-06:00

I guess if Brent can do it, so can I. After much coaxing and goading, I am going to keep a blog of cool stuff that I figured out. This is not so much a vanity site, I really don't believe that anyone will read what I have to say as it is my anti-altzheimers mechanism of choice.
Or, to put it another way...

"Cool. Why don't you put that in a blog so that when I ask you how you did it a year from now you won't just give me a blank expression" - Brent

Oh, yea, and Scott has said almost the same thing from time to time.

Now, for those of you who don't know me... What are you doing here? You must really be bored.