Diary Of A CEO - The Really Hard Part

If you are just coming across this article, you might want to read the previous installment(s) first.

• Preface
• Chapter 1 - Question Everything

Introduction

After deciding the general approach behind the server's mechanisms and communication with the client, it was time to start testing some of this in code. The design of the client application was not settled at all, but the rest could be prototyped. This turned out to be more challenging than the design. As a preliminary, experimental project, this was a relatively low priority, but the first few attempts bore little resemblance to the desired end goal. At that time, the in-house development platform of choice was Delphi, and most of the early offerings from the programmer/analysts ended up using remote datasets. "Well, you said you wanted multi-tier, and that's how Delphi does it!" they would say. In the end, it was necessary to create a simple example (starting with a demo program from the Indy Project) to demonstrate what was needed. With this sample in hand, a prototype came together fairly quickly.

False start #1

The first phase of the prototype was to create a hard-coded screen that would perform the following steps:

1. Accept an input value.
2. Place it in the client container.
3. Update the server's container with that information.
4. Run a business rule that used the updated data (hard-coded at this point).
5. Change a different field in the server's container.
6. Return that to the client.
7. Update the client's display.

As illustrated by:

It certainly sounds simple, and once the concept was clear, it turned out to be fairly straightforward. Big hurdles still loomed, however. Separate programmers were assigned the next two tasks:

1. Come up with a way to identify and run a business rule that was not compiled into the application.
2. Create a user-definable screen that could connect to the new data objects.

Business rules suddenly took on a life of their own. The relatively simple requirement became a highly convoluted set of XML specifications (essentially a new programming language), driven by user screens that drove flow of control, assignment of variables (including a complete expression evaluator), database access and the ability to run stored procedures. In addition to being very rigid, it also tied the application to a single database platform, and would have been very difficult to port to other operating systems (which was one of the design goals.) With a little staff shuffling (and a couple of departures), focus returned to the original goals.

False start #2

Starting all over on business rules turned out to be a good move. Recent experiments with Python for simple scripting had identified it as a good option. It could be loaded at runtime and a library was available to both call scripts and to pass data back and forth with Delphi. The business rule name would be the name of the Python file.

Deciding on this course of action and laying out the basic plan took a day. Describing it to the programmer took another day. Within a week, the entire process worked like a charm. Performance was fairly good (considering the flexibility offered). Tests were run that made 10,000 calls to the business rules and the results were acceptable enough to encourage continued development.

While the business rule distraction was underway, development plowed ahead on the user interface portion. The programmer/analyst assigned to this task was the same one who had created the excellent user field and user file capabilities in the application that we were trying to replace. Of course, his approach was to use the same mechanisms for the new system. Some major changes were required:

• All screens needed to be user-defined. Even standard screens could be modified.
• Data on the screens might be standard application data or user-defined data.
• Menu options needed to be user-defined, and run server-side business rules.
• All screens needed to be non-modal.

Within a few months, an application was put together that met all of the new requirements. It ran fairly quickly, and allowed users to change screen designs directly from the running program (getting these into "Layers" was left for a later exercise.) Encouraged by this progress, it was decided to demonstrate the prototype at a trade show in February of 2004. Since cross-platform compatibility was one of the design goals, it was decided to test this concept. Within two weeks, a server and client had been compiled with Kylix (Delphi's Linux twin) and both servers and clients were displayed at the show. By simply changing the server connection string, the same application could be run on Windows Server with SQL Server and on Linux with MySQL. The Python business rules were implemented on both platforms as well.

As exciting as these developments were, there was still a nagging suspicion that not all of the design goals had been met. It seemed that the client was being continually "tweaked" to add new functionality. Closer examination proved that this client could do anything that the user wanted, as long as the user wanted the right things. While very flexible in what information was to be displayed, the flow of control was highly regimented, and the look and feel of the screens was virtually cast in stone. Also, the Python business rules were running well, but there was some concern about telling clients that they needed to use Python to develop their rules (although a simple user interface was planned). It was also a somewhat convoluted install process.

As painful as it was, one of those decision points had been reached. It was time to start over. Part of this analysis was the development tool that we were using. We needed platform independence. That pretty well meant C++, or one of the environments like Java, Perl or Python. C++ was virtually eliminated because of the development cycle and the difficulty of connecting new business logic at runtime. That left the virtual machines and interpreters. Performance was a concern with all of these platforms, plus the availability of programming help. It seemed like the world was moving to .NET, but that was just Microsoft, right?

.NET enters the picture

It was then that the Mono project entered the picture. While the .NET Framework was a Microsoft product, they had opened the specification and there was an active project close to releasing a version that would run on Linux. Since the current product that was being replaced only ran on Windows, it was OK for the new product to start there. It was only necessary that there be a development path to other environments. .NET would allow us to compile business rules and load them at runtime. It looked like the answer.

Skill set issues

Convincing the current staff to start all over was going to be a challenge. First, none of them had any experience with .NET. In addition, they were justifiably proud of the work that they had done so far, and were loath to abandon it. Finally, other priorities (tasks that were actually paying the bills) were taking up all of their available time. Short of hiring an additional staff of programmers just for the new project, there was only one solution: outsourcing.

Outsourcing

As an isolated project with a clear end result, this seemed like a perfect outsourced project. The in-house staff would be trained on .NET, and they would be ready to take over the application development when the base platform was delivered. The search began.

With no experience in outsourcing, it was necessary to narrow the field somewhat. There were a great number of firms advertising on the Internet, from all over the world. With no way to distinguish one from another technically, a totally non-technical consideration drove the decision. One of our in-house programmers was a native Russian. There were outsource firms in Russia. We would use one of those.

During July of 2004, a number of firms were contacted with specifications for a very small part of the project. Create a data container, with data objects. Implement the IsDirty flag to synchronize the two. Use a simple TCP client/server mechanism to send the command packets. Make sure that this simple task would run under Mono, as well as Windows. The solicitation wanted several responses:

• How do you price your services?
• How much would this mini-project cost?
• If this is really trivial, feel free to write this part of the system as a demonstration.

Twelve solicitations were sent out, and virtually all replied. Five of those were confident enough to produce a test project. Two of those actually thought to ask questions to clarify what was really wanted. In the end, only three of the programs actually did what was needed (one of the others used remoting instead of sockets). It was determined that one of the projects seemed to be closest to the desired style and they had communicated well. Since there was a great deal to be done, this development team was put on retainer starting at the end of August, 2004 to do continuing work. Their rates, as expected, were very reasonable.

User interface abstraction

With a resource available, work began on the user interface again. This time, the goal was to create an interface that would allow clients on any platform to work with the new server. For example, the new system was to have a browser-based product that would mirror all of the capabilities of the intranet desktop client version. The prospect of maintaining two separate sets of user interface logic was not attractive, even if the business logic was the same. This requirement led to a decision to separate the specification of the user interface from its implementation. In simple terms, we needed to be able to lay out screen designs and "wire them up" to data and processing without actually coding that to any specific platform.

As pointed out earlier, there has been considerable progress made on cross-platform development. The user interface has been a late-comer to this party. While some user interfaces run on multiple platforms, they seldom look like native applications. One approach to the "run anywhere" goal is to implement the user interface inside of a browser. While this does provide the ability to run the interface on any browser-equipped system, browser incompatibilities are legion. This is especially true when trying to "push the envelope". Most browsers can handle simple HTML screens. Unfortunately, limiting your application functionality to that simple level produces a horrible user interface for anything more than the very basic application. Many recent efforts have been made to produce a "rich client" (referring to the experience, not the cost.) Some of these results have been very good, but the best still does not compare to a native application on a intranet desktop client.

How is it possible to create a client that will provide a native-application feel, be available on a number of platforms, provide excellent performance and yet be fairly easy to build? The answer involved several decisions:

• Like in the browser, provide display specifications in a platform-neutral form.
• Separate the "processing logic" of the client from the display.
• Communicate to the server in a platform-neutral way.
• Implement a platform-specific client that utilizes the platform-neutral parts with a specific display interface.

Like standing up a bunch of dominos and pushing the end one over, each decision led almost inescapably to the next. The design started to come together. Browsers use HTML to specify screen layouts. While this is a platform-neutral screen layout, it is highly limited and verbose. Since we were using XML elsewhere (even in our original screen layout attempt), it probably made sense here too, but we needed a standard way to describe the layout. A search began for that standard mechanism.

XML again

Unfortunately, there are a significant number of "standards" to specify screen layouts. One recent one was XAML, which was to be part of the Longhorn project from Microsoft. It was close to the right idea, but it was highly tied to the Avalon namespace, very .NET Framework specific, and encouraged (at least at that time) the placement of code in the screen, which was not what the design called for. XAML's entanglement with the Avalon namespace meant that it didn't do everything that was needed and conversely, too much of what wasn't needed. It looked like we were back to creating our own layout and parsing engine.

The layout wasn't the hard part, however. Once the XML layout was created, it needed to be interpreted by the client in order to create the screen display. During the ongoing search for a tool to help in that task, a discovery was made that would change the path of the project forever.