Epic Battle: Simplicity Versus Complexity

I’ve seen this too many times to not consider it a common theme in every project. I’m sure I’m not going out on a limb when I say that everyone starts their project with the goal of simplifying their current process, but with all that can be accomplished with the Extranet framework and starter solutions, the temptation to add additional complexity is often tough to resist. I submit the following for your consideration:

Complexity is evil and should only be allowed into your project when absolutely necessary

Strong statement, I know, but I find that we don’t ask the question “Is there a simpler way?” often enough. I’ve been wondering lately why this is the case and have formed a theory:

Simple is boring and we don’t like being bored

Now, before you disagree with me, let me explain.

The Dark Side of the Moon feeling

This morning I had a moment to reflect on how differently I’m able to interact with my two sons, one 15 and the other 13. My oldest son has reached a point in his life where he enjoys conversation. No subject is out of bounds and I think I get as much or more out of the discussions as he does. This is in stark contrast to what my younger son is able to provide at the moment. With him it’s the typical parent-child dialog where the parent has to play 20 questions just to get anything out of the child and there are more moments of silence than moments of verbal exchange. I’m not complaining, I realize this is typical and I had the same experience with my older son when he was that age. I’ve come to consider that age as a “Dark Side of the Moon” period. But it is very frustrating.

Phone, Web Conferencing and Email are all inferior

As I sit here in the Detroit airport facing a 4 hour delay and scheduled arrival back in Portland at 2AM PST, it would be easy for me to rail against the need for face to face visits, but I won’t. It’s incredibly valuable to get together with the team who is ultimately responsible for delivering solutions and countless updates of those solutions to a user base that is always asking for more. In fact, my visit with just such a team this week at University of Michigan only serves to reinforce my belief that there is no replacement for the occasional face-to-face get together. Phone, Web Conferencing, and Email are all inferior to physical presence.

Tips for managing WebrCommon files

Today’s post is short and sweet and I hope will save you a headache or two down the road. It’s related to managing the contents of your WebrCommon directory.

Tip #1
As most of you have already upgraded to Extranet 5.6 or are actively working your way there, it’s a good opportunity to emphasize that the best way to add or update your WebrCommon files is through Site Designer. This is especially true with Extranet 5.6 because, as of that release, the Click Commerce Extranet framework actually manages two locations for the WebrCommon files. In the past you were able to get away with simply dropping files directly into appropriate directory in the Windows file system. With the addition of the new location, this isn’t enough and can lead you to scratching your head about why your newly updated custom.css file doesn’t seem to be working. If you drag the file into Site Designer, on the other hand, all the locations are updated correctly. More information about the changes to WebrCommon in Extranet 5.6 can be found in this article: HOWTO: Manage the Webrcommon Directory in Extranet 5.6

Even in Extranet 5.5.3, there are benefits to dragging and dropping in Site Designer. Using this approach instead of direct file system access will avoid issues related to WebrCommon files that have their file attributes or permissions set in such a way as to break backup/restore or use within your site. Dropping the files into WebrCommon via Site Designer causes them to set up correctly.

It’s as easy as Drag-and-Drop.

Tip #2
Since you are updating the contents of your WebrCommon directory, I can safely assume that you are doing this in your development store that is integrated with Source Control (right?…please tell me I’m right!). This means that any new files need to be added to your version repository as well. In most cases, Process Studio is the tool you use to do this, but there is one exception and that is with Binary files such as images. In the case of binary files you should use the Visual Source Safe client directly to add or update them. This is because Process Studio makes the assumption that files are textual. This assumption doesn’t work too well when they aren’t and can result in images not looking right. Once in source control, however, there are no concerns about them being included in your next configuration update and being correctly applied to your test and production stores. Support for binary files will be added to Process Studio with the release of Extranet 6.0.

Cheers!

What’s new in Professional Services

With only one week before the annual Click Compliance Consortium meeting in Chicago, the office is buzzing. The agenda is firming up and it looks like we’ll be hovering near record attendance levels. Each CCC has it’s own flavor thanks to the fact that it’s really an event hosted, presented, and attended by members of the consortium itself – yes, that means you. Of course there’s always the expected presentation by those of us at Click Commerce, but it’s learning what all of you are doing and the free exchange of ideas that I look forward to the most.

This year we’re trying something different that I’m excited about. On the second day the agenda splits into three tracks and we all get to spend a couple of hours in an open discussion about one of three topics: IRB, Grants, or Animal Operations. I, with the help of a couple of customers, will be moderating the Animal Operations discussion and I hope those of you with interest will be able to join in on the conversation. I hoping that this will be a lively session.

Closer to home, I’ve been continuing the drive to refine our development practices. I’m not quite ready to share any results but here’s a short list of things we’re working on:

Formalization of best practices related to the development of reports using SQL Server Reporting Services.
We presented on this at C3DF and I’ll touch on it again during my CCC Reporting presentation. We’re currently experimenting with different applications of this technology so that we can determine the situations where it is a good fit and, equally important, where we would recommend other approaches.

Automated Testing
I’m especially excited about our early progress on this topic. One of the biggest challenges in any enterprise scale development project is knowing that the most recent change has no adverse effects. Did this cool new enhancement break any pre-existing capabilities? What is the performance impact of this new enhancement? We’re currently implementing automated tests for a large IRB project using three open source tools (Cucumber, Watir, and Celerity). It’s amazingly rewarding to push a button and see the browser run through an entire workflow scenario without any further assistance from me. I’ve come to really like the color green, which is what the final report shows for tests that pass. When I see red on the other hand, representing failed tests, I get a strong urge to immediately make it green. I’ll definitely be blogging more about this in the future.

Introduction of Agile development practices
One of my own personal goals is to improve the way information flows from Project Manager to developer, provide a more real time snapshot of the status of the project, and establish good development techniques early in the project before the stress level is too high to want to try something new. For all of these reasons, I’m a fan of an Agile development methodology. In my mind, the development model I prefer is to establish a routine from the beginning of the project which is based upon a repeating cycle of development that involves these elements:

• Establish 3 stores: Development, Test, and Future Production. The development store is integrated with Source Control from day 1 and implementation is broken down onto 2 week iterations. At the end of each iteration a patch is created and applied to both Test and Future Production.
  
  Test is where all user acceptance testing takes place. Testing on the development store is limited to the developer’s themselves testing each other’s work as part of unit testing.
  
  Future Production starts out as a pristine store, empty of all data and is given the white glove treatment. As patches are built after each iteration, they are applied to this store to keep it up to date. In this way, we have a clean store to use when going live without first having to worry about clearing gout all the data that collects through the course of development and testing.
• Only update Test and Future Production via patches. This means that the same process we recommend once your site is live is enforced early in the project. In my mind this has many benefits. It means that by the time you go live, this process is second nature and it avoids the “deer in the headlights” feeling of being asked to introduce this process at the time of going live, which is a time when you are already stressed about a lot of other things.
• Test incrementally. Thinking about how to test the entire system all at once is a bit frightening. Though the process of applying patches after each development iteration we can parcel out the testing burden and at the same time get end user feedback much earlier in the development process when there is actually time to properly respond.

As a part of this effort, we’ve also been piloting the use of a new tool to help us track all the development and so far it’s being received well. At the end of our pilot, I’ll post a summary of our experiences and conclusions.

I look forward to seeing all of you at the CCC conference next week in Chicago!

Cheers!

Centralizing your implementation of validation rules

It’s your business rules that allow you to turn all the building blocks provided by Click Commerce Extranet into your own solution. Business rules manifest themselves in the names of the workflow states you use, how a project moves through it’s lifecycle and the actions available to the user at each step along the way, how you define your users, security policies, the information you collect, and criteria against which that information is verified. All of these configuration choices, supported by the Click Commerce Extranet platform, are what makes your site uniquely yours. Verifying information according to your institutional requirements involves the implementation of Validation Rules. This post will present one approach to their implementation.

Consider this validation rule:

In your IACUC solution, if animals are to be sourced through donation, you require that there be a description of the quarantine procedures that will be used.

Sounds like a reasonable requirement, right? So, where would you enforce that rule? A key advantage to the Click Commerce Extranet platform is it’s flexibility but sometimes determining the best approach requires that you weigh the pros and cons. This rule is an example of a conditionally required field. Here are a few of the most common approaches to implementing this type of rule:

1. SmartForm branching
   You can enforce the rule by separating the selection of animal source and the follow-up questions onto different SmartForm steps and use a combination of SmartForm branching and required fields. This is by far the easiest implementation because it allows you to take advantage of the built-in required fields check and can be accomplished without any additional scripting. It does however, require that the questions be separated into multiple SmartForm steps. This isn’t a big deal if there is already an additional steps where the follow-up question could be placed but this may not always be the case.
2. Conditional Validation Logic
   With a bit more work you can keep the questions on a single view and implement conditional validation logic in a script. This allows you to keep the fields together but the follow-up question will be visible to the user in all cases. You will need to include instructional text into the form to let the user know that the second question is required if the initial question is answered in a particular way.
3. Conditional Validation Logic with Dynamic Hide/Show
   With yet even more work, you could dynamically show the relevant dependant questions only when the user is required to answer them. They would otherwise be hidden. This technique is the subject of an upcoming post but it’s important to understand that the enforcement of the validation rule is still accomplished through custom script.

By far, the easiest implementation is option1 because the Extranet application will perform all the validation checks for you. But what if your rule doesn’t fit within a simple required field check or your users won't let you separate the questions onto different SmartForm steps? In these cases, you will have to implement some logic. Knowing that, you are still faced with the decision of where to put it. Again, there are options:

1. Add custom logic in a View Validation Script
   This is the preferred approach as the configuration interfaces are exposed via the standard Web-based configuration tools.
2. Override the Project.setViewData() method
   This technique has been replaced with the View Validation Script but before that script hook was available was the best place to add custom logic. It is no longer the recommended approach.
3. Override the Project.validate() method
   This method is called when validating the entire project as would happen when the user clicks on Hide/Show errors in the SmartForm or executes an activity where the "validate Before Execution" option is set. It is not invoked, however, when a single SmartForm step is saved so is really only a good place for Project level validation.

Option 1 is the most common approach and is much preferred over option 2. Both approaches will allow you to enforce validation rules whenever a view (or SmartForm step) is saved. This is what I call “view validation.” All view validation rules must be met before the information in the view can be considered valid. This means that a user cannot save changes or continue through the SmartForm until all rules for the current step are met. This is applicable to most needs but not all. Let’s consider another rule that also must be enforced:

In order for a protocol to be submitted for review, the PI and Staff must have all met their training requirements.

Enforcing this rule when posting a view or SmartForm step would be overly restrictive. The PI and Staff should be able to complete the forms even if their training is incomplete. The Rule is that the PI cannot submit the protocol for review until all have met the training requirements, so a View validation won’t work. What is needed is Project level validation which can be accomplished by overriding the Project.validate() method.

So this means that some rules are implemented in a View Validation script using the web-based configuration tools, while other rules are implemented using Entity Manager. This approach definitely works and is seen in a large number of sites. The downside is that code is maintained in different places and script is required for all rules.

In addition, all the scripting approaches It doesn’t take into consideration that many rules follow common patterns. For example,

• If field A has value X, then a response for field B is required, or
• Field A must match a specific pattern such as minimum or maximum length

What if these patterns could be formalized into a standard way of defining a validation rule? What if all rules were defined in the same way and in the same place? Would that make implementation, testing and maintenance easier? I certainly think so.

Introducing the ProjectValidationRule implemenation

ProjectValidationRule is a Selection Custom Data Type whose sole purpose is to serve as a central place to define and execute your site’s validation rules. Some of you may have previous experience with a CDT named “SYS_Validation Rules” and any similarities you see are not a coincidence. That type provided the seeds from which this new implementation was grown. It allows for rules following common patterns to be defined without authoring any additional script and provides the flexibility to define whether the rule is to be enforced at a View or Project Level. As an added bonus, you can also define activity specific rules.

Download the ProjectValidationRule package for specific implementation details. This approach is still evolving and it’s proven very effective so far but there is always room for improvement so feedback is always appreciated.

Cheers!

How best to kick off a new project?

Within Click Commerce Professional Services, we are always looking to improve ourselves. A team that is content to continually do the same things the same way is one that will cease to be relevant, so recently I’ve begun to take a hard look at how we begin the process of implementing a new customer solution.

At a certain level of abstraction, most projects follow a common pattern. This is true no matter the project or technology, but becomes even more true when all projects leverage the same platform. All Extranet-based solutions have the following elements in common:

• Workflow
• SmartForms
• Notifications
• Business Rules
• Presentation
• Reports
• Robust, context-driven security
• Integration with external systems

We begin each project by demonstrating our starter solution then walking through the workflow in detail with a broad customer team representing as many of the different constituencies as possible, including the central administrative staff, project sponsors, other domain experts, and the development team. This approach has proven incredibly valuable in refining the project requirements by identifying where the specific customer workflow deviates from (or is the same as) the workflow in the starting solution. From this Kickoff meeting, Click Commerce is able to define a detailed project implementation plan. This has been a well established process for the past few years and it has been very effective. I have to ask myself, however, can we do even better? It’s good to challenge the status quo now and again, right?

If you look at any Click-based project objectively, it’s easy to recognize that Workflow is only one part of a complete deployment. Observing the course several projects have taken recently, I’m beginning to think that SmartForms are just as important to understand early in the project. To put it another way, the method of collecting data is just as important as the path that data takes through its lifecycle and understanding both as early in the implementation project as possible increases the quality of the initial design and reduces the need for costly design changes later.

Many years ago, IBM promoted a system design process called JAD which stands for Joint Application Design.  The whole premise of this approach was to examine the current paper-based process as a way to design a new computer application.  Many aspects of that process now seem “old school” but, as is often said, there are very few truly new ideas. Most are improvements on old ones. The process we use at Click certainly doesn’t break new ground but has been tailored to our products. The JAD process promoted a multi-day working session, where all of the domain experts (people who actually follow the current process as a part of their actual jobs) get together in the same room and talk through their jobs. Each is required to bring with them the paper forms they work with and describe how they are used in their part of the process. I participated in a few of these JAD sessions and it was surprising to me to discover how often one person’s part of the process was completely unknown to others involved in the same overall process. Jobs functions tend to get pretty isolated and silos are a natural result of people being so busy it’s all they can do to focus on what they have to do. There’s no time to understand the nuances of another’s job. The one thing about JAD that I came to appreciate was that it focused both on process and information collection. Understanding what information is collected and how it is collected was the real driver for designing an effective data model.

Fast forward to the present. In our Project Kickoff, we demonstrate the starter SmartForms and Workflow then spend the rest of the kickoff walking through the Workflow in detail in order to identify customer specific deviations. Is workflow the right place to start? I wonder what would happen if we began by understanding the what and how of data collection first then followed that up with a discussion of how the collected information travels through the workflow. I see several benefits with this approach:

1. After the demonstration, the Kickoff continues with what is most familiar to the assembled customer team – the current forms and a discussion of what works and what could be better
2. A clear understanding of what goes into the project makes the discussion of workflow and the review process easier – we now have better context
3. An understanding of the collected information is the first step toward a solid object model
4. Workflow cannot exist without the object model. If the model supports the data collection process from the beginning, the implementation of the workflow, SmartForms, reports, and presentation will be easier and will result in less rework caused by having to change the model when implementing the SmartForms and less need to implement creative solutions to leverage the model upon which the workflow is based

Is the information collection process more important to understand in a Kickoff than Workflow? That’s a tough question to answer. Understanding the workflow is critical as well and ideally both should be discussed in detail as early in the project as possible. I wonder if that would be too overwhelming to the Kickoff attendees to cover both topics during the meeting. My crystal ball gets a bit cloudy at this point. However, I’m of the opinion that extending the Kickoff to cover both would result in even better and more predictable implementations.

I’d love to hear your thoughts on this.

Cheers!

Technology Preview: Multi-Solution Development in Extranet 5.6

Extranet 5.6 includes an early peek at what I expect will become an important tool for those of you who have implemented multiple solutions in your site. It’s called “Module and Solution Support” and its goal is to allow for the independent development and release of configuration updates between the different solutions you are maintaining.

Before you get too excited, it’s important to realize that as promising as this feature is, it’s not a panacea. There are many methodologies and project management techniques to deal with your constantly evolving workflow solutions and, while this enhancement adds another tool to your toolbox, it doesn’t meet every need you could imagine. What it does provide, however, is a big step toward being able to manage different solutions on different development schedules.

Enabling this option is a one way trip so it’s best to first explore this new feature in an isolated experimental development environment. If you…

• are already familiar with using Process Studio to manage the development of your site,
• have more than one solution deployed (such as IRB, IACUC, COI, etc.), and
• face the challenge of wanting to deploy updates to the different solutions on different schedules,

then Module and Solution Support is worth a look. I’m currently using this feature on one of my projects and will update you all on my experience in a later post.

Module and Solution Support is provided as a technology preview with Extranet 5.6 and is only one of many cool new features. Start planning for your upgrade today using the Extranet 5.6 Pre-Installation Package. If you want to know more about how to upgrade, drop me an email, and I’ll fill you in on the details. To accelerate your upgrade, check out our new Extranet 5.6 Upgrade Service.

Cheers!