Tips for managing WebrCommon files

Today’s post is short and sweet and I hope will save you a headache or two down the road. It’s related to managing the contents of your WebrCommon directory.

Tip #1
As most of you have already upgraded to Extranet 5.6 or are actively working your way there, it’s a good opportunity to emphasize that the best way to add or update your WebrCommon files is through Site Designer. This is especially true with Extranet 5.6 because, as of that release, the Click Commerce Extranet framework actually manages two locations for the WebrCommon files. In the past you were able to get away with simply dropping files directly into appropriate directory in the Windows file system. With the addition of the new location, this isn’t enough and can lead you to scratching your head about why your newly updated custom.css file doesn’t seem to be working. If you drag the file into Site Designer, on the other hand, all the locations are updated correctly. More information about the changes to WebrCommon in Extranet 5.6 can be found in this article: HOWTO: Manage the Webrcommon Directory in Extranet 5.6

Even in Extranet 5.5.3, there are benefits to dragging and dropping in Site Designer. Using this approach instead of direct file system access will avoid issues related to WebrCommon files that have their file attributes or permissions set in such a way as to break backup/restore or use within your site. Dropping the files into WebrCommon via Site Designer causes them to set up correctly.

It’s as easy as Drag-and-Drop.

Tip #2
Since you are updating the contents of your WebrCommon directory, I can safely assume that you are doing this in your development store that is integrated with Source Control (right?…please tell me I’m right!). This means that any new files need to be added to your version repository as well. In most cases, Process Studio is the tool you use to do this, but there is one exception and that is with Binary files such as images. In the case of binary files you should use the Visual Source Safe client directly to add or update them. This is because Process Studio makes the assumption that files are textual. This assumption doesn’t work too well when they aren’t and can result in images not looking right. Once in source control, however, there are no concerns about them being included in your next configuration update and being correctly applied to your test and production stores. Support for binary files will be added to Process Studio with the release of Extranet 6.0.

Cheers!

What’s new in Professional Services

With only one week before the annual Click Compliance Consortium meeting in Chicago, the office is buzzing. The agenda is firming up and it looks like we’ll be hovering near record attendance levels. Each CCC has it’s own flavor thanks to the fact that it’s really an event hosted, presented, and attended by members of the consortium itself – yes, that means you. Of course there’s always the expected presentation by those of us at Click Commerce, but it’s learning what all of you are doing and the free exchange of ideas that I look forward to the most.

This year we’re trying something different that I’m excited about. On the second day the agenda splits into three tracks and we all get to spend a couple of hours in an open discussion about one of three topics: IRB, Grants, or Animal Operations. I, with the help of a couple of customers, will be moderating the Animal Operations discussion and I hope those of you with interest will be able to join in on the conversation. I hoping that this will be a lively session.

Closer to home, I’ve been continuing the drive to refine our development practices. I’m not quite ready to share any results but here’s a short list of things we’re working on:

Formalization of best practices related to the development of reports using SQL Server Reporting Services.
We presented on this at C3DF and I’ll touch on it again during my CCC Reporting presentation. We’re currently experimenting with different applications of this technology so that we can determine the situations where it is a good fit and, equally important, where we would recommend other approaches.

Automated Testing
I’m especially excited about our early progress on this topic. One of the biggest challenges in any enterprise scale development project is knowing that the most recent change has no adverse effects. Did this cool new enhancement break any pre-existing capabilities? What is the performance impact of this new enhancement? We’re currently implementing automated tests for a large IRB project using three open source tools (Cucumber, Watir, and Celerity). It’s amazingly rewarding to push a button and see the browser run through an entire workflow scenario without any further assistance from me. I’ve come to really like the color green, which is what the final report shows for tests that pass. When I see red on the other hand, representing failed tests, I get a strong urge to immediately make it green. I’ll definitely be blogging more about this in the future.

Introduction of Agile development practices
One of my own personal goals is to improve the way information flows from Project Manager to developer, provide a more real time snapshot of the status of the project, and establish good development techniques early in the project before the stress level is too high to want to try something new. For all of these reasons, I’m a fan of an Agile development methodology. In my mind, the development model I prefer is to establish a routine from the beginning of the project which is based upon a repeating cycle of development that involves these elements:

• Establish 3 stores: Development, Test, and Future Production. The development store is integrated with Source Control from day 1 and implementation is broken down onto 2 week iterations. At the end of each iteration a patch is created and applied to both Test and Future Production.
  
  Test is where all user acceptance testing takes place. Testing on the development store is limited to the developer’s themselves testing each other’s work as part of unit testing.
  
  Future Production starts out as a pristine store, empty of all data and is given the white glove treatment. As patches are built after each iteration, they are applied to this store to keep it up to date. In this way, we have a clean store to use when going live without first having to worry about clearing gout all the data that collects through the course of development and testing.
• Only update Test and Future Production via patches. This means that the same process we recommend once your site is live is enforced early in the project. In my mind this has many benefits. It means that by the time you go live, this process is second nature and it avoids the “deer in the headlights” feeling of being asked to introduce this process at the time of going live, which is a time when you are already stressed about a lot of other things.
• Test incrementally. Thinking about how to test the entire system all at once is a bit frightening. Though the process of applying patches after each development iteration we can parcel out the testing burden and at the same time get end user feedback much earlier in the development process when there is actually time to properly respond.

As a part of this effort, we’ve also been piloting the use of a new tool to help us track all the development and so far it’s being received well. At the end of our pilot, I’ll post a summary of our experiences and conclusions.

I look forward to seeing all of you at the CCC conference next week in Chicago!

Cheers!

Centralizing your implementation of validation rules

It’s your business rules that allow you to turn all the building blocks provided by Click Commerce Extranet into your own solution. Business rules manifest themselves in the names of the workflow states you use, how a project moves through it’s lifecycle and the actions available to the user at each step along the way, how you define your users, security policies, the information you collect, and criteria against which that information is verified. All of these configuration choices, supported by the Click Commerce Extranet platform, are what makes your site uniquely yours. Verifying information according to your institutional requirements involves the implementation of Validation Rules. This post will present one approach to their implementation.

Consider this validation rule:

In your IACUC solution, if animals are to be sourced through donation, you require that there be a description of the quarantine procedures that will be used.

Sounds like a reasonable requirement, right? So, where would you enforce that rule? A key advantage to the Click Commerce Extranet platform is it’s flexibility but sometimes determining the best approach requires that you weigh the pros and cons. This rule is an example of a conditionally required field. Here are a few of the most common approaches to implementing this type of rule:

1. SmartForm branching
   You can enforce the rule by separating the selection of animal source and the follow-up questions onto different SmartForm steps and use a combination of SmartForm branching and required fields. This is by far the easiest implementation because it allows you to take advantage of the built-in required fields check and can be accomplished without any additional scripting. It does however, require that the questions be separated into multiple SmartForm steps. This isn’t a big deal if there is already an additional steps where the follow-up question could be placed but this may not always be the case.
2. Conditional Validation Logic
   With a bit more work you can keep the questions on a single view and implement conditional validation logic in a script. This allows you to keep the fields together but the follow-up question will be visible to the user in all cases. You will need to include instructional text into the form to let the user know that the second question is required if the initial question is answered in a particular way.
3. Conditional Validation Logic with Dynamic Hide/Show
   With yet even more work, you could dynamically show the relevant dependant questions only when the user is required to answer them. They would otherwise be hidden. This technique is the subject of an upcoming post but it’s important to understand that the enforcement of the validation rule is still accomplished through custom script.

By far, the easiest implementation is option1 because the Extranet application will perform all the validation checks for you. But what if your rule doesn’t fit within a simple required field check or your users won't let you separate the questions onto different SmartForm steps? In these cases, you will have to implement some logic. Knowing that, you are still faced with the decision of where to put it. Again, there are options:

1. Add custom logic in a View Validation Script
   This is the preferred approach as the configuration interfaces are exposed via the standard Web-based configuration tools.
2. Override the Project.setViewData() method
   This technique has been replaced with the View Validation Script but before that script hook was available was the best place to add custom logic. It is no longer the recommended approach.
3. Override the Project.validate() method
   This method is called when validating the entire project as would happen when the user clicks on Hide/Show errors in the SmartForm or executes an activity where the "validate Before Execution" option is set. It is not invoked, however, when a single SmartForm step is saved so is really only a good place for Project level validation.

Option 1 is the most common approach and is much preferred over option 2. Both approaches will allow you to enforce validation rules whenever a view (or SmartForm step) is saved. This is what I call “view validation.” All view validation rules must be met before the information in the view can be considered valid. This means that a user cannot save changes or continue through the SmartForm until all rules for the current step are met. This is applicable to most needs but not all. Let’s consider another rule that also must be enforced:

In order for a protocol to be submitted for review, the PI and Staff must have all met their training requirements.

Enforcing this rule when posting a view or SmartForm step would be overly restrictive. The PI and Staff should be able to complete the forms even if their training is incomplete. The Rule is that the PI cannot submit the protocol for review until all have met the training requirements, so a View validation won’t work. What is needed is Project level validation which can be accomplished by overriding the Project.validate() method.

So this means that some rules are implemented in a View Validation script using the web-based configuration tools, while other rules are implemented using Entity Manager. This approach definitely works and is seen in a large number of sites. The downside is that code is maintained in different places and script is required for all rules.

In addition, all the scripting approaches It doesn’t take into consideration that many rules follow common patterns. For example,

• If field A has value X, then a response for field B is required, or
• Field A must match a specific pattern such as minimum or maximum length

What if these patterns could be formalized into a standard way of defining a validation rule? What if all rules were defined in the same way and in the same place? Would that make implementation, testing and maintenance easier? I certainly think so.

Introducing the ProjectValidationRule implemenation

ProjectValidationRule is a Selection Custom Data Type whose sole purpose is to serve as a central place to define and execute your site’s validation rules. Some of you may have previous experience with a CDT named “SYS_Validation Rules” and any similarities you see are not a coincidence. That type provided the seeds from which this new implementation was grown. It allows for rules following common patterns to be defined without authoring any additional script and provides the flexibility to define whether the rule is to be enforced at a View or Project Level. As an added bonus, you can also define activity specific rules.

Download the ProjectValidationRule package for specific implementation details. This approach is still evolving and it’s proven very effective so far but there is always room for improvement so feedback is always appreciated.

Cheers!

Technology Preview: Multi-Solution Development in Extranet 5.6

Extranet 5.6 includes an early peek at what I expect will become an important tool for those of you who have implemented multiple solutions in your site. It’s called “Module and Solution Support” and its goal is to allow for the independent development and release of configuration updates between the different solutions you are maintaining.

Before you get too excited, it’s important to realize that as promising as this feature is, it’s not a panacea. There are many methodologies and project management techniques to deal with your constantly evolving workflow solutions and, while this enhancement adds another tool to your toolbox, it doesn’t meet every need you could imagine. What it does provide, however, is a big step toward being able to manage different solutions on different development schedules.

Enabling this option is a one way trip so it’s best to first explore this new feature in an isolated experimental development environment. If you…

• are already familiar with using Process Studio to manage the development of your site,
• have more than one solution deployed (such as IRB, IACUC, COI, etc.), and
• face the challenge of wanting to deploy updates to the different solutions on different schedules,

then Module and Solution Support is worth a look. I’m currently using this feature on one of my projects and will update you all on my experience in a later post.

Module and Solution Support is provided as a technology preview with Extranet 5.6 and is only one of many cool new features. Start planning for your upgrade today using the Extranet 5.6 Pre-Installation Package. If you want to know more about how to upgrade, drop me an email, and I’ll fill you in on the details. To accelerate your upgrade, check out our new Extranet 5.6 Upgrade Service.

Cheers!

The SDLC, Part 3 – Common pitfalls when applying configuration updates

You’ve followed the recipe to the letter…only to discover you’re out of propane for the barbecue.

You brush and floss just like you’re supposed to….but the dentist tells you you have a cavity anyway.

You’ve conducted your workflow development with as much rigor and care as humanly possible…but your configuration update fails to apply.

Sometimes things just don’t go your way. This is true with all things and, when it happens, it often leaves you scratching your head. When it comes to failed Configuration Updates, it’s sometimes difficult to figure out what went wrong, but there are some common pitfalls that affect everyone eventually. I’ll discuss a few of the more common ones with the hope that you are one of the fortunate ones who can avoid pain by learning from the experiences of others.

Pitfall #1: Developing directly on Production

The whole premise of the SDLC is that development only takes place in the development environment, and no where else. While this sounds simple, it’s a policy frequently broken. Workflow configuration is a rich network of related objects. Every time you define a relationship through a property that is either an entity reference or a set, you extend the “object network”. In fact, your entire configuration is really an extension of the core object network provided by the Extranet product.

The Extranet platform is designed from the ground up to manage this network, but its world is scoped to a single instance of a store. The runtime is not, and in the case of the SDLC, should not be aware of other environments. This means that it can make no assumptions about objects that exist in the store to which the configuration update is applied. It must assume that the object network in development reflects the object network in staging and production. Because of this, it’s trivially easy to violate this assumption by configuring workflow directly on production. If that’s done, all assumptions that the state of production reflects the state of development at the time the current round of development began are incorrect and the Configuration Update is likely to fail.

Errors in the Patch Log that indicate you may be a victim of this pitfall will often refer to an inability to find entities or establish references.

One common cause for such an error is when you add a user to the development store but there is no corresponding user with the same user ID on production. Some objects include a reference to their owner. In the case of Saved Searches, for example, the owner will be the developer that created the saved search. In order to successfully install the new saved search on the target store, that same user must also exist there.

Troubleshooting this type of problem is tedious and sometimes tricky because it’s often necessary to unravel a portion of the object network. It’s a good idea to do whatever you can to avoid the problem in the first place when you can.

Bottom Line: Only implement your workflow on the development store and make sure that all developers have user accounts on development and production (TIP: You don’t need to make them Site Managers on production).

Pitfall #2: Not applying the update as a Site Manager

If your update fails to apply and you see a message that has this in the log entry:

Only the room owner(s) can edit this information.

you are probably not specifying the credentials for a site manager account when Applying the Update.

This can happen when a user is not provided in the Apply Update command via the Administration Manager or if the provided user is not a Site Manager. The installation of new or updated Page Templates causes edit permissions checks for each component on the page template and unless the active user is a Site Manager those checks will likely fail.

Bottom Line: Always specify a site manager user when applying a Configuration Update. Technically this isn’t always required depending upon the contents of the Configuration Update, but it’s easy to do so make a habit of doing it every time.

----

More pitfall avoidance tips next time….

Cheers!

The SDLC Part 2 – Process Studio and Source Control

 

Last time I introduced the notion of the recommended Software development Lifecycle (SDLC). Now it’s time to get a bit more specific. As mentioned last time, the best way to support a disciplined development process is to make use of three distinct environments: Development, Staging, and Production. Each environment can either be made up of a single or multiple servers. While there is no requirement that each environment be like the others, it is recommended that your staging environment match production as closely as possible so that experience gained from testing your site on staging will reflect the experience your users will have on the production system. It’s also useful because this will best enable you to use your staging server(s) as a warm-spare in case of catastrophic failure of the production site.

Further Reading…. FAQ: Everything You Wanted to Know about Source Control Integration But Were Afraid to Ask HOWTO: Apply Large Configuration Updates

To go into everything you can do when configuring and implementing your workflow processes would take more time than I have here and there are several good articles and online reference guides available in the knowledgebase. We also offer both introductory and advanced training courses. Instead I’ll focus on how to manage the development process.

A key principle of the SDLC is that development only takes place in the development environment and not on staging or production. The work you do in the development environment gets moved to staging so it can be tested through a configuration update. A configuration update is a zip file that includes the full set of changes made during development that need to be tested then deployed to production. In order to accurately identify the changes that should be built into the configuration update, each individual change is versioned in a repository using Microsoft Visual Source Safe.

Making a change or enhancement to workflow configuration begins by checking out the elements from the configuration repository using a tool called Process Studio. Once checked out, development takes place using the web based tools, Entity Manager, or Site Designer. Before you should consider the change complete, the changes are tested locally on the development server. If everything works as expected, the changes are checked back into source control using Process Studio. This process repeats itself for all changes.

When the development of all intended fixes/enhancements is complete, it’s time to put them to the test. While developers are expected to test their changes in the development environment before they are checked into source control, official testing should never be done on development. The reason for this is that the development environment is not a good approximation for production. Developers, in the course of their work, make changes to data and environment that make it hard to be able to use the test results as a predictor of how the changes will work on production. Instead, a configuration update is created using Process Studio so it can be applied to staging for official testing. Before applying the update to Staging, it’s a good idea to refresh the staging environment from the most recent production backup. This gives you the best chance of understanding how the changes will behave on production.

If issues are discovered during testing on staging, the process is reset.

1. The issues are fixed in development (check-out, fix, check-in),
2. A new configuration update is built,
3. Staging is restored from a production backup,
4. The configuration update is applied to staging
5. The changes are tested

If all the tests pass, the exact same configuration update that was last applied to staging is then applied to production. Though not required, it’s a good idea at this point to refresh your development environment with a new backup of production. The closer the development environment is to the real thing, the fewer issues you’ll have going forward.

At this point, development can begin on the next set of features, fixes, and enhancements. And the cycle repeats…

To learn more about the role of source control in your development lifecycle, please read the following article:

FAQ: Everything You Wanted to Know about Source Control Integration But Were Afraid to Ask

That article does an excellent job describing all core principles and processes. Of course, not everything goes as planned as you apply updates to your staging and production sites. Next time, I’ll discuss some common challenges and how to troubleshoot when issues do arise.

Cheers!

The Software Development Lifecycle – Part 1

Well, it was inevitable. My goal of posting at least weekly to this blog is being threatened. It’s been over a week since my last post so it’s time to pick it back up again.

This week at Click was certainly a busy one and it made me realize that it’s time for a refresher on our recommended Software Development Life Cycle (SDLC). All software development follows a repeated cycle, sort of like the “Wet Hair, Lather, Rinse, Repeat” instructions on your shampoo bottle – simple, but effective. Generally speaking, software development follows  a simple cycle as well:

Define –> Design –> Implement –> Test –> Deploy –> Repeat

This is true no matter the technology or tools. Working with Click Commerce Extranet base solutions is no different. Putting the cycle into practice requires discipline, familiarity with the development tools, and an ability to troubleshoot problems when they arise. Over the next couple of posts, I’ll be describing the Click Research and Healthcare SDLC. Along the way, I’ll highlight common problems and how to address them. Hopefully this will lead us to a discussion on how best to handle the concurrent development of multiple solutions, which is the topic of a panel discussion I’ll be hosting at the upcoming C3 conference. So…let’s get started.

Three Environments
To effectively practice the SDLC, three environments are required:

1. Development
   This is where all active development takes place. Developers typically will work together in this environment, benefitting from and leveraging each other’s work. All work is formally versioned through the use of Source Control integration via a tool called Process Studio. We’ll be discussing the use of Process Studio in more detail a bit later. This is the only environment where development should place.
   
2. Staging (Test)
   This environment is ideally a mirror image the production environment and is used as a place to test completed development before it is deemed ready for production use. It also can serve as a warm standby environment just in case there are issues with the production site that can’t immediately be resolved.
   
3. Production
   This is the live system and the only site end users will use to perform their daily tasks.

Work performed in the development server is packaged up into what’s called a Configuration Update which can then be applied to Staging, where it is tested, and, if all the tests pass, to Production. For more information on what is included in a Configuration Update, check out the following Knowledgebase Article:

INFO: Understanding Configuration Updates in Click Commerce Extranet

Next time, we’ll talk about how configuration updates are built and special things to consider in order to make sure they can be correctly applied.