Implementing a Hybrid CDT - UPDATE

This is a follow-up to my earlier post on implementing Hybrid CDTs.  If you haven’t yet, I encourage you to read that post first or very little of this will make sense.

I’ve been pleased to hear from those of you who have taken this approach and identified areas within your own configuration where it provides value. Some of you have asked about what happens when an entity is deleted and I thought I’d follow up with some additional detail as I didn’t directly address that case in the original post.

The deletion case is relatively simple to address but does require another trip to Entity Manager.  Every eType has a built-in method called @unregister and Custom Data Types are no exception. This method is called whenever an entity is being deleted from the site. For the deletion to have real-time effect across all the references to the associated Selection CDT entity, you will also need to delete the corresponding Selection CDT entity. This is done by implementing logic into the @unregister method of the Data Entry CDT type which also unregisters the associated Selection CDT entity.

In the example described in the original post, you would implement an @unregister method on the MyChildDE type that looks like the following:

function unregister()
{
try {
        // the supertype @unregister is called by default by the framework
        
        // If the data entry entity (this) is unregistered, then also unregister the 
        // referenced Selection CDT
        var child = this.getQualifiedAttribute("customAttributes.child");
        if (child  != null) {
            // This is the complete solution, but actually unregistering the selection CDT entity at this
            // time might have an adverse performance impact. If that is the case, it might be better to simply
            // remove the entity from all sets.
            child.unregisterEntity();
            
            // If, in your data model, MyChildSE entities are only referenced as members of sets, you can
            // improve performance by limiting the work done at this time to the removal if the
            // child entity from all sets and defer the remaining deletion until the next time Garbage Collect is
            // run.
            // removeFromAllSets(child);

            // Even more performant would be to explicitly remove all known uses of the child entity but this
            // approach requires ongoing maintenance as it needs to stay in synch with wherever the Selection
            // CDT can be used.
        }
    }
    catch (e) {
        wom.log("EXCEPTION _MyChildDE.unregister: " + e.description);
        throw(e);
    }
}

With this extra bit of code, the effect of a user deleting the data entry entity is immediately obvious in that any reference to the associated selection entity is also removed.

Cheers!

Implementing a “Hybrid CDT”

I’m beginning to think I should change the tagline for my blog from weekly to monthly. One of our Project Managers has been giving me a hard time about my blog getting a bit stale. I told her I was waiting for her to create a post to her (currently non-existent) blog and she only laughed. Where’s the equity in that?!? ;-)

Life continues to be extremely busy here at Click (the real reason it’s been so long since my last post) but good things are happening. Extranet 5.6 is now officially released (Yay!) and exciting work is being done in all groups. My work on our new Animal Operations solution is progressing and I’m excited to see it all come together later this year. Within Professional Services, we’ve been working to drive consistency in our approach to both projects and development, including a major revision to our naming standards and coding guidelines. I hope to make that available to customers very soon so you have the opportunity to adopt the same standards as we have.

Today, I want to talk about an approach to solving the thorny problem of being able to select CDT entities that were previously created as part of the same project. Now I won’t be the first to solve this problem, but the solutions I’ve heard about involve the creation of custom layouts using Site Designer or clever hacks to the standard view controls. Neither of those approaches appealed to me so I set out to come up with an approach that could be done through simple configuration.

Selection versus Data Entry Custom Data Types

Before I go into the technique, it’s important to understand why this is a problem to begin with. To do that, one must understand the difference between Selection and Data Entry custom data types. Selection types serve the purpose of providing a data source that serves as a list of choices. Data Entry custom data types serve as an extension to something else and is not allowed to be referenced by projects or custom data types other than the type it is extending. The distinction is important to the base Framework so that data lifetime, security, and presentation can be effectively managed.

• Data Lifetime
  By knowing that an entity belongs to a Data Entry CDT, the base application knows that it is owned by only one project, person or organization thus, if the project is deleted or a reference to the entity is removed, that entity can also be deleted. Selection CDT Entities on the other hand are intended to be referenced by multiple entities so do not get deleted when all references are removed.
  
• Security
  Since the a Data Entry entity belongs to a single project, it is subject to the same security rules as the project itself. Selection CDT Entities have no such allegiance to a single project and can be referred to by many projects, or none at all. Their purpose is to serve as the contents of selection lists so are visible to all users.
  
• Presentation
  How references or sets of CDT entities are presented differs depending on the CDT being Selection or Data Entry. A Selection CDT entity can only be selected by the end user, never created, modified, or deleted. Data Entry CDT entities are intended to serve as an extension of the data comprising a project, person, or organization so, by their very nature, can be created, edited, and deleted.

So what happens when you need both characteristics in a single set of data?

Implementing a “Hybrid CDT”

You won’t find the term Hybrid CDT anywhere in the product or product documentation. That’s because i just made it up ;-) In fact, the term is a bit misleading in that it makes you think there is a single CDT when, as you’ll see, there are really two. But, conceptually the two types serve a single purpose . I’d gladly consider other name suggestions but, for now, I’m going to use the term out of sheer convenience.

The goal is to define a “type” that can be used for both data entry and selection.

Step 1: The Basic Setup

We’ll need to create two Custom Data Types. One that is a Selection CDT, and the other is a Data Entry CDT.

The selection CDT will define all the attributes you need to hold all the specific detail that needs to be captured. We’ll keep this example simple by only adding a single string attribute called “name”, but any number of attributes could be added.

MyHybridCustomDataTypeSE

• name String

When creating this type, be careful to not select the option to source control the data (entities) as it is our goal to allow the end user to create the entities so that they can be used in selection lists later.

Next, we’ll define a Data Entry Custom Data Type. The only attribute on this type will be a reference to the selection type we just created.

MyHybridCustomDataTypeDE

• data (MyHybridCustomDataTypeSE)

By creating the types in this way, we’ve effectively created a Selection CDT that is “wrapped” by a Data Entry CDT.

Step 2: Creating the Data Entry Form

Now that types are defined, the next step is to create the data entry view on the data entry CDT. To make this all work we want to expose the attributes on the selection CDT into a view on the data entry CDT. With our simple data model, this means the view will only have a single field on it:

“customAttributes.data.customAttributes.name”

Of course, in the view editor, you’ll see the attribute in the tree as

data

-> name

It’s not critical to define a view on the selection CDT but you could if you wanted to tailor the appearance of the Chooser for this type.

Step 3: Insuring proper data lifetime

As mentioned earlier, a Selection CDT entity will not automatically be deleted when there are no other entities referring to it. In most situations, this is exactly what we want because the data is intended to be available for selection across any number of project types and custom data types. In the Hybrid CDT example, however, our intent is that this data is specific to the entity that refers to it and it should be deleted when the referring entity is deleted. There are a few ways to do this, but only one way to do it without having to write any code.

1. Launch Entity Manager and connect to your development store
2. Open the details of the selection CDT you just created
3. At the bottom of the details display you will see a checkbox that allows you to indicate if the extent is “weak”. In this case we want it to be so check the box
4. Save your changes

Following these steps will cause any unreferenced entities to be deleted the next time Garbage Collection is run.

Step 4: Putting the “Hybrid CDT” to use

Believe it or not, we now have a “Hybrid CDT”. So now we get to put it to use on a Project. To add color to this, we’ll work through a simple example. We’ll define a project type to manage trivial information about your children. Let’s call it MyChildren. It will allow a user to specify the names and ages of their children and then answer a couple of simple questions requiring them to select one of the kids in response. So, here’s the setup:

MyChildSE (the selection CDT representing a single child)

• name (string)
• age (integer)

MyChildDE (the data entry CDT representing a single child)

• child (a reference to the MyChildSE entity containing information about the child)

MyChildren (the project type)

• children (Set of MyChildDE)
• childMostLikelyToBeARockStar (entity of MyChildSE)
• childrenWithAllergies (Set of MyChildSE)

It’s pretty simple and any real world example will likely be far more complex. The complexity will be in the volume of data, not the structure so hopefully this will clearly demonstrate the technique. I’m also going to overlook that case where there is only one child in which case the questions don’t make much sense (such is the price of a contrived simple example).

Now that the types are defined, we can add the fields to views on the project which can then be used in a SmartForm. The first view will be constructed to allow the user to specify their children. All this requires is to add the children attribute and configure the view control just like you do for any other data entry CDT.

The second view will be used to prompt the user for the following information:

• Which child is most likely to become a Rock Star?
  • This will be done by adding the childMostLikelyToBeARockStar attribute to the form and configuring the view control like you would any other reference to a Selection CDT.
• Which children have allergies?
  • This will be done by adding the childrenWithAllergies attribute to the form and configuring the view control like you would any other set of Selection CDT entities.

Simple, right? There’s really only one step left and that is to make sure that the list of children presented for selection are just those added on the current project. If this isn’t done, then any child created on any project would be presented and that wouldn’t make much sense. This is accomplished through the use of a new feature in Extranet 5.6: The “Data Bind Override” feature of the View Controls for the selection data type reference and set. You will add this script to the controls for both questions by clicking on the script icon following “Data Bind Override” in the view control properties form

// Return a set of MyChildSE entities that are derived from the
// set of MyChildDE entities in the children set
var browseDataSet = rootEntity.getQualifiedAttribute("customAttributes.children");
if (browseDataSet != null) {
  browseDataSet = browseDataSet.dereference("customAttributes.child");
}

These two views can either be used individually or as part of a SmartForm. You will just need to make sure that the children are added before the additional questions are asked or the choice list will be empty.

All Done! We have now defined a Hybrid CDT and put it to use. If you found this to be of value in your own work, please drop me a note. I’d love to hear how it worked for you.

Cheers!




UPDATE: I've posted a follow-up to this post to address what happens when data is deleted by the user: Implementing a Hybrid CDT – UPDATE

UPDATE 2: Another follow-up post to address what happens when the Hybrid CDT is cloned: Hybrid CDTs and Cloning

Modeling a multiple choice question

Sometimes the fact that a problem can be solved in many ways isn’t a good thing as it forces you to weigh your options when only one approach encapsulates current best practices. One such case is how to model a multiple choice question. You have a need to allow the user to select from a list of items. Your example if “safety equipment” but it could really be anything. In your question, you pose two possible approaches and are asking which is the preferred implementation:

1. A separate Boolean fields for each possible choice, or
2. A Selection CDT and an attribute that is a Set of that CDT

While you could make both approaches work, there are significant advantages to using a Selection CDT that is populated with data representing all possible choices rather than separate Boolean fields.

I’ll use an example to demonstrate. Suppose you want to have the researcher specify which types of safety equipment will be used and the types to choose from are gloves, lab coats, and eyeglasses (I’ll intentionally keep the list short for this example but it would obviously be longer in real life)*

In option 1, you would define the following Boolean properties:

• gloves
• labCoat
• eyeglasses

You could either define them directly on the Project, or, more likely, create a Data Entry CDT called “ProtectiveEquipment” and define the attributes there. Then you would define an attribute on Protocol named “protectiveEquipment” which is an entity reference to the ProtectiveEquipment CDT.  Once the data model is defined, you can add the fields for each property to your view.

It’s pretty straightforward, but not the path I would recommend. The reason I say this is that by modeling in this way, you would embed the list of choices directly into the data model which means that if the choice list changes, you would have to update the data model and anything dependant upon it. Embedding data into the data model itself should really be avoided if at all possible.

The same functional requirements could be met by option 2. With this approach, you would define a Selection CDT named “ProtectiveEquipment” and add custom attributes to it that may look something like this:

• name (String) – this will hold the descriptive name of the specific piece of safety equipment. Recall, that you automatically get an attribute named “ID” which could be used to hold a short unique name (or just keep the system assigned ID)
• You could add other attributes if there was more information you wanted to associate with the type of equipment, such as cost, weight, etc.

Then, on the Data Tab in the Custom Data type editor for the ProtectiveEquipment CDT, you can add all the possible types of equipment. There will be one entity for each type meaning, for this example, one each for gloves, lab coat, and eyeglasses.

The last step in completing the data model would be to then add an attribute to your protocol named “protectiveEquipment”. This attribute would be a Set of ProtectiveEquipment so it can hold multiple selections.

Next you can add the protectiveEquipment attribute to your Project view. In doing this, you have a couple of options for how the control is presented to the user. You can specify you want it to display as a Check Box list, in which case the user would see a list of checkbox items, one per ProtectiveEquipment entity, or you could use a chooser, in which case the user would be presented with an “Add” button and they could select the equipment from a popup chooser. If the number of choices is small (less than 10, for example) the checkbox approach works well. If the number of different protectiveEquipment types can get large, you’re better off going the way of the chooser. Both visual representations accomplish the same thing in the end but the user experience differs.

So why is option 2 better?

The list of choices can be altered without having to modify the type definition. You have the option of versioning the list of entities in source control so that they are delivered as part of a patch to your production system or only versioning the type definition and allowing the list to be administered directly on production.

1. Views will not have to change as the list of choices change
2. Code does not have to change as the list of choices change. Your code avoids the need to reference properties that (in my opinion) are really data rather than “schema”. This is critical because if the list of choices change, you won’t have code that has to change as well. Instead your code will simply reference the custom attributes
3. You have the ability to manage additional information for each choice which you may or may not need now, but is easy to add because you’re starting with the right data model.

(Note: These reasons are also why I would recommend against defining properties of type “list”)

I hope this explanation is clear. If not, please let me know.

Let’s say, for the sake of argument, that you had the additional requirement that the user must not only say they will use eyeglasses but also have to specify how many they will use. This is easily accomplished but changes the recommended approach a bit. To support that requirement, you would set up the following data model.

ProtectiveEquipment (Selection CDT)

• Name (string)

ProtectiveEquipmentUsedOnProtocol (Data Entry CDT)

• EquipmentType (Entity of ProtectiveEquipment)
• Count (integer)

Protocol

• protectiveEquipmentUsed (set of ProtectiveEquipmentUsedOnProtocol)

You would then add the Protocol attribute protectiveEquipmentUsed to your view. When rendered the user will be presented with a control that includes an Add button. When the user clicks that button, a popup window will be displayed that prompts the user for the equipment type (which can either be a drop down list, radio button list, or chooser field) and count. You can define a view for the ProtectiveEquipmentUsedOnProtocol to make the form look nice since the default system generated form is kind of bland.

I hope this helps. Let me know if you’d like me to clarify anything.

Cheers!

* Thanks to The Ohio State University for the example

Project or Custom Data Type? It's a tough decision sometimes

I'm sure many of you have first hand accounts of how the flexibility of the Extranet platform has enabled you to do things that would be very difficult in other, more rigid environments. But, as I often quote, "With great power comes great responsiblity." The fact that there are so many options to solve a problem within the product also means that choices have to be made. How do you know what choice is the best one? What are the advantages and disadvantages between two seemingly good choices? It's not always easy to know. The Services team is able to rely on the experience of having deployed many solutions so we're in a unique position to assist you in your design and implementation efforts, but we recognize that your ability to nurture and evolve your own applications is essential to your success as well. This means you have to make choices that are occasionally difficult.

Once such choice is how best to model your information. As I blogged earlier, implementing a good data model makes nearly everything easier. Sometimes, however, the correct choice isn't always clear. A good example of that is when to use Sub-Projects instead of Custom Data Types. Projects and Custom Data Types (CDTs) are both viable ways to segment your data model. When modeling information for a Project or Activities, CDTs are the fundamental means of "normalizing" your model, creating one to many relationships, and referring to items from selection lists. The data maintained in a CDT can often be entered using simple forms that are natively supported in the broader context of a project.

Projects typically represent distinct workflow processes, IRB Study, IACUC Protocol, Funding Proposal, etc. These processes involve the use of SmartForms, Workflow States, Pre-defined User Actions (Activities), review capabilities and workspaces. Custom Data Types are used to provide a structure to organize the data managed through the workflow process. Sub-Projects are simply Projects that represent processes related, but separate, to another process. Amendments, Continuing Reviews, and Adverse Events all fall into that category.

The definitions seem clear, right? So why would you ever consider using a Project instead of a CDT? The simple answer is when the needs of the process by which the information you would have modeled in the CDT require the use of features provided by Projects. One example of this is when data collection is best accomplished through a SmartForm with conditional branching. Projects natively support this feature so it makes sense to take advantage of being able to configure a SmartForm for the subset of information that you would have otherwise modeled into a CDT. Both CDTs and Projects offer the same flexibility in terms of being able to define your own data model.

Choosing to use a Project over a CDT should only be done when the needs exceed the simplicity of a CDT. If the data is modeled as a project, there is extra work in configuration because you have to address the configuration or avoidance of all the features a project provides. When using it to model complex data extensions for the purpose of being able to use a SmartForm, you also need to make choices about how to configure the use of the "sub-data". Decisions have to be made about whether or not to use a workspace, whether or not there is any workflow, what are the security rules, how to handle validation, etc. Achieving desired results take a little bit of planning but it's good to know you have options.

"With great power comes great responsibility....and through responsible decisions, you can build powerful applications."

...and we're here to be your guide when you need one.

Cheers!

It all starts with a good data model...

Yesterday I was asked to assist in the development of a custom display that was proving difficult. By all accounts, the display should have been simple to build. It was basically a two tier report with the following structure:

• 1. Top Level Entity
  • 1.1 Child Entity
  • 1.2 Child Entity
• 2. Top Level Entity
  • 2.1 Child Entity
  • 2.2 Child Entity

When I saw the desired format, I imagined a data model that represented a Set of Sets. The first set being the set of top level entities. Each of those entities, would have an attribute that is a set of the Child Entities. A model like this lends itself very well to representing a 2 tier tree and it's easy to imagine how additional tiers would be added.

Instead, what I found was a model where an entity type held a attribute for the top level set and a different set attribute for each possible value of the top level set. Bear with me as I try toclear up the confusion I most likely have created. Suppose that we have a set of food groups (Vegetable, Fruit, Meat, Grain). for each food group, we also want to record examples of foods

• Vegetable
  • Carrots
  • Peas
• Fruit
  • Apple
  • Banana
  • Orange
• Meat
  
  • Chicken
  • Beef
• Grain
  
  • Wheat
  • Rice

There are really two acceptable ways to model this.

Set of Sets

MyEntityType

• foodGroups (Set of FoodGroup)
  

FoodGroup

• name (String)
  
• foods (Set of Food)
  

Single Set with Reference To grouping type

MyEntityType

• FoodItems (Set of Food)
  

Food

• name (String)
  
• foodGroup (entity of FoodGroup)
  

Both approaches allow for a direct navigation of the the data model. In the real life example, I saw a model that resembles this:

MyEntityType

• foodGroups (set of FoodGroup)
  
• meats (Set of Meat)
  
• vegetables (Set of Vegetable)
  
• grains (Set of Grain)
  
• fruits (Set of Fruit)
  

To make matters even more challenging, the types Meat, Vegetable, Grain, and Fruit were similar in structure but not exactly the same as one another. While I can see how this might make sense in certain circumstances, it is a difficult model to work with because there is no uniformity. Another concern is that there are attributes and types that model the possible data values of FoodGroup. What happens if another food group is added? The unfortunate truth is that in order to add another entity of FoodGroup, the model itself would need to be updated.

Even if it was known that no additional food groups were ever going to be added to the list, using this model to build the display requires convoluted logic.

All of this is meant to reinforce the point that data models matter. A good data model makes everything easier. Views, Custom Searches, custom logic are all easier to implement and maintain which reduces the overall cost of ownership. Putting time into a solid data model is a wise investment.

I hope I haven't confused anyone with this post. Pictures would certainly help illustrate my point and one day I might find the time to include them. Also, if you see any similarities in your own data model, it's purely coincidental ;-)  

Cheers!