Tag Archives: API

Working with the SKY Query API in Chimpegration

Beginner, Not Code, , ,

Blackbaud recently added the SKY Query API for Raiser’s Edge and Financial Edge. At first I was not really sure how this would be of benefit to our products. We have worked with Lists in Raiser’s Edge and on the database view we have generated static queries of records that have been processed. I never thought that we had a need for adding criteria to queries.

Now that I have seen the new Query API, I have been inspired.

How have we got around the lack of a query API so far?

In Chimpegration we push data from Raiser’s Edge to Mailchimp. In order to decide which records to push, we let the user select an NXT list that has previously been created. There are some issues with this.

  • Firstly, the list selection criteria is limited. The user cannot, for example, specify that a constituent with no email address or a blank email address should be ignored.
  • The list functionality does not allow you to choose specific output fields. We offer a limited range of fields that we think that the user may want to export.
  • The lists are static. If you want to update them, you can, but this prevents data being pushed to Mailchimp according to a schedule. (There are some workaround involving Queue but these are awkward).

How does the Query API solve this?

The query API allows you to programmatically list all queries and to load one of them in particular. You can then run the query and fetch the results. This makes it fully dynamic. Scheduled data exports to Mailchimp would be up to date with the latest information.

The user can choose the output data. Whereas previously they have been limited to the areas we offer the user, now they can choose any value available to them in a query. Most organisations won’t want to push a membership attribute or a gift notepad to Mailchimp but there is bound to be one out there that wants to do something like that or some other option that we had not considered. With the full range of output fields they are no longer restricted to what we offer them.

The same goes for criteria. Previously the user was restricted to the fields available in lists. Now the whole range of query fields can be a part of the criteria. If an organisation only wants to export constituents attending a specific event with a large t-shirt size, now they can!

What else can we do with Query API?

In Chimpegration Classic and in Importacular we look up constituents with criteria sets. It has been much simpler to look up constituents with a wide range of criteria. There is some scope for this based on the constituent list API endpoint. However, the Query API really gives this some muscle.

At first I thought that, even if we could do it, it would not be practical to create a query each time the user wanted to use criteria to search. We would end up creating a lot of queries saved in Raiser’s Edge. It is not certain that the user would have rights to delete a query. It just felt wrong.

However, the Query API includes the ability to generate a query on the fly. When including the filter, output fields and sort values in the json payload, the query can be run without explicitly saving it to the organisation’s environment. This is a real game changer.

The SKY API documentation does suggest that this should not be used instead of the constituent list and constituent search endpoints as they are optimised for search. However being able to search on more obscure areas of Raiser’s Edge certainly adds a lot of power to look up records that was previously missing.

When are we releasing the new Chimpegration functionality?

Update: This has now been released!

We are actively developing this functionality. However the Query API is still in preview so it is uncertain when this will be released. We may release it also with the caveat that our functionality is in preview and may break at any time (due to changes in the Query API). This is a matter of a few weeks though so watch this space for a demo!

GDPR and Consent in Raiser’s Edge

Beginner, , , ,

I have been really busy of late. While many an EU non-profit have been kept awake at night because of GDPR, I have not quite been kept awake but nevertheless been very involved in implementing GDPR into our products. This has taken the form of the new consent module.

If you are not in the EU or otherwise not on the latest version of RE7 then you may wonder what I am talking about when I mention the new consent module. I am not going to go into too much detail here as Blackbaud have some good resources that cover it here.

What I will say is that the implementation of consent is very different from many other modules in RE. Luckily there is less and less scope for RE7 API developers as Blackbaud moves towards NXT and expands the REST based SKY API. So I am wondering if as, a last challenge towards those remaining in the RE7 API game (myself included), Blackbaud decided to make the new consent module even less consistent than previous modules.

Here are a few of its features:

  • The consent collection cannot be found within the regular BBREAPI assembly. You have to look elsewhere for it.
  • It does not save alongside the rest of the constituent records but has its own save routines. What this also means is that the VBA events are not fired when a consent record is saved.
  • In the first release it does not cause an exception when you do not supply a valid combination of channel and category as it does in the UI but if you are really clever (or decipher the sample code), you can determine how to do your own validation.

Now I should not be too harsh on the BB developers. Introducing a new module like this is extremely difficult. There are so many intertwined areas that must be accounted for and I am sure that the design decisions were taken for a reason. (One of which being that it is much easier this way to work with very many consent records if they are a standalone entity)

How are we updating our applications to work with the consent module?

Audit Trail:

As you would expect changes made on consent records will be tracked but because consent records are saved as a standalone entity they will only be saved if the constituent record is also saved afterwards.

Validatrix:

This is a tricky one. We have included consent records as part of Validatrix but because they are a standalone entity and are open and saved in their own rights, they do not fire the VBA events that tell Validatrix to prevent a save. That means that a user can add a consent record and shut the constituent without saving the constituent. You cannot therefore have a consent record as the primary criteria. However, if you have the consent record as a dependency of a constituent based field then it will be included in the criteria when you save the constituent.

Importacular:

As you would expect, Importacular allows you to import consent records. You can match on any combination of channel, category, date, response and source to ensure that you are not creating duplicate consent records (although by default it matches on channel, category, date and response).

Chimpegration:

This is perhaps our most ambitious development. Until Blackbaud add consent information to query and export you are not able to export consent records to MailChimp. However it is probably more useful to export the outcome of the consent records i.e. solicit codes which show you a good picture of a constituent’s intentions.

On managing campaigns you can add a consent record based on the action i.e. if a subscriber unsubscribes you may want to add a consent record.

Sync is where the most complex piece of development occurs. We allow you to map individual groups and group items to the addition of different types of consent records. Equally when specific solicit codes are added (in response to consent records being added previously) these can be mapped to group items. We have a longer description of this on our knowledgebase.

When is this available? Importacular, Audit Trail and Validatrix are already live. Chimpegration is live for self-hosted and will go live in the near future for hosted organisations.

Moving over to SKY API

Beginner, Not Code, Opinion, , ,

This blog has mainly consisted of the COM based RE7 API and my musings of all things Blackbaud related. I am sure that the latter will no doubt continue but I have realised for a while now that as time goes on the RE7 API is becoming less and less relevant (although not entirely so) and that there is a natural progression towards the SKY API.

In general there is probably less to be said about SKY API. Firstly Blackbaud have been doing a much better job at documenting it than they ever did with the RE7 API. They are also putting a lot more thought into it so that there are far fewer inconsistencies (so far at least) than there ever were with the RE7 API. Where there are difficult, new, areas they have written up good documentation or blog posts to explain. In short they have made my job here somewhat redundant… Well thanks a lot Blackbaud!

Actually, yes, thank you. I would much rather a clean usable API than one where I have to write up blog posts explaining how to do things. I am sure that there will be moments but there will quite possibly be fewer of them.

One topic that I think deserves some discussion though is the porting of existing functionality from RE7 to SKY. Those of us that have products written for RE7 are keen to see the functionality available on SKY in order that we can port the solutions over.

I have been very keen to transfer Chimpegration but one of the stumbling blocks has been the lack of bulk data processing on SKY. Specifically the ability to return filtered lists of constituents. On many platforms this means simply specifying a last data changed or a keyword search. On RE7 though it is possible to make use of a query to retrieve that information. The user would themselves set up the criteria in the query and the application would allow the user to select that query.

There is some discussion on the SKY API forums about how imperative it is that this be ported over to SKY and that we should be allowed to once again select an existing query.

Despite really needing this functionality for Chimpegration, I am not convinced that this is the best course of action for SKY. This new API should embrace a general approach to this problem. It cannot be based on RE7’s query module. There is definitely a need to generate lists based on complex filters and criteria and that should be exposed somehow to the developer community but to simply port the existing RE7 functionality to SKY would be short-sighted and not take into consideration the other applications what will one day make use of the same API. I want an API that will work with RE, BBCRM, ETap and others. To simply port queries to SKY would confuse the issue and make for an API that is not consistent.

Our MailChimp API Headache

Beginner, ,

It has been a while since I last posted on this blog. The main reason is that we have been so very busy for working and struggling with version 3 of MailChimp’s API.

Around two years ago we became aware of MailChimp’s latest API. It was certainly a very good example of a clean REST based API having all the characteristics of a well designed interface. All of the structures made sense and the anybody used to working with REST based APIs would have no difficulty in picking up this new API and creating a new powerful application with it.

So what went wrong?

About 18 months ago MailChimp informed the community of developers that it would be retiring all their other APIs in a year’s time so that they could concentrate on v3 as being the sole API. This meant shutting down not only V2 and older instances but also the export API.

This was a really big deal for us as Chimpegration made solid use of V2 and definitely the export API.  When we first gave MailChimp a demo of Chimpegration they told us that it was one of the most complex and intricate applications making use of their API. They were blown away by the detail available in synchronizing and the level of user choice that made up the application. What is more many of the features that exist today were not available at that time including the synchronization of groups, use of profiles and the ability to remove subscribers based on a query.

The news that MailChimp were going to shut down their API came as a shock to us. It meant some very big changes in Chimpegration without much to show for it. How can you possibly sell the fact that Chimpegration now uses V3 instead of V2. Nobody cares as long as the application carries on working.

What was clear to us when we started working with V3 was that those who designed this new API really had not spoken to anybody else about it. It was as if they had been working in a isolated area of the building, kept away from any V2ers or for that matter, existing users of V2.

They changed the structure of areas of the API such as groups. (The ids of V2 groups no longer match the ids of V3 making it very difficult to transfer to the new version unless we only consider the names of group items rather than the ids). They added a batch method so that you could send very many calls to MailChimp but they had no idea what the maximum allowable would be. We would wait, send through our batch updates and wonder if it would time out or not. They changed the unique identifier for a subscriber so that it had no bearing on the previous unique identifier. This meant that clicking on a link to bring up the contact in MailChimp no longer worked because the unique identifier no longer existed. (This has now been fixed, or at least we are redirected to the correct contact).

One of our biggest headaches which is still ongoing is around segments. The new API made it very difficult for us to look up a subscriber by constituent id (The Raiser’s Edge unique identifier). This, we had always insisted, be stored as a merge variable. Of course if we were starting from scratch we would use the new MailChimp unique identifier and store that in Raiser’s Edge but this would be an enormous change and one that would not work with existing organizations.

Previously we would feed in a segment into the method that would get the subscriber details. This segment would retrieve the subscriber by constituent id. This was removed and with it an enormous stumbling block was put in front of us.

Some relief was forthcoming about a month before the cutoff deadline. MailChimp announced that they would not be shutting down the export API. Clearly it was realised that the batch methods were just not good enough for large data transfers. This was important news for us as we relied heavily on this part of the older API.

It still leaves us with the issue of segments. The V3 developers did not look at the V2 segments. Instead they just changed the format. The problem, or course, is that the export API takes in segments in the V2 format. It is all very well fetching the user’s existing segments and then passing them into the export API (an extremely powerful feature that we have been making regular use out of) but all of a sudden we have to adjust each segment in order to convert it to a version that the export API understands.

What we would like MailChimp to do:

  • Either expose the V2 segment definitions again or allow us to pass in V3 segments into export. Either way should not mean that the existing method is just turned off. We need to time to convert and test.
  • Update the export API so that it is fully compliant with V3 (again keeping the existing calls backward compatible)
  • Produce a guide to transitioning from v2 to v3. I have been asking for this since we started and amazed that this was never forthcoming.

 

To our Raiser’s Edge Chimpegration customers: thank you for your patience. We realise that you do not care who is to blame for issues with Chimpegration as long as the application works. We understand that and, as ever, strive to release fixes to issues as soon as we are told about them. After much work we hope that we have reached a point where this latest version of Chimpegration is now more stable than ever. We continue to work on the plug-in to ensure it remains as useful as possible in integrating these two great applications.

Adding an Education Record to a Non-Constituent Individual

Intermediate, , ,

I have been struggling to find a way to a an education record to a non-constituent individual relationship. There is a Blackbaud knowledgebase article here that outlines how you add an education relationship but it takes a lot for granted. Here is the gist of it:

Dim oEdu As CEducation2
Set oEdu = New CEducation2
oEdu.Init REApplication.SessionContext
With oEdu
   .Fields(EDUCATION2_fld_RECORD_ID) = 678   'Selects the record to add the Education record to
   .Fields(EDUCATION2_fld_SCHOOL_ID) = "Berry College"
   .Save
End With

However where does the 678 come from? For a constituent this is simple. It would be the constituent system id found under the Records_fld_Id field.

However what would it be for a non-constituent individual relationship. After all they can also have education relationship records.

I tried the field Individual2_fld_Id but the error I got back told me that I could not add an education to an organization record! Not very helpful.

In the end, with a bit of trial and error I worked out that you need to put the field Individual2_fld_Relation_id.

This is the id for the corresponding entry in the Records table which consists of both constituents and non-constituents.

Hope this helps somebody.

Working with Phones/Email without addresses in Raiser’s Edge 7.94

Intermediate, , ,

Prior to the release of The Raiser’s Edge version 7.94 I was working with all of our products to ensure compatibility with this latest version. If you have not heard (and if you have not heard where have you been hiding), this release of RE removes phones and emails from physical addresses.

When the concept of emails was new, it was possible that you would have an email address tied to your telephone provider or an email address specific to your place of work.  Your phone would either be at home or at work. Having these connnected to your physical address made sense. However it quickly became aparent that with the arrival of mobile phones and of email addresses that were accessible no matter where you were located, phones and emails (and for that matter all types of communication links) should be tied directly to the constituent record and not to a physical address. This is what has happened with the release of RE7.94.

This is a big shift and in terms of developing applications, we have had to allow for both possibilities so that our programs are compatible with users still on 7.93 and below and those that have made the leap over to 7.94.

The good news is that the old way of doing things still works in 7.94. You can still access phones via the CConstitAddress.Phones collection for an address. However you will probably want to access them how they are intended… Free from addresses.

This is done using the new interface IBBPhonesParent. This is implemented by CRecord, CIndividual2 and COrganization2. The collection of phones is a CConstitPhones object which contains the usual methods. You can iterate the collection to give you one CConstitPhone object but here is the problem.

For reasons that I don’t fully understand (I was told due to binary compatibility reasons) there are no properties or methods on the CConstitPhone object. Instead you have to convert this object to an IBBDataObject in order to access the Fields property. This is a real pain but to save myself some trouble I put together two extension methods for the CConstitPhone object which does this for me. (Unfortunately extension properties do not exist so that is why I cannot simply create an exact corresponding Fields properties. Those working with C# will be familiar with this as there is not a Fields property but rather  get_Fields and set_Fields methods)

<System.Runtime.CompilerServices.Extension()> _
Public Function Fields(ByVal phone As CConstitPhone, fieldConstant As ECONSTITPHONEFIELDS) As Object
   Dim dataobject As IBBDataObject = CType(phone, IBBDataObject)
   Return dataobject.Fields(fieldConstant)
End Function
<System.Runtime.CompilerServices.Extension()> _
Public Sub Fields(ByVal phone As CConstitPhone, fieldConstant As ECONSTITPHONEFIELDS, value As Object)
    Dim dataobject As IBBDataObject = CType(phone, IBBDataObject)
    dataobject.Fields(fieldConstant) = value
End Sub

With these extension method you can simply access the Fields methods on a CConstitPhone object in almost the same way as you would with other objects.

So here is an example of creating a new phone on a constituent record using the above extension code.

Dim constit As CRecord = GetConstituent()
Dim phonesParent As IBBPhonesParent
Dim phones As CConstitPhones
Dim phone As CConstitPhone

phonesParent = CType(constit, IBBPhonesParent)
phones = phonesParent.Phones
phone = phones.Add()
phone.Fields(ECONSTITPHONEFIELDS.CONSTIT_PHONES_fld_PHONETYPE, "Home")
phone.Fields(ECONSTITPHONEFIELDS.CONSTIT_PHONES_fld_NUM, "123-4567")
phone.Fields(ECONSTITPHONEFIELDS.CONSTIT_PHONES_fld_IS_PRIMARY, True)

constit.Save()
constit.CloseDown()
constit = Nothing

Overcoming the password update policy for custom Raiser’s Edge applications

Advanced, ,

A recent question on the Blackbaud forums got me thinking about this problem. The issue is this. If an organisation has a password policy in RE in place that ensures that users have to update their password every X days, what happens to a custom application that runs every day in order to perform some maintenance / export / import etc.? It is also required to update its password. This is somewhat problematic because most scheduled tasks are just meant to be run and more or less forgotten about.

The obvious solution is to turn off this functionality. However you are only able to do this for the whole organisation which is problematic.  Another solution is to use Windows authentication to log into RE. That way it is Windows that decides the password policy. This is also not always possible.

Here is a third, programmatic way of doing this. You need to make use of the “other” API. I have mentioned this previously in Checking Security. You need to make use of the  Blackbaud.PIA.RE7.SecData7 assembly. This gives you access to the security objects that are not present in BBREAPI.

You set up a database table with three columns; a primary key id, a password and an expiration date. You then fill the password column with a list of passwords that could be used.

On starting the application you select from the table the password with the most recent expiration date (which may be in the future). You use this to log into RE using the usual code. Once that is done you determine whether or not you need to change the password. If the expiration date is in the past then you should change the password using the code below. The new password should be the next password in the table that has a blank expiration date and lowest id.

Dim user As New CUser
user.Init(SessionContext)
user.Load(SessionContext.CurrentUserID)
user.Fields(Blackbaud.PIA.RE7.BBInterfaces.EUSERFields.USER_fld_PASSWORD)

You then set a new expiration date on this password. This expiration date should be a good few days before the actual date you are required to change the password that way the existing password will still be good.

If there are no passwords left in the table you can remove all the expiration dates and start from the first value in the table i.e the password with the lowest id. I assume that RE allows you to use the same password as at some point in the past if not the most recent values.

One variation to this is to just create a random passwords and not have a list of passwords. That way you would only have a table with one row and an expiration date.

Any improvements? let me know in the comments.

Checking access restrictions in The Raiser’s Edge

Intermediate, ,

In a recent project I had to ensure that specific confidential information was being saved as an action. I had a custom screen where this data was going to be viewed and edited. Only certain users had access to that action using security by action types. I had to check to see if the current user was able to view the action and if they could whether or not they were able to edit the details. There are useful methods for this in the API and I assumed it would be a simple task to use them. Continue reading Checking access restrictions in The Raiser’s Edge →

Batch API – error with tempRecords is nothing

Intermediate,

“Nothing :
Init method must be called before using this object.” My colleague was testing a new application that I have written for a client. Every so often she would tell me that this error message was appearing in the control report. Everything else appeared to work. Just what did this mean? It was one of those classic RE error messages that means nothing. When I started to debug the problem I saw the error quite soon. Again it was one of those Batch API idiosyncrasies that just needed to be dealt with.

I was creating several gifts in a batch. I was looping through the gifts and creating each one.

For Each donation In donations
    tempRecords = CType(batch.TempRecords, Blackbaud.PIA.RE7.BatchData.CTempRecords)
    PopulateOneGift(constitSysId, batch, tempRecords.Add, donation)
    tempRecords.Save()
 Next

The error occurred on the second iteration of the loop. Whenever the tempRecords.Add
was called the error was raised. This was strange because normally you get this kind of error when you have not initialized the object or done a Closedown on it. Anyway the solution to add one line before calling the Add:

For Each donation In donations
    tempRecords = CType(batch.TempRecords, Blackbaud.PIA.RE7.BatchData.CTempRecords)
    tempRecords.Reload()
    PopulateOneGift(constitSysId, batch, tempRecords.Add, donation)
    tempRecords.Save()
 Next

That seemed to fix the problem.

EDIT: or so I thought. It appears as though it worked once then failed after that. I have now resorted to saving and closing the batch after each iteration and then opening it up again at the beginning. Ah the joys of the batch API.

The Raiser’s Edge Integrated with Potentiality… A series of case studies (2)

Beginner, , , , ,

Towards the end of 2010 we started working with Potentiality to integrate their system with The Raiser’s Edge. At the time I knew very little about them and their products. Their headquarters is in Melbourne, Australia with an office in London. Even though they have an office in London our main contact was with their Melbourne office. Their product was a community based content management system (CMS) targeted towards schools and universities but also used by any organisation that was trying to build up a community of supporters.

Continue reading The Raiser’s Edge Integrated with Potentiality… A series of case studies (2) →