API Clients and Servers

Overview

Black Magic Data provides an API that enables pairs of sites to communicate using Magic Data. Client and server sites are defined in the dashboard, then the client site can use API_​EVALUATE and API_EVALUATE_END to wrap sets of symbols to evaluate on the server site.

Dashboard Configuration

In the dashboard at Dashboard > Magic Data > API Clients and Servers, you can use the Clients tab to list and configure clients - other sites that can use the Magic Data API to evaluate expressions on this site, and servers - other sites that this site can send Magic Data expressions to for evaluation.

The details of a client defined on one site need to be paired up with the details of a server defined on another site. Currently these interfaces need to be paired 1:1. Future updates of the BMD API may include 1:Many interfaces and even public interfaces. However, for now we are keeping the pairing tight to enhance security.

Client Sites

On a site providing an API server, you need to define client sites.

Client sites are defined by site name, url and optionally an IP address filter. All clients are listed in a summary, then an individual client can be edited or new client added.

70 bmd api client list.png 

Here we see the edit dialog in more detail. Note how the client can only evaluate symbols in the permission set 'No Black Magic' on this server. The user is defaulted to 'Guest', but any user could be selected (and given concrete5 user permissions) the API limited to the pages and files visible to a client.

API calls and returns are encrypted using the API key. An API key can be generated automatically on the server, or entered as any text you like. Unless you have good reason to do otherwise, it is recommended you use the automated key generation. You then need to copy that key from the server site's API client edit form and paste that key to the equivalent API server edit form on the client site.

75 bmd api client edit.png

Server Sites

On a site that is a using the API to evaluate Magic Data on API server sites, you need to define Server sites.

The setup is easier than for clients. All you need to enter are the server site name and URL, then paste in the API key copied from the server site.

An information (i) button is provided in both client and server lists to show the information needed for the paired site.

80 bmd api servert list.png

Evaluating Magic Data on an API server

When a client site is configured with interfaces to one or more API servers, the symbols API_​EVALUATE and API_EVALUATE_END can be used within Magic Data expressions to evaluate subexpressions on API servers.

LIST_API_SERVERS 10
APPLY_EACH
  WITH_API_SERVER
  SET 1
  API_EVALUATE
    AS_USER USERNAME
  API_EVALUATE_END
  . ' on ' . ( CURRENT_API_SERVER )
END_APPLY_EACH
HTML_UL

Going through the above line by line:

  1. The client site lists up to 10 servers it can connect to
  2. For each of those listed servers
  3. Configure the following API evaluation to use that server
  4. Set the user id to 1 (the super administrator)
  5. Make an API call to the server
    (the next line is passed to the server for evaluation)
  6. On the server, find the user and get their username.
  7. The result is passed back to the client.
  8. Some text manipulation to concatenate the server name to our output
  9. Repeat for the next server
  10. Finish by presenting our list of administrator and server names.

This also illustrates that we set the user id we want to name on the client. That is passed to the server as the initial value (previous result) by the API_EVALUATE symbol. We didn't really need to do that for such a simple expression, but passing a starting value to the server can be useful for more sophisticated applications. Here is the result of evaluating the above in the symbol tester on one of my sites (you can see more detail by popping up the screenshot gallery to the right).

90 bmd api example.png

 

To keep the amount of data down, the API_EVALUATE symbol only passes the previous result value to the server and API_EVALUATE_END only returns the immediate outcome of the symbols evaluated on the server.

For more complicated interactions, you can alternatively use the symbol API_​EVALUATE_​EXTENDED to copy magic data memories from the client to the server, then copy them back again at API_EVALUATE_END.

Performance Considerations

Every block of symbols in an API_EVALUATE .. API_EVALUATE_END subexpression involves an http post request from the client to the server. If you do a lot of that in the middle of a big loop, it will have both a bad impact on your page load times and potentially create a big overhead on the server.

Within any page, keep the number of API_EVALUATE .. API_EVALUATE_END sections to a minimum. Where possible, make one API_EVALUATE .. API_EVALUATE_END block assemble a batch of various subexpressions all in one go and use memories to pass their results back all in one go.

Another potential performance issue comes with evaluating rules for Symbol Permissions associated with a client. These are defined on the server and for speed, it is best to have as few rules as possible in any one set. For example, its much faster to list a many packages or symbols in one rule than it is to have separate rules for every package or symbol.

Cacheing

If you have Uber List installed, you can use Uber List's cache mechanism to cache API calls. For example, the above could be cached to only retrieve new information once each day by wrapping the API call within a cahed subexpression. 

LIST_API_SERVERS 10
APPLY_EACH
  WITH_API_SERVER
  SET 1
  CACHE "global" 86400
  API_EVALUATE
    AS_USER USERNAME
  API_EVALUATE_END
  END_CACHE
  . ' on ' . ( CURRENT_API_SERVER )
END_APPLY_EACH
HTML_UL

Or even the entire loop of remote API calls, or the entire expression.

CACHE "global" 86400
LIST_API_SERVERS 10
APPLY_EACH
  WITH_API_SERVER
  SET 1
  API_EVALUATE
    AS_USER USERNAME
  API_EVALUATE_END
  . ' on ' . ( CURRENT_API_SERVER )
END_APPLY_EACH
HTML_UL
END_CACHE

Snippets

Within and API_EVALUATE .. API_EVALUATE_END subexpression, the magic data snippets available are those defined on the server, not those defined on the client making the API call.

Snippets evaluate with the server's own symbol permission set, as selected in the Symbol Settings dashboard page. If a snippet contains magic data that should not be available to an API client, you can exclude that snippet when setting up the permission set for the interface.

For a very tightly controlled interface, you could also take advantage of the opposite, exclude All symbols, then specifically enable only those symbols needed to evaluate snippets where you have pre-scripted what a client site can do. With this approach you can create powerful interfaces as snippets, but prevent an API client site from doing anything provided by those snippets.

Faults and Fault Handling

If a server fails to respond, the API evaluation on a client will timeout. You can configure this timeout via the Symbol Permissions dashboard page and it is defaulted to 10 seconds.

Following a timeout, the API_EVALUATE_END symbol will return null. Your magic data expression on the client should be able to cope with that, either by being immune to a null return or by specifically testing for a null return from the server, rather than failing completely.

There is no protection within API_EVALUATE subexpressions to prevent further nested API_EVALUATE subexpressions. Long chains of such could easily cause server timeouts at the start of the chain. Even worse, if a chain ended up in a recursive loop, where servers pass sub-expression evaluation round between them, you could lock up all servers in that loop. So please take extreme care if you really must nest API_EVALUATE expressions.

If you do create such a loop, the only way to stop it is to hope you can get into the dashboard Symbol Settings page and disable magic data.

By default, API calls are made using CURL. CURL is supported by most servers and is the preferred interface. However, sometimes a php installation will not have CURL available, or it may be buggy. Alternative methods can be selected in the Symbol Settings page.

 

IP Address Filtering

As a basic security mechanism, An API server can configure a client interface to only be available to a speficic IP address. Sometimes a client server could have an IP address that varies within a pool of addresses. Any section of an IP address can contain a * for any match, or nnn-mmm for a range match.

Some examples:

  • 104.25.24.33 - only accept API clients from this address
  • 104.25.24-25.33 - accept API clients with third section in the range 24-25
  • 104.25.*.33 - accept anything from 0 to 255 in the third section of the IP address

IP address filtering is not very thorough. An attacker could spoof a known IP address. However, for the majority of use it has the advantage of being quick.

If you really want speed, you can also set up IP address filtering in an .htaccess file.

Encryption

For stronger security, the Black Magic Data API is encrypted.

Encryption is adapted from the core encryption helper. The API key is used to encrypt communication from the client to the server. Within each communication, the client generates a new key that is passed to the server used to encrypt the data returned from the server to the client.

Roadmap

The Black Magic Data API is currently restricted to 1:1 pairing between concrete5 sites. Both client and server sites need to have Black Magic Data installed.

If a sever needs to support multiple clients, rows on the API client list can be copied for each client. Similarly for a client that needs to access multiple servers. But this is a manual process. The API is convenient for a few clients, but not for hundreds or thousands.

Future enhancement may add capabilities for:

  • 1:Many server to client interfaces
  • Unencrypted interfaces
  • Alternative encryption methods
  • Public interfaces
  • Interfaces accessible from JavaScript
  • Automated API key distribution from server to client
  • Automated API setup in response to client requests

Last updated: over a year ago