1) SSO and SCIM

Some of my favorite open standards! Single Sign On (SSO) is an absolute must. Even with a decent password manager, dealing with separate credentials is a pain. And let’s face it- given an opportunity to reuse passwords, many end users will choose the same [email protected]for all of their work applications. Instead of managing password expiration, complexity, and 2Factor/Multi Factor Authentication yourself, leave it to a good Identity Provider (IdP). SAML 2.0 is a bare minimum and will work with practically any SSO solution. Bonus points available for supporting OAuth and direct SSO with Google. OAuth with G Suite not only gives a unified login experience, but you can also request additional OAuth scopes for programmatic access to data in GMail, Google Drive, and more.

SCIM, or System for Cross-domain Identity Management (what a mouthful!), is a newer standard that is particularly slick. While SSO can provision users using just-in-time (JIT) provisioning, it cannot actively synchronize user profile information, nor can it disable or delete users. This is where SCIM shines! SCIM can synchronize all of your user profile information such as location, job title, manager, phone number and more with any supported application. For example, Spoke uses user location from SCIM to surface pertinent knowledge- if we know you work in the NYC office, we’ll show you the NYC lunch menu before SF’s menu. With manager information from SCIM Spoke can automatically trigger an approval workflow to your manager when you request a new laptop or admin access.

2) Audit Logs

There are a million reasons to have good audit logs! I expect any of my SaaS tools to capture every action, every login, every configuration change, and every transaction in the system. Troubleshooting a lost notification, tracking down an errant config change, or narrowing down a user’s problem becomes much, much more difficult without solid logs. A good audit log is detailed, read-only, searchable, filterable, exportable, and accessible via API. For critical systems I stream these Audit Logs into my security logging platform, or SIEM to have a “single pane of glass” view of everything going on in my environment. Audit Logs and Centralized Security Logging are often a compliance requirement in HIPAAISO27001, and other regulated environments. Logs… don’t be caught without them!

3) Access Roles

Being able to delegate certain administrative functions without giving away “the keys to the kingdom” is incredibly important. I need more than an “all or nothing” approach to permissions! A good SaaS application will do granular, role based permissions, and allow me to create custom roles. I should be able to grant different levels of access/permissions for each data type, part of the application, or part of the org. Being able to empower HR to, say, change job titles but not be able to create API tokens, or allow Tier 1 helpdesk to reset passwords but not read the CEO’s email, is a huge win for both efficiency and security. I like to see thoughtful “out of the box” roles such as the following:

Super Admin- full access to everything. Usually reserved for Senior IT Administrators, there should only be a handful (literally, a handful or less) of these accounts. Best practice is to keep these powerful accounts separate from regular logins and only used when needed.

Help Desk- ability to change user profile information and view logs, but not make global configuration changes. You should give the Help Desk enough access to troubleshoot issues, but not enough to make large-scale changes (without going through an existing change management process).

Read-Only/Compliance- this role is invaluable in any regulated environment. Being able to give the internal compliance/security team or an external auditor self-serve access to configuration and logs reduces the audit burden on the IT team and encourages a healthy relationship between IT and Information Security.

Billing Access- I happen to appreciate applications that have a separate role for billing. This role is able to edit payment details, download invoices,view licensing information, etc.. This is a great role to give to your finance team for visibility. For expensive applications it’s even better if there’s a license that allows for billing and user management access without consuming a license for the application itself. For example, I like how Adobe Creative Cloud allows free accounts for admins to manage licensing, but doesn’t require each admin to be licensed themselves unless they need application access.

4) Open API

A modern RESTful Application Programming Interface (API) is a must. In the On-Demand Workplace, we pick the best tool for each problem and end up with a very heterogeneous environment. Gone are the homogenous days where the entire stack was Microsoft or IBM or Oracle and everything integrated out of the box. Instead, modern SaaS tools use APIs to securely integrate with each other. Tools bolt together as easy as Legos. Emails or CSV exports are not integrations! Your API should be public and well documented. I’m a big fan of the OpenAPI Specification for uniformity and detail, and that’s what Spoke happens to use to document our API. A good API encourages better integration and adoption of the product, and platforms like Zapier or Tray.io make it so easy to shuffle data around.

5) Security Certification(s)

The best way to prove the security of your service to customers is with one or more security certifications. SOC 2 is very popular, focusing on five “trust service criteria”—security, availability, processing integrity, confidentiality, and privacy. SOC 2 evaluates how an organization manages risk, governs change, and the effectiveness of their system operations and access controls. The SOC 2 framework is tailored for service organizations like SaaS providers and therefore addresses the core of security concerns that companies have when selecting a new vendor.

Another common certification is ISO27001. This is a very thorough certification and audit that covers the following domains:

A.5 Security Policy

A.6 Organisation of Information Security

A.7 Human Resources

A.8 Asset Management

A.9 Access Control

A.10 Cryptography

A.11 Physical and environmental security

A.12 Operations Security

A.13 Communications Security

A.14 System acquisition, development and maintenance

A.15 Supplier Relationships

A.16 Information Security Incident Management

A.17 Information Security Aspects of Business Continuity Management

A.18 Compliance

The ISO27001 certification process spans several years and is broken up into two stages. ISO27001 is often adopted by enterprise organizations that have a global customer base since it’s an international standard that is recognized world-wide.

Most SOC 2 and ISO27001-compliant organizations conduct regular penetration tests against their network and applications through a 3rd party firm. Penetration tests are designed to identify security weaknesses that could be exploited if not addressed. While vendors won’t share the full details of their penetration testing, they will often share a redacted report showing the severity of the issues found (Critical/High/Medium/Low) and the efforts made to remediate the findings.

I like to see organizations performing penetration tests at least once a year by a well-recognized firm. A report on compliance combined with security testing demonstrate that an organization has a well defined information security program and it has been evaluated through an independent party. In this day and age, organizations should apply a healthy amount of scrutiny when choosing new vendors. If a vendor doesn’t maintain one or more security certifications, you may want to look elsewhere.Reporting and Analytics – any good SaaS tool needs to be able to surface the right metrics to me in the right way. Good reporting will not only show me what’s happening “live”, but also allow me to go back in time and look for trends and events. Surfacing this with searchable and filterable views, clean charting/graphing, and the ability to share metrics and dashboards is another must.

6) Reporting and Analytics

Any good SaaS tool needs to be able to surface the right metrics to me in the right way. Good reporting will not only show me what’s happening “live”, but also allow me to go back in time and look for trends and events. Surfacing this with searchable and filterable views, clean charting/graphing, and the ability to share metrics and dashboards is another must.

While “in-app” analytics are very useful, I also appreciate the ability to export data via CSV download, API, or streaming into a cloud storage bucket. Just like how competent IT organizations will centralize their logs into a SIEM, they will also centralize their data into a data warehouse. Pulling data together from many systems is a force multiplier, correlating information from many applications into a single view that can greatly inform business decisions. I happen to be a huge fan of Tableau to present data to almost any audience. Here are some of the data types I’ve come to expect:

  • Login Analytics – How many active users are there? Do I have unused licenses? Where are my users logging in from?
  • Usage Data – How engaged are my users? What parts of the application are they using the most and the least? How has adoption been since launch? Are certain features not being used, indicating a bug or lack of training?
  • Sharing/Permission Information- Is any data shared externally, and if so, is that sharing authorized? Do users have enough access to data, or do they have too much? Is someone snooping around for items they’re not authorized to view?
  • Platform Specific – Anything more specific to the platform, but usually more transactional data. For example, on an email marketing platform I’d like to see the number of emails sent, email bounces, unsubscribes, email opens, and more.

7) Support and SLA

Let’s face it, even the best SaaS applications still have bad days. I recently covered how to deal with SaaS outages in a previous post, and now it’s time to discuss support and SLAs. I firmly believe that support is not really a service or a function; instead it’s actually part of the product offering itself. Good support can elevate a problematic application, and bad support can frustrate even on the best software. Email and web support during business hours is the bare minimum, yet I happen to like live chat support for most issues and a phone number for critical things. Support needs to be knowledgeable, empathetic, and excel at follow-up. During a trial or Proof of Concept deployment, I’ll often submit a “test” support request to see how quick and helpful the response is.

I firmly believe in Service-level agreements (SLA). An SLA is a contractual commitment made by the vendor regarding the performance and availability of their system. For example, Google guarantees 99.9% (“three nines”) availability of G Suite, meaning they commit to having less than ~9 hours of downtime per year. Like any meaningful SLA there are financial penalties, with Google offering service credits if monthly uptime dips below 99.9%. This is a way to “put your money where your mouth is” – publicly committing to customers that your service will be up when they need it, otherwise you’re holding the bill.

8) Flexible Billing

In the cloud world, Enterprise Agreements (EA), Volume Licensing, and Site Licensing are mostly gone. There are no more 5 year commitments and no more selling your soul! To be considered “cloud” I expect to be able to pull out my credit card and have your service going in 5 minutes or less. I don’t want to depend on a salesperson, and I certainly don’t want to wait for my tenant to be provisioned. I appreciate clear and simple pricing and I expect to see those numbers publicly on your site, not behind a “contact sales” form.

Per (active) user, per month billing is the new norm. I should only pay for what my users are actually consuming, and that number may change every month. Slack has pioneered this model by only charging for active users instead of all users provisioned, even if inactive. Monthly billing with a discount offered for an annual commitment is the way to go. With clear billing I can budget all of my cloud costs very easily, and then I can go to the CFO and say “It costs us $229/mo for each salesperson, and $87/mo for each support agent. We can hire as many as we need, just VLOOKUP these costs against our hiring plan to budget!”

Alright, these are the top 8 things I look for when evaluating any SaaS tool. There are of course more things to consider, but this is what is top of mind for me. I recently discovered a cool checklist called Enterprise Ready that dives deeper into what it takes to be “Enterprise Ready” as a SaaS tool, and it’s a phenomenal resource whether you’re building or buying SaaS.