It’s called success when a company reaches the billion dollar mark, and while there isn’t anything inherently special about that particular base10 number, critical issues affecting the business become public knowledge. In this case, Zoom Video Communications is that relevant company, and the critical issues are those linked to their flagship product Zoom.
In a rush to scale a product to millions of users, products are rapidly spun up and delivered. Startups have become more and more agile and focus more and more on whether it could scale rather than making sure it does. In this race, all too often it comes down to cutting corners and skipping important milestones in hopes the product idea will be your goldmine. “We can do it later”, “It would be wasted effort if we did it now”, and other musings fly around product-engineering meetings at the best entrepreneurial businesses. But that’s a tangent of how or why those companies gamble on security. What’s important is--to make sure you are successful, you need to forgo implementing non-user-visible features, even if they are critical.
Many products are fraught with this peril but as their companies squander their opportunity to gain traction, the issues never come to light. Others that have over-engineered and over-invested in technology and solutions that no one is interested in, also disappear. In essence, it is a never ending struggle to release a successful product while still solving every technical problem that appears along the way.
And the worst of these critical, but non-user-visible, issues are security related. Security is the most hidden aspect to your product, one that you don’t think about as a user, but are fully expectant that it is there in full capacity. Additionally, while the concept is simple, the implementation and knowledge required for understanding it are extremely esoteric. It isn’t common to find engineers who think about the details of security, and even harder to find those who can implement a solution.
Over time, Zoom encountered a number of issues related to its product, I’ll reiterate through some of the key security related ones here:
- FBI warns of ZoomBombing - Anyone can join a zoom given the url. There are a bunch of nuances to this however though. These links also showed up in online search results for the company and video linked with their passwords.
- Insecure use of apps - App integration can share unnecessary user information and security access tokens offering a way to intercept and infiltrate even the most secure apps.
- Leaking of user privacy data - Automatic grouping of users with similar email addresses.
- Unsecured application resources - Zoom provides the ability to save recordings of the video sessions. This can be saved locally or in the cloud. Although it was just a matter of finding these links in order to access that saved session. So even if the initial meeting was password protected, the video itself was still viewable.
- User experience at the cost of security - building security sacrifices time spent delivering a great user experience. As can be seen from why companies gamble on security, there’s a lot of motivation to have great UX and the security, hidden from users' view suffers.
- Using XML in APIs is insecure by default - yet another XML related security vulnerability, that shows connecting to a zoom session creates a security vulnerability that allows anyone in that meeting to send attacks to your machine. Do not connect to Zoom sessions that don't trust the machines of everyone that is connected. With Zoombombing, this is a critical issue.
(This list actually goes on and on, you can find more of examples and their details if you search the internet)
Why did these happen? Were they avoidable?
To answer questions like these, we need to look at the core features that are required to exist, and then we can talk about what solutions to these security challenges would look like.
The features
We’ll focus on having great user experience, which means being able to deliver core functionality. I’ll enumerate those features that are related to the above security flaws. The goal is to design and deliver the best video sharing experience without sacrificing security. Can we do it?
- Easily share meetings with others
- Prevent unwanted attendees
- Record meetings
- Pull users from company directory for scheduling
- Provide 3rd party apps ability to record, listen, and augment sessions
That’s all very specific to Zoom, so I’ll abstract these a bit:
- Allow anyone with a link to access a resource with specified permissions
- Restrict access rights to specific people
- Save resource data and restrict to certain users
- Integration with SSO providers
- Provide app integration security
That list starts to sound really familiar, actually too familiar. Almost every single application ever has to develop solutions to these exact problems. Now we know what we need to implement, how do we go about it? This isn’t a short list, and every one of these requires attention to make sure it has the right user experience, but also the best security practices.
The implementation for these security based features has to look something like:
- Allow the users to create and manage access records. These records have to contain the details of lists of users as well as their permissions. Be it to record, mute/unmute others, change configuration, talk in chat, invite others, or automatically enter session without waiting in a “waiting room”. These need to be captured somewhere.
- Keep track of the users with access, so that when a user without access attempts to access a session they are met with restrictions based on configuration that makes sense. Is the resource open to everyone, only to internal, or requires explicit permission to access.
- The creation of new resources should be tied to the initial zoom meeting or perhaps not. Each resource that is created may need similar permissions as the initial meeting. But then again you might want to share the recording with others that aren’t there. Also, don’t you want to know who has seen the recording?
- When you are selecting users to join your meeting or attempting to schedule it, having autocomplete is a lifesaver. Where is that directory though? SSO is the answer. Most people-directory providers already implement solutions that either integrate out of the box or provide the ability to manually configure who is in the group.
- 3rd party app integration. You want to make your app extensible, whether it be webhooks, a REST api, public exposed resources, web RTC or websockets. Doing this requires careful control over the resources in your app. You don’t want to allow the apps to take full control of the users’ sessions, but you also want to allow them to interact in the way the users’ want. Security app tokens is the way to do this.
Getting it right
While we can easily suggest that having a stronger attention to security related aspects on day one might have allowed Zoom to avoid these disasters, that suggests too much hindsight for me to like. So what’s the alternative? Assume that these problems exist everywhere and the users just have to suffer the consequences? I don’t like that answer either.
The only thing I can think of, is to delegate the responsibility to security-first systems. Zoom is attempting this with the acquisition of Keybase, a firm known for their public GPG repository and security knowledge, but that doesn’t solve everything. Knowing you need to make changes is one thing, but it still takes time. Not to mention that not every company can just go out and purchase a security team on the spot, so there has to be a better solution.
The list is long of necessary features that also focus on security, and none of them are free and just fundamentally take time. They require a complete understanding of the space and focus on their delivery to get there. It isn’t surprising that apps like Zoom didn’t implement them to start. And while they are trying to get through them as fast as possible, security isn’t Zoom’s core competency, and chances are it isn’t yours either. Just like email isn’t my core competency, and I rely on email providers to handle it for me (mostly because I hate SMTP protocol because it isn’t RESTful), figuring out how to handle security isn’t yours.
There are many solutions sitting out there that can help implement these security features out of the box, so finding the right one becomes the main challenge. When searching, look for how easy it is to implement or integrate with the solution. The easier it is, the quicker your team can deliver the security your users expect. As an example, let me detail how Zoom could configure Authress to work for their users:
- Scope all resources in the following way: ZoomAccounts/{AccountId}/Meetings/{MeetingId}/Resources The main meeting will be the top level MeetingId and anything created as part of that meeting would be a sub resource.
- Create an access record per meeting. Those access records can contain as many people as make sense along with their permissions to the individual resources. The limits on resources and users in access records is much higher than most products need, ~500 for Zoom.
- Create the following permissions--EnterMeeting, EditConfiguration, Host, MuteOthers, UnMuteOthers, Record, etc… Then on each action the explicit permission could be checked to ensure that the user can enter. On the Meeting resource you can set it as either Public or Private, meaning that users without access cannot join or can join, but only after approval.
- Use an app directory for integrations. Rather than trying to keep track of the users in an account, use the GSuite, Microsoft, etc… app directory for the app. In the case you want it to be open and not use SSO or a default user directory, create one per account using the Organizations Manager. Using the domain name by default is a great start, but allowing users to automatically get permissions by the domain is not.
- Create Authress Service Clients and grant them access to only the part of the user or meeting they should have access to. On the client side, clients can be generated per user, but also per meeting depending on the integration. When configuring an integration, pass credentials to the app that it can use at a later point for integration. It’s important to pick a provider with OIDC compliant JWTs as the standard otherwise every app would need to write a custom integration.
It’s easy to see how many companies get to the place where product features exist, but critical security features do not. Spending precious time working on security means diverting attention from your core competency. Instead of doing that, find the existing security solutions that meet all the requirements a high volume app needs.