Whitepaper
Fighting Data Breaches from Zombie APIs
An Application Programming Interface (API) is great for scalability of your applications and protecting corporate logic and database information to backend procedures. Like any infrastructure, APIs should be monitored – both the API itself and the infrastructure that hosts it. Developers might retire APIs to make way for new endpoints during upgrades to applications, and in doing so can also forget to disable endpoint functionality. When APIs are no longer monitored but still publicly available, they become shadow APIs, also referred to as zombie APIs.
Having an API for applications is more common in business, often allowing for better scalability in mobile applications. APIs have many benefits for businesses, but they should be monitored for any anomalous behavior. This paper explains shadow APIs, how it happens, the pitfalls of leaving shadow endpoints on your infrastructure, and what p0 can do to help you avoid data breaches from these threats.
What is a Shadow API (also known as a Zombie API)?
If you’ve ever dealt with shadow IT, you know that unknown infrastructure on a network can lead to undetected threats exfiltrating data from your environment. The concept of shadow IT can be used to understand the characteristics of a zombie API (also referred as “shadow API”). If you’ve heard of shadow IT, you know that it describes hidden infrastructure in your environment. Hidden infrastructure could be a malicious device like a smartphone, or it could be an unmonitored router missed during an IT audit. Malicious shadow IT equipment could silently steal data or allow remote access to your local IT environment. A legitimate unmonitored router could open vulnerabilities when its firmware isn’t updated with the latest security patches. Shadow IT is a real threat to the security of your data and network environment.
Zombie APIs open similar vulnerabilities – they become unmonitored publicly available infrastructure allowing for potentially exploitable actions. Often, zombie APIs are a developer oversight. In a large environment, it’s not uncommon to have several APIs including those meant for testing. If developers abandon a project, the testing APIs might still be available. Developers might retire specific APIs and replace them with updated endpoints for new applications. Whether forgotten or deprecated, unmonitored APIs open vulnerabilities and allow for threats to persist and steal data long-term.
When an API is publicly available and still functions but remains unmonitored, it becomes a shadow API (or zombie API). It’s usually unknown but still remains functional for queries. For compliance and data security, all publicly available APIs should be monitored just like any other infrastructure. When APIs are forgotten or unmonitored, they could be used in a data breach where attackers have a large window of opportunity to steal data.
In 2019, the largest local search service, JustDial, experienced the consequences of a zombie API. JustDial uses several API endpoints to provide search results for various vendors in India. Search queries are linked to customer information including phone number, full address, and other personally identifiable information (PII). Databases for JustDial stored about 100 million user accounts, and they used an API to handle search queries.
After two decades in business, a security researcher found that a zombie API publicly available on the internet allowed anyone to perform a search and return full PII on any JustDial customer using a zombie API endpoint. The security researcher confirmed that the API endpoint existed since 2015, but it had not been used in production and was forgotten by developers. It’s unclear if any unauthorized third party used the API to collect PII, mainly because the API was not monitored, but the zombie endpoint left 100 millions individuals open to identity theft.
Because of the JustDial incident, attackers are more aware of zombie API potential. Exploiting an endpoint is much more convenient. No hacking is necessary, and the API functions like a valid endpoint. Zombie APIs aren’t monitored, so any exploitable vulnerabilities would go unnoticed by corporate intrusion detection systems. Any data exfiltration can be performed without the stress of transferring data as fast as possible to avoid discovery.
How do APIs become Zombie APIs?
Whether an API becomes an oversight or it’s left available but unmaintained, it’s likely that the API is also no longer monitored. Unmonitored APIs are just as dangerous for your security as shadow IT. You won’t know if the zombie API has a vulnerability, if that vulnerability has been exploited, or if the API discloses sensitive data. Without any monitoring, data disclosure or threats could persist in your environment for years.
A common developer strategy is to run deprecated APIs alongside new ones until everyone ports over to the new application. Developers might announce the discontinuation of support for older APIs, but users will refuse to switch unless forced. Some deprecated APIs could persist for years, and they could be forgotten as development staff rotate in and out of the organization.
A few examples of events that lead to zombie API creation:
Legacy applications are retired but still used by small groups either externally or internally to the organization.
Development of updates to applications might require creation of new endpoints and old endpoints are unused and forgotten.
New development stalls and a product is never released. APIs tied to the new product are never disabled.
Testing APIs are deployed to a misconfigured or publicly available cloud location making them unknowingly accessible to the public with no monitoring enabled.
There are several scenarios where the above examples can happen. In an enterprise environment, it’s not uncommon to have numerous APIs. Mobile apps often use APIs to communicate with devices. Internal applications can use APIs to communicate in the cloud and locally. APIs give business applications scalability, and a growing business might find itself losing track of deployed code as the codebase grows.
As an example, suppose that your business develops an MVP for a potential mobile application. You wouldn’t develop a full range of API endpoints until you have the final application fully built. Developers will create basic API endpoints to support the MVP frontend code. Once the MVP is developed, deployed, and metrics analyzed, you might decide to extend the application to a fully functional customer mobile application.
Transforming an MVP to a fully functional application takes extensive changes, and it’s likely many endpoints will change. It’s common for developers to build additional features and API endpoints, and they might abandon older pre-launch endpoints. If old unused endpoints are never disabled, you could create a zombie API.
Another common example is when legacy endpoints are never deprecated and disabled. Let’s say that you decide to rebuild an application, refactor the frontend, and add customer features. Refactoring an application is a large project, and many backend functions change. It’s likely that API endpoint names and features will change. Developers might add a new version API endpoint to the code and deprecate the older one without disabling it.
To support the existing production application, developers leave API endpoints in service and run legacy code side-by-side to new production code. As more customers move to the new application, legacy code is maintained less. As more developers leave the organization, it’s possible for old API endpoints to be forgotten and eventually excluded from monitoring software. With old legacy APIs forgotten, they can lead to quiet unmonitored data exfiltration with no ability for developers or operations to receive alerts on suspicious traffic.
Results from p0’s solution allowed Domino’s India to disable endpoints and ensure monitoring was in place for all APIs, increasing their security posture. By disabling zombie APIs, Domino’s India better protected their data, avoided unmonitored data breaches, and avoided costs associated with incident response and litigation.
Traditional monitoring of API activity:
To monitor API activity, the first thought for many operations people is to deploy an agent across the network environment. The agent runs on critical infrastructure and servers hosting API activity. For example, an agent can run on database servers and API hosting servers to record data requests. Every connection, database query, and processing request gets logged to a file that can then be parsed and analyzed. Security tools ingest event logs and automatically send alerts to key people during potential attacks.
Agents are great at monitoring network traffic, and they can detect many of today’s current threats and anomalous behavior. With shadow APIs or zombie APIs, they are unknowingly running in a production environment. An agent must be configured to monitor an API, so a zombie API would not be added to the agent’s list of traffic monitoring. Agent monitoring is a legitimate way to monitor network traffic, but administrators must know what to monitor or miss out on logging critical infrastructure, including zombie APIs.
You could use a technique called fuzzing to find APIs, enumerate them, and add them to an agent. Fuzzing to find API endpoints involves performing requests on servers using a dictionary list of common API endpoint names. For example, an environment where frontend applications pull customer information likely has an endpoint named /api/customer, /customer or /customers. A fuzzing script makes requests to these three URIs and returns a positive result if one matches.
This example only uses three possible endpoint URIs, but fuzzers work with thousands of possible endpoint names. Your organization can work with fuzzers to find endpoints, and it’s common for third-party threats to use fuzzing to discover your shadow APIs. Fuzzers work for both good and nefarious purposes, but they aren’t fully accurate. It’s possible to have a URI that is not included in a dictionary list. Your fuzzer might miss the URI, but an attacker’s fuzzer finds it. The result is an open API possibly vulnerable to data disclosure without being monitored by agents.
Pitfalls of leaving Zombie APIs in your environment:
It’s common for small businesses or busy development teams to ignore shadow APIs. The thought is that no one could possibly find an open endpoint, especially one with an uncommon name. Some business administrators aren’t aware of vulnerabilities in API requests and think that no harm can be done from a simple API returning minimal information.
It’s possible that malformed requests could do more harm if APIs have not been pen-tested. A penetration test looks for opportunities to disclose data using malformed database queries or unexpected input. APIs that do not require authentication to return data could also be a risk of data disclosure. When these APIs aren’t monitored, attackers can freely take their time in finding vulnerabilities and eventually exfiltrate data slowly without rush of being identified by intrusion detection systems.
Virtual machines often host API endpoints, and they must also be monitored and maintained. If the server hosting API endpoints is also abandoned and forgotten, the server itself will not be patched with the latest security updates, leaving the operating system also vulnerable to exploits. During the discovery phase of an attack, the original vulnerability can start with the zombie API and result in a full compromise of the hosting server. Without monitoring, administrators would be unaware of the compromise.
Several compliance standards oversee protection of consumer data. Slow exfiltration of data from unmonitored zombie APIs leave organizations at risk of hefty fines for compliance violations. Personally identifiable information (PII), medical records, financial data, or any other compliant-regulated data should be protected. Without monitoring endpoints supplying this type of data, an organization could face fines, possibly in the form of seven-figure penalty payments.
The previous JustDial example illustrates an endpoint that could disclose data, but vulnerable zombie APIs could offer a wide range of exploit opportunities. Most people think of Silicon Valley technology companies as the best in security standards, but even large enterprises with the best security staff in the world can fall victim to shadow APIs and forgotten infrastructure. Facebook is one such organization that fell victim to zombie API vulnerabilities.
In 2016, Facebook patched a security bug in their password reset infrastructure, namely publicly available zombie APIs. When users sent a request for a password reset, a six-digit code was sent to the user’s smartphone. Users entered the six-digit code in the frontend to reset their passwords, and the six-digit code was sent and validated on a backend API endpoint. The production API was rate-limited to 10 requests before the user account was blocked from making additional requests.
To test this functionality, Facebook engineers hosted beta versions of the new feature on beta.facebook.com and mbasic.beta.facebook.com. After the production version was deployed, Facebook engineers forgot about the other subdomain-hosted versions of the API and left them publicly available. The production version was rate-limited to 10 attempts before the user was blocked, but testing versions were left open to unlimited attempts. If attackers found the test API versions, they could use them to brute force six-digit reset codes and take over accounts.
Once the API is discovered, an attacker could first send a password reset request. The user would receive notification and might ignore it, leaving an attacker open to brute forcing the six-digit verification code on the test Facebook API subdomains. Because no monitoring was set up on the shadow API, an attacker could continue to send up to one million “guesses” to the password reset API, which is the total possibilities for a six digit number. After the code was successfully submitted, an attacker could reset the account password to their own value and take over any user’s account.
Luckily for Facebook, the bug was found by a researcher looking for a bounty, but it’s unknown if any accounts were taken over. The shadow APIs were unmonitored and unknown to Facebook engineers, but they were remediated and no longer pose a threat to user accounts. Shadow APIs affect any organization, but their biggest threat is the longer window of opportunity for threats to exfiltrate data without detection.
The p0 solution: Monitoring for Zombie APIs in code
As many times as you fuzz your environment and monitor APIs using agents, you still leave a large margin of error for potential unknown vulnerabilities. A better way of preventing zombie API-based vulnerabilities is to identify when APIs are created during deployment of your code. The p0 solution finds APIs deployed in code and monitors your logs for a complete audit of endpoints and your attack surface.
Your code likely sits on one of the major repository platforms (e.g., GitHub or GitLab), and p0 monitors your codebase whether you run one or several repositories. Let’s use GitHub as an example. Your business might deploy its application code to GitHub and make the repository public or private. p0 continually runs in the background searching your code for API endpoint URIs. URIs could be fully qualified domain names or relative URIs (e.g., /api/customer), but p0 will identify either form.
It’s important to note that p0 does not interfere with any repository activities including deployments, forking, compiling, editing, or any other developer action. p0 is a background application that silently performs code scanning and alerts administrators when potential zombie APIs are found.
To identify a known production API versus a zombie counterpart, p0 scans log files for activity. The p0 engine uses artificial intelligence and machine learning with large language models (LLMs) to ingest events from API network traffic. Let’s say that your /api/customer endpoint is production-ready and actively used, but /api/beta/customer is retired and not used for any current applications. The p0 engine ingests log events, analyzes them, and determines that /api/customer is actively in use but /api/beta/customer is marked as a zombie API.
Any marked zombie APIs are displayed in an admin dashboard where administrators can review them. Administrators can work with developers to remove APIs from code and disable them on hosted servers. After cleaning up zombie APIs, your corporate attack surface and cyber-risks from endpoints are reduced.
How p0 works:
Reducing your risk of a compromise has several benefits:
Better reputation management to avoid loss of customer trust and loyalty.
Better compliance.
Tracking potential risks benefits risk management and helps identify security infrastructure opportunities.
Potentially lower cyber-insurance premiums.
Increased return on investment on your security strategies.
Fewer incident response and emergency mitigation and containment of threats.
Better visibility into endpoint activity.
Protecting your data and environment using p0:
Proactive security is critical to your data protection, and the p0 solution puts security at the forefront of your API development. Developers and administrators know when API endpoints are no longer active, so they can disable or better manage them after being retired. Legacy applications are always difficult to manage between support and security, but p0 helps you lower risks and monitor your attack surface for unknown publicly available APIs.
Even large enterprises struggle with managing and tracking zombie APIs. You can stay ahead of trending cybersecurity strategies by proactively monitoring your codebase for vulnerabilities. p0 monitors API endpoints with AI-based scans and lets administrators review their infrastructure. To find out more about what p0 can do for you please contact kunal[at]p0[dot]inc
