Project Lighthouse Attacked: Operation Purge 2

LBP Union and Project Lighthouse are not affiliated with Sony Interactive Entertainment or their subsidiaries. Project Lighthouse is a clean room reverse engineering project of now defunct PlayStation 3 and Vita LittleBigPlanet online features. No proprietary code is distributed. Under no circumstances will we endorse or support piracy. You must have your own copy of the game in order to use the custom features once they become available. When using these features, you release Sony Interactive Entertainment (Sony) as well as any employees or agents of Sony, from any and all liability, corporate, or personal loss caused to you or others by the use of Lighthouse custom servers for LittleBigPlanet.

Project Lighthouse, a community made custom LittleBigPlanet server, has been in development since October 2021. It’s been awhile, and we are slowly getting closer to our goal: public beta. We want to be able to let everyone play LBP online again on PlayStation 3 and Vita consoles. However, despite having implemented nearly all the minimum required features for public beta, there’s one other thing that’s important before we make the plunge: stress testing.

Let’s talk about how our Project Lighthouse testing server was attacked (by ourselves!) in Operation Purge 2. Bear in mind, this will get a little technical!

Union Space Corps

Shortly after the Project Lighthouse repository was first created in October 2021, private beta testing of the server was put under the charge of the Union Space Corps. This division of the LBP Union is dedicated to testing an instance of Project Lighthouse, called Beacon, and administrating the private beta community.

The emblem of the Union Space Corps features a lighthouse with three stars and the Union compass rose at the bottom.

Union Space Corps emblem.

As such, the USC’s leadership team, called Star Command, is responsible for bringing in new beta testers and conducting tests of the server software to look for bugs. The more bugs the USC can find, the more opportunities developers will have in safeguarding Lighthouse from failure and attack. One of the best ways to do that is to attack your own server. So, that’s exactly what we did!

Planning Operation Purge 2

On January 18th, LBPU Lead Protector m88youngling called upon Star Command to ask them to begin plans for stress testing Beacon. Chief Star Commander PorkchopGMX decided that the Lunar New Year would be the date of the stress test, known as Operation Purge 2. The mission was named after a previous stress test conducted last year.

Operation Purge 2 Has Begun  Attention, all pilots. Operation Purge 2 begins now and will last for the next 24 hours.  Your objective is to use whatever means necessary to attempt to take down or disrupt Beacon. The server will be restored from a backup after the operation concludes.  Please remember this is not an excuse to break the rules. Be respectful to others. If you have been asked to keep information secret, continue to maintain discretion.  Please report your activities and results in Operation Purge 2  Additionally, account creation today is suspended. Cadets, please continue to list your requested Usernames in ❗Create Account Requests, but we will not be able to begin fulfilling these requests again until Monday.  Good luck, and have fun trying to break stuff!

The announcement for Operation Purge 2.

Operation Purge 2 was planned to be stress test upon Beacon itself by its own users that would last for 24 hours. Everyone in the beta test was informed in advance that it was happening and were asked to plan their attack upon Beacon. It might be spammed levels, attempting to circumvent safety features, or attacking the server itself in a denial-of-service attack.

Lead Protector m88youngling predicted that the mission wouldn’t result in server downtime. However, the team was surprised by the results.

Website Downtime Detected

The stress test began at midnight, Sunday January 22nd. Five hours later, the website came under attack. The first attack lasted for a little over three and a half minutes. Later that hour, the website went down again for nearly five minutes. The outages continued to get longer and longer. The longest outage took place at 9:52 PM EST, lasting 25 minutes!

***    Beacon Outage Report: Core/Website    ***
*** Compiled at 11:59 PM EST Sun Jan 22 2023 ***

Filter: Date Ascending

[502 Bad Gateway] Jan 22, 2023 05:07:05 AM EST:  217 seconds
[502 Bad Gateway] Jan 22, 2023 05:53:22 AM EST:  293 seconds
[502 Bad Gateway] Jan 22, 2023 01:21:15 PM EST:  295 seconds
[502 Bad Gateway] Jan 22, 2023 05:36:56 PM EST:   10 minutes
[502 Bad Gateway] Jan 22, 2023 05:58:07 PM EST:   15 minutes
[502 Bad Gateway] Jan 22, 2023 06:23:46 PM EST:   10 minutes
[502 Bad Gateway] Jan 22, 2023 06:39:05 PM EST:   15 minutes
[502 Bad Gateway] Jan 22, 2023 06:59:25 PM EST:  299 seconds
[502 Bad Gateway] Jan 22, 2023 07:25:46 PM EST:   10 minutes
[502 Bad Gateway] Jan 22, 2023 08:18:27 PM EST:   15 minutes
[502 Bad Gateway] Jan 22, 2023 08:38:47 PM EST:  292 seconds
[502 Bad Gateway] Jan 22, 2023 08:49:37 PM EST:   10 minutes
[502 Bad Gateway] Jan 22, 2023 09:52:58 PM EST:   25 minutes

The above data was provided by R&D Technician Koko.

It wasn’t long before the issue was first reported and the attackers declared their success. In fact, it was an extremely simple script created by Metraberryy, who goes by the PSN name Catgirlfishing.

The script created by Metraberryy allows a user to quickly upload comments to the website over and over again. When multiple users run the script at once, it causes the website to slow down so much that no one can access it. This is a classic distributed denial-of-service (DDoS) attack. However, if only one user is running the script, it’s considered only a denial-of-service attack rather than a distributed one. The following video shows the Project Lighthouse console log’s view of the attack.

Alongside Hyperfied, Jazzkha11, Ratchet, and Despicable_Kee, and a separate script written by R&D Technician Koko, Metraberryy and company managed to accumulate over two hours of website downtime between 13 unique outages. Metraberryy earned herself a promotion to USC Surveyor for discovering the vulnerability!

Why Didn’t Rate Limiting Work?

To prevent attacks like this, Beacon has a server-wide program called Fail2Ban, a Python-based rate limiter and intrusion prevention program. If Fail2Ban is meant to prevent DDoS attacks, why didn’t it work here?

The answer lies within how Fail2Ban works and a particular failure of Project Lighthouse that needs to be corrected. The rate limiter works by monitoring server log files generated by Project Lighthouse. These files are just like the ones in the video above. As you can see, the log in the video generates a new line to let administrators know what the server is doing. In this case, a user is submitting hundreds of comments in quick succession.

A portion of Beacon’s log.

There are a few reasons why Fail2Ban may not be catching these requests. One issue may be that each line of the log doesn’t record the user’s IP address. This means that Fail2Ban isn’t associating the requests with a particular user, preventing it from rate limiting them. Another reason may be that Fail2Ban is searching for failed requests. This means that if a request appears to have succeeded, the rate limiter may find that it’s acceptable.

Implications of the Vulnerability

Because of this stress test, we’ve learned that it’s critical to find a solution to the issue before public beta. Otherwise, anyone could turn their computer into a Lighthouse-killing machine! In fact, that’s exactly what LBP Union R&D Developer Acidiclight did to test the implications of scripts created by Metraberryy and Koko.

Acidiclight’s script, similar to Koko’s, spams the server to try and access nonexistent webpages instead of using the same request over and over again. This further confuses the rate limiter. He jokingly refers to the script as ‘Nukehouse’. In fact, Acidiclight says that “Nukehouse can handle 4000 simultaneous connections.” In the wrong hands on multiple machines, scripts like these could render a Project Lighthouse website out of commission for an indefinite amount of time without proper security.

What About the Gameserver and API?

The website isn’t the only thing to worry about. Project Lighthouse is split into three components: the website, the gameserver, and the API. The gameserver is what handles user requests in the game. The API provides various connections to and from the server that allow certain features to work properly.

The gameserver and API appear to have been unaffected by attacks during Operation Purge 2. Only the website suffered downtime. This is a good thing for now, but in the future, more stress testing will be needed to discover vulnerabilities in Project Lighthouse.

Tracking Downtime

Thanks to the diligent efforts of R&D Technician Koko, we were able to track Beacon server downtime using her new status tool. You can check out the incident reports for yourself on our status page!

Koko wrote the following postmortem announcement for Operation Purge 2:

Postmortem: On January 22nd, 2023 – We conducted a stress-test on Beacon, our official Lighthouse instance. The purpose for this stress-test was to allow beta testers to contribute to the development of Lighthouse by allowing them to break and pen-test Beacon by methods of their liking. Out of the three core Beacon services, the most heavily affected service was the Website, suffering over 2 hours of downtime, and 13 unique outages.

What’s Next?

Beacon needs more testing before it’s ready for public beta. There have been serious discussions by leadership about what features are needed before public testing. As we have learned, improving rate limiting is an absolute must for the server. There will be further stress tests in the near future that will experiment with blocking attacks like these.

Contribute to Project Lighthouse!

There are various ways to help us with Project Lighthouse. If you’re a developer, we need your help! Help pave way for LittleBigPlanet’s future on PlayStation 3 and Vita by submitting a pull request on our Github repository for Project Lighthouse!

We also need help testing the server! You can sign up for the private beta waitlist by joining our Discord server. You’ll need to head to the settings channel to opt into the waitlist. We select new beta testers every weekend from that waitlist.

Thank you for your support! We can’t wait to bring you further updates about Project Lighthouse!