GDG Hanoi photo gallery

High-level design components

What I’ve already had

Let’s start with listing the existing resources before jumping into what components I should use:

• 1 existing frontend site: gdghanoi.com hosting on Google Cloud Run.
• Approximately 800 USD in Google Cloud credits, which will expire in August of this year.

I pick Google Cloud as the primary cloud provider for other components of the project.

For the region, the service mostly serves clients in Vietnam, so single region deployments is enough. Based on the Google Cloud latency dashboard, I choose Taiwan (asia-east1) as the primary region of the service to minimize the latency to Vietnam.

By deploying components in the same region, I can ensure the communication between them experiences near-zero latency. It also helps to reduce the risk of cross-region latency and cross-region bandwidth cost. It is important because resizing photos may require a high bandwidth of data transfer between storage and workers.

Back-of-the-envelope metrics

It is obvious that the primary entity of the service is a photo.

Assuming that:

• An original photo can be up to 10MB, 6000x4000 pixels, 240 pixels/inch, and in general formats (PNG or JPEG).
• The WebP format can reduce up to 85% of the file size.

I will resize and format it into 9 WebP versions: 320, 512, 768, 1024, 1280, 1600, 2048, 2560, and original size.

Let me note down some metrics that I have in mind about it.

Building blocks

Storage

I need blob storage to store the photos. Here, I choose to use Google Cloud Storage (GCS), which provides me utilities:

• Public URLs for serving photos, as long as I choose to open public access to the bucket.
• Notification subscription for each file-uploaded event. This feature helps simplify the image processing trigger.
• Its free tier is generous enough for the number of GDG Hanoi photos.

Database

The database is where I store photos’ metadata, including object IDs, event labels, public URLs, upload timestamps, etc.

I choose to use Google Cloud Firestore (Firebase Firestore). The reasons are:

• Its design is optimized to read-heavy applications.
• I can get an (almost) free database with the most sufficient performance for this small-scale project, such as:
  • It is a serverless document database, so I don’t have to deploy it on a dedicated machine.
  • Its architecture is built on top of GCP, so it takes advantage of the network to provide access with minimal latency for the GDG Hanoi’s web, which is hosted on GCP already.
  • It also provides backup and recovery features and auto-scales horizontally, so I don’t have to worry about the database scalability and availability.

The only drawback that I can come up with is that Firestore does not support offset-based pagination queries, so I have to implement the schema more manually to support the paging mode properly. I will dive deeper into the schema in the next post.

Photo worker

This worker is essential to the responsive image capability of the gallery.

Because the logic may be varied, I will write dedicated software to perform this specific task.

The major process is image resizing, which is compute-intensive. Currently, the actual resource consumption for me is unknown. Thus, I choose to containerize it and deploy on a computing instance so that I can customize and evaluate the resource usage easily.

Message queue

Under my observation, for each new photo uploaded to GCS, the worker has to acknowledge the event so it can start downloading the newly uploaded file and create various sizes of it.

Therefore, I have to make the worker “listen” to the GCS events. As a consequence, I have to set up a Pub/Sub queue to play the role of a communication channel between GCS and the worker.

Uploader frontend

The media team needs a web interface to upload photos in batches. I need to provide a dedicated web interface to let them:

• Specify an event to upload.
• Choose multiple photos to upload at once.
• View the status of the upload.
• Reflect the photo lists with the gallery pages.

Gallery backend & frontend

The GDG Hanoi website is built on Next.js; therefore, I’ve already had a server that performs 2 tasks:

• Serve the static pages & server-rendered React components.
• Serve some simple HTTP REST API routes.

I will let the web connect to Firestore directly on the server side and deliver to the client side the gallery pages and open APIs for fetching photos. That will minimize the required efforts to implement the gallery pages.