You have unlimited access as a PRO member
You are receiving a free preview of 3 lessons
Your free preview as expired - please upgrade to PRO
- Advanced Techniques in NoSQL
- Group Collection Query
- Shopping Cart + Ecommerce NoSQL Model
- User Follow/Unfollow System
- Threaded Comments or Hierarchy Tree Structure
- Tags or HashTags
- The End
- Object Oriented Programming With TypeScript
- Angular Elements Advanced Techniques
- TypeScript - the Basics
- Cloud Scheduler for Firebase Functions
- Testing Firestore Security Rules With the Emulator
- How to Use Git and Github
- Infinite Virtual Scroll With the Angular CDK
- Build a Group Chat With Firestore
- Async Await Pro Tips
Advanced Data Modeling With Firestore by ExampleEpisode 86 written by Jeff Delaney
In the previous lesson, we learned the fundamentals of relational data modeling with Firestore. Today I want to push further and look at several more practical examples. In addition to data modeling, we will look at techniques like duplication, aggregation, composite keys, bucketing, and more.
Keep in mind, Firestore is still in beta. Firebase engineers hinted at some really cool features on the roadmap (geo queries, query by array of ids) - I’ll be sure to keep you posted :)
Source code available to pro members. Learn more
Full source code for the threaded comments project.
There are several fundamental techniques at your disposal for managing complex NoSQL structures. Let’s talk briefly about each one of them.
Data duplication is a very common technique to eliminate the need to read multiple documents. Simple example - we might duplicate or embed a username on every tweet doc to avoid making secondary query to the user doc. Or we might duplicate 20 recent tweets on the user document to show on the user profile. This strategy results in faster reads, but slower writes. Think about it - we can read all data in a single document, but may need to update multiple documents when the embedded data changes.
Data aggregation is the process of analyzing a collection of data, then saving the results on some other document. The simplest example would be saving a count of the total documents in a collection. Normally, Firestore aggregation is done server side via Cloud Functions.
A composite key is simply the combination of two or more unique document ids, for example
userXYZ_postABC. This is especially useful for modeling relationships in denormalized structures because it can enforce a unique relationship between the two documents.
Bucketing is a form of duplication/aggregation that breaks collections into single documents. Using Twitter as an example, let’s imagine we have a collection of tweets, but want to bucket a certain user’s tweets month-by-month. This would allow us to read tweets for a given month very efficiently, but the drawback is some additional bookkeeping to ensure all data stays in sync when updates occur on the source document (Twitter probably has good technical reasons for not allowing you to update tweets).
In many NoSQL databases, you must shard to scale. Sharding is just the process of breaking the database down into smaller chunks (horizontal partitioning) to increase performance.
In Firestore, sharding is handled automatically. The the only scenario where you may need to control sharding is when you consistently have many write operations occurring at intervals of less than 1s. Imagine the compute requirements of updating the like count on a new tweet from Selena Gomez.
Another cool feature in the Firebase SDK is the ability to make read requests in a non-blocking manner called pipelining as explained by Frank van Puffelen. When you query Firestore, you don’t need to wait for response A to send request B. You can send all requests individually and Firebase will respond with data as soon as it becomes ready.
Pipelining isn’t a data structuring technique, but it drives our decision-making process.
Let’s imagine you have an array of document ids. You can pipeline each request from a child component by looping over the ids, then performing a document read from the child component, i.e
afs.doc('items/' + id). Because the requests are non-blocking, there’s no major performance hit for structuring your app this way.
A group collection query occurs when you want to query a common subcollection across its parent owners. For example, you might to get blog posts for all users who wrote a post categorized as Angular.
It is not possible to make this query via the subcollection. The easy solution is to denormalize posts to a root collection, but if that’s not possible here’s plan B…
First, embed some duplicated data on the parent. When a new post is created in the subcollection, we update the
categoriesUsed object on the parent doc where the category is the key.
This opens the door to make a query like so:
users.where('categoriesUsed.angular', '==', true);
At this point we have all the users who posted to Angular, then we can query each user’s posts subcollection individually and join the data client-side.
Building an ecommerce solution is no simple task - that goes for both SQL and NoSQL. In this example, I show you how to model a shopping cart with basic inventory management. Keep in mind, you are likely to have other concerns, such as payments, returns, etc.
- One-to-One: User to Cart
- Many-to-Many: Products to Users (via Cart)
- One-to-Many: User to Orders
Shopping carts work great with Firebase Anonymous auth when your users can checkout as a guest.
The challenges are as follows:
duct prices may change and should be reflected in the cart.
duct inventory stock is limited.
Users (root collection): Basic user data
Products (root collection): Product data and current inventory.
Carts (root collection): A one-to-one relationship is created by setting the
userID === cartID for documents in either collection. When an order placed, the cart data can be cleared out because there is no need for a user to have multiple carts.
Orders (user subcollection): When an order is created and confirmed, you can run a cloud function to decrement the product availability.
Naturally, let’s use Twitter as our example here. We can take advantage of composite keys to manage the relationship in its own collection. Using a unique ID of
followerID_followedID is like saying userFoo follows userBar.
Users (root collection): Basic user data with aggregated follow counts.
Relationships (root collection): This works similar to a intermediate table in SQL, with a composite key to enforce uniqueness for each user to user relationship.
Does UserA follow UserB:
Get a user’s 50 most recent followers:
Get all users that are being followed
db.collection('relationships').where('followerId', '==', userId);
Let’s look at how we might model a tree structure that goes multiple levels deep - think threaded comments on Hacker News or a directory of subcategories. In fact, I am modeling my data after Hacker News which is hosted on Firebase RTDB.
This implementation is designed to take advantage of pipelining in Firebase - instead of requesting a single collection, we will request a bunch of documents individually.
In the future, Firestore may support a queries based on an array of doc IDs
First start by querying the root comments with
comments.where('parent', '==', null), then pass them to our comment component.
<app-comment *ngFor="let comment of comments | async"
This allows us to build a recursive component (a component that calls itself) to render out a tree. In other words, while the comment document has children, it will continue rendering new branches of the threaded comment tree. It would also be easy to lazy load each branch by adding “view replies” button before loading the children.
Going back to the Twitter theme, let’s now try to model hashtags in Firestore. Hashtags must be a single continuous string and should be case insensitive. This means we can use the tag itself as the document ID. For example, a hashtag of
#AngularFire would have a doc ID of
Whenever a new tweet is created, you can use a cloud function to aggregate the
postCount property on the tag document.
Get all tweets with a specific tag:
db.collection('tweets').where('tags.angular', '==', true);
Get the most popular tags:
Get a specific tag:
tagId = 'SomeCoolTag'.toLowerCase();
I love data modeling. If you want to talk about your own complex data structure challenges, please reach out on Slack. In the future, I will take some of these models and build out entire features that you can drop into any Angular app.