You have unlimited access as a PRO member
You are receiving a free preview of 3 lessons
Your free preview as expired - please upgrade to PRO
- Step 1 - Initial Setup
- Step 2 - Parsing Raw Data
- Step 3 - Building your own CLI Tool in Node
- The End
- Object Oriented Programming With TypeScript
- Angular Elements Advanced Techniques
- TypeScript - the Basics
- Cloud Scheduler for Firebase Functions
- Testing Firestore Security Rules With the Emulator
- How to Use Git and Github
- Infinite Virtual Scroll With the Angular CDK
- Build a Group Chat With Firestore
- Async Await Pro Tips
CSV to Firestore - Build a Firebase Admin Command Line UtilityEpisode 95 written by Jeff Delaney
If you want to migrate your existing database to Firestore, you might be wondering… How do I import JSON or CSV to Firestore?. In this lesson, I will provide you with a reliable pipeline for reading raw data and parsing it to a writable format for Firestore.
Everybody has different data modeling needs, so I will teach you how to quickly roll out your own NodeJS CSV-to-Firestore command line import tool. Its purpose is to take your local files (CSV or JSON format in this case), parse it to an array of JS objects, then use the Firebase admin SDK to perform an atomic batched write. Not only will this handle migrations, but it will also give you a solid development utility that you can extend with additional functionality.
CSV format was chosen for this demo because it is commonly used as an export format for relational SQL databases and MS Excel spreadsheets. If you want the opposite, watch episode 69 to learn how to export Firestore to CSV.
Source code available to pro members. Learn more
Full source code for the Firestore migrator CLI tool.
Let’s go ahead and create an empty folder and initialized NPM.
You can write this utility in vanilla JS, but TypeScript will dramatically improve tooling and productivity long-term.
npm install -D typescript
Now update your tsconfig.json with the following content. Basically, we’re just telling TS to compile our code in the /dist folder as commonjs that can be understood by NodeJS.
At this point, you can run
tsc from the command line to compile your code.
npm install firebase-admin --save
You will need to download your service account from the Firebase admin console. Save it in the root of this project and name it credentials.json. This will give your local app full access to the project and bypass all security rules.
Make sure to keep your credentials private. If using git, add the line
credentials.json to your .gitignore file.
Commander.js is a tool that makes it easy to parse command line arguments in Node. We will use it to pass a local file path and Firestore collection path argument to the migration command.
npm install commander --save
Lastly, let’s install FS Extra to interact with the local file system. When it comes to CSV, there are a bunch of different Node packages, but CSVtoJSON works especially well for this task because it has a callback that emits each row from the spreadsheet as JSON.
npm i --save csvtojson fs-extra
The final initialization logic should look something like this.
Reading a raw JSON file is an easy one-liner thanks to fs-extra.
Reading a CSV is a bit more work. CSVtoJSON uses callbacks, but we will Promisify it in the next section. It gives us a handful of listeners that emit data when a row or document is finished processing.
While it’s possible to process command line arguments in Node without any dependencies, I highly recommend the Commander.js package to make life easier for your team.
Here’s how we want our CLI command to work:
fire-migrate --src bunnies.csv --collection animals
It should read the CSV source file, then write each row as a document in Firestore. Accessing arguments from the command is as simple as defining them as an options.
As an added bonus, we get instant documentation for the CLI tool.
Now it’s time to put everything together into a CLI tool that we can actually use.
To connect our Node executable file to the local command line PATH, we need to register it in the bin object in
You can compile the source code and link the command by running:
tsc && npm link
As you can see, creating your own CLI tool for Firebase development is pretty simple. You can expand on this code to build additional utilities that increase your development productivity.