Week 5
Data persistance and intro to MongoDB

# Agenda

• KPI Survey (30 min)
• AMA (10 min)
• Quiz (10 min)
• Break (10 min)
• EX5-1 File system (15 min)
• Relational Databases v.s. Document Databases (20 min)
• EX5-2 Install MongoDB & Compass (15 mins)
• Break (10 mins)
• EX5-3 Connect with Mongoose (30 mins)

# File system

Data persistance is about saving the last know state of all data elements in our application. Ultimately, this means saving it to a file on disk somewhere. This could be a local JSON file. It could be a cloud storage service like Amazon's S3 or Google Cloud Storage. More often it is handled by an intermediate database service.

# Synchronize cars.json

We are going to focus mostly on the database scenario for the rest of the course, but let's examine the local JSON file example for just a minute. If we examine our in-class project from last week, every time we restarted the application our Cars collection reset to the initial four objects loaded from our cars.js file. With just a few extra lines of code, we could ensure that the current state of our application data was saved every time one of the route handlers is called.

# Convert cars.js to cars.json

[
  {
    "id": 1,
    "make": "Tesla",
    "model": "S",
    "colour": "Black"
  },
  {
    "id": 2,
    "make": "Tesla",
    "model": "3",
    "colour": "Indigo"
  },
  {
    "id": 3,
    "make": "Tesla",
    "model": "X",
    "colour": "Brown"
  },
  {
    "id": 4,
    "make": "Tesla",
    "model": "Y",
    "colour": "Silver"
  }
]

# Create new utility function

In our carsRouter.js module, we will need to import the fs core node module -- let's use the Promises version.

Then create a new function called writeJsonData that takes a JavaScript data structure as an argument. The function will open our /data/cars.json in write mode, and then write out the data argument converted to JSON format.

Wrap the logic in a try/catch block to handle any errors, and ensure the filehandle gets closed by using the finally clause.

const fs = require('fs').promises

const writeJsonData = async data => {
  let filehandle
  try {
    filehandle = await fs.open(`../data/cars.json`, 'w')
    await filehandle.writeFile(JSON.stringify(data))
  } catch (err) {
    console.log(`Problem writing cars.json`, err)
  } finally {
    if (filehandle !== undefined) await filehandle.close()
  }
}

# Databases

Most real world applications need to store structured data that reflects the state of the application and is selectively searchable, with appropriate authorization credentials. This is the job of a database.

Traditionally databases systems have been based on a series of tables, like a spreadsheet, with the columns representing the properties on a record and each row containing the complete set of properties for that unique record.

# Relational Databases

As we build applications that follow an Object Oriented Design, each table maps to a collection of objects of a given class. Often several of these objects will have some relationship to another. For example in a blog application, an Article will have an Author. Article and Author represent two classes of objects and there is a relationship between specific instances of those objects.

We have typically solved the storage of this kind of data with Relational Databases -- invented be a guy named Ted Codd at IBM back in the early 1970s.

The standard API for interacting with a relational database is the Structured Query Language (SQL). Some of the most common relational database systems are:

• MySQL / MariaDB
• PostgreSQL
• Oracle DB

# Document Databases

About 10 years ago, along with the development of Node.js people started wondering if there was a different kind of database model that was a better fit for the rapidly iterating development cycle of the JavaScript world. Many variations on the Document Database model also know as NoSQL Databases came into being. All of them centered on the JSON document as the storage model.

The biggest attraction with NoSQL databases is the flexibility. There is no need to predefine a table schema as there is with SQL databases. Some examples of NoSQL databases are:

• Google Firestore
• AWS Dynamo
• CouchDB
• MongoDB

# Relational DB v.s. Document DB

# Schema

A database schema, means to define the shape of the data that we want to store. With SQL databases, this must be defined in advance of storing data. With document databases, there is not database imposed schema. If you can put it in a JSON document, you can store it in the database.

You will sometimes hear NoSQL databases referred to as schema-less. This does not mean that our application has no structured data model. It just shifts the responsibility for enforcing it to the application code rather than the database.

Let's look at how we would represent the data for a simple attendance tracking application.

# SQL

# MongoDB

Embedded data structures to model relationships. Although there is some benefit to this flexibility for rapid prototyping, there are trade-offs. The biggest one is synchronization of duplicate data. It takes just as much careful planning to build a robust and scalable NoSQL database.

Students

students: [
  {
    _id: 201800001,
    firstName: 'Mickey',
    lastName: 'Mouse',
    nickName: 'Steamboat Willie',
    email: 'mickey.mouse@disney.com',
    githubId: null,
    birthDate: '1928-11-18',
    imageUrl: null
  },
  {
    _id: 201800002,
    firstName: 'Minerva',
    lastName: 'Mouse',
    nickName: 'Minnie',
    email: 'minnie.mouse@disney.com',
    githubId: null,
    birthDate: '1928-11-18',
    imageUrl: null
  }
]

Courses

{
  courses: [
    {
      _id: 1,
      code: 'MAD9124',
      title: 'Mobile API Development',
      description: `Students enhance their JavaScript skills to become productive with Full-stack development. They use a hands-on approach to build APIs using Node.JS and a variety of tools, frameworks, libraries and packages. The creation of these modern APIs also requires the students to develop skills with persistent scalable database storage systems. Project work culminates with students creating APIs to be used with websites and mobile applications. Students work individually as well as with other students to complete tasks.`,
      sections: [
        {
          _id: 'mad9124w19s300',
          number: 300,
          name: 'Alpha Team',
          students: [201800001, 201800002],
          classes: [
            {
              _id: 1,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 2,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 3,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001]
            },
            {
              _id: 4,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 5,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800002]
            },
            {
              _id: 6,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 7,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 8,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 9,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 10,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 11,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 12,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 13,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            },
            {
              _id: 14,
              room: 'WT127',
              startTime: '',
              endTime: '',
              students: [201800001, 201800002]
            }
          ]
        },
        {
          _id: 'mad9124w19s301',
          number: 301,
          name: 'Bravo Team',
          students: [],
          classes: [
            {_id: 21, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 22, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 23, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 24, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 25, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 26, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 27, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 28, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 29, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 30, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 31, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 32, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 33, room: 'WT127', startTime: '', endTime: '', students: []},
            {_id: 34, room: 'WT127', startTime: '', endTime: '', students: []}
          ]
        }
      ]
    }
  ]
}

# Hands on with MongoDB

# Install MongoDB

There are several ways to get MongoDB running in your local development environment.

1. Download the runtime installer from the MongoDB website.
2. Use Homebrew to install MongoDB.
3. Use Docker and run it in a container.

For this course, we will use the Docker container option. This has the benefit of letting us use different versions for different project and keep the test/development data isolated between project. We will create a separate container with it's own data folder inside each project.

# Docker Compose

Because this will be running only in our development environment, we don't need to worry about complicated security and user authentication setup.

In your project folder, create a new file called docker-compose.yml with the following content.

version: '3.1'

services:
  mongo:
    image: mongo:bionic
    restart: always
    volumes:
      - ./data/mongo:/data/db
    ports:
      - '27017-27019:27017-27019'

Now using the terminal in your project folder, start your local MongoDB server with ...

docker-compose up -d

Verify that the MongoDB container is running ...

docker-compose ps

You should see something like ...

    Name                  Command             State                                      Ports
----------------------------------------------------------------------------------------------------------------------------------
week5_mongo_1   docker-entrypoint.sh mongod   Up      0.0.0.0:27017->27017/tcp, 0.0.0.0:27018->27018/tcp, 0.0.0.0:27019->27019/tcp

To shut it down, simply run ...

docker-compose down

# Install MongoDB Compass

If you have worked with MySQL databases in the past you probably used a tool like phpMyAdmin, MySQL WorkBench or Sequel PRO to manage and query your databases. MongoDB has a similar tool call Compass. You can read all about its capabilities on at MongoDB.com.

Lets download and install it from the MongoDB Download Center. Make sure that you select the latest stable 'community edition'. That is the free one.

With the correct version selected, click the big green download button.

# Running MongoDB locally

To run MongoDB in our local development environment, we need to start the docker container. Open the terminal in your project folder and run this command ...

docker-compose up -d

This starts the MongoDB server as a daemon process running in the Docker container. It will be accessible to MongoDB Compass (and any other client application) on port 27017 - that is the default port for MongoDB.

Now open the MongoDB Compass app and connect using all the default settings.

# Connect to MongoDB from Node.js

In our Node.js app, we need to open a connection to the MongoDB database so that we can store and retrieve our resource objects. MongoDB provides a official NPM package which we can use to interact with the database. To install it in your project, simply run npm install mongodb in the terminal.

However, for this course we will instead be using a popular data modeling library for MongoDB called Mongoose.

# Mongoose Quick Start

npm install mongoose

Create a new file in your project folder called mongooseTest.js. We will give the mongoose.connect method the connection string for our local MongoDB at localhost:27017 and then append the name of the database we want to connect to at the end: /test.

const mongoose = require('mongoose')
mongoose.connect('mongodb://localhost:27017/test', {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

const Cat = mongoose.model('Cat', {name: String})

const kitty = new Cat({name: 'Spot'})
kitty.save().then(savedKitty => console.log(`${savedKitty.name} says 'meow'`))

The mongoose.model method is a class constructor that creates a resource class that we can use to interact with the database. In the example above we are giving it two properties: 'Cat' is the name of the class of objects to store in a collection in the database, and {name: String} is the expected shape of each 'Cat' document.

An instance of a Model is a Document.

OK. Lets run it and then check the results in MongoDB Compass.

node mongooseTest.js

Let's try adding another Cat instance to the collection. Change the name on line three and rerun it.

const Cat = mongoose.model('Cat', {name: String})

const kitty = new Cat({name: 'Patches'})
kitty.save().then(savedKitty => console.log(`${savedKitty.name} says 'meow'`))

Let's try adding an age to our next Cat.

const Cat = mongoose.model('Cat', {name: String})

const kitty = new Cat({name: 'Callie', age: 6})
kitty.save().then(savedKitty => console.log(`${savedKitty.name} says 'meow'`))

This time, when we check it in Compass the age was not saved. Our Mongoose Model automatically discards any properties that are not predefined. This time change the model schema as well.

const Cat = mongoose.model('Cat', {name: String, age: Number})

const kitty = new Cat({name: 'Callie', age: 6})
kitty.save().then(savedKitty => console.log(`${savedKitty.name} says 'meow'`))

We can also insert many objects into the database at once.

const Cat = mongoose.model('Cat', {name: String, age: Number})

const newCats = [
  {name: 'Fluffy', age: 2},
  {name: 'Pink Spots', age: 12},
  {name: 'Tabatha', age: 16}
]
Cat.insertMany(newCats).then(docs => console.log(docs))

We have added many documents to our collection. How do we retrieve them? Add a new line to find all of the cats from the database.

Cat.find().then(result => console.log({listAll: result}))

How about listing just the Cats with the name 'Spot'?

Cat.find({name: 'Spot'}).then(result => console.log({spot: result}))

That found all of the cats with the exact name 'Spot'. But, what if we wanted to check for names like Spot? We can use regular expressions to match patterns. Replace the quotes with slashes. Now it will match 'Spot', 'Spots', 'Spotty', or 'Pink Spots'.

Cat.find({name: /Spot/}).then(result => console.log({regex: result}))

Regular expressions can be a very powerful tool. Checkout the resources linked below to learn more.

Putting it all together, the final version of our mongooseTest.js should look like this.

'use strict'

// Import the Mongoose library
const mongoose = require('mongoose')

// Connect the mongoose instance to our MongoDB
mongoose.connect('mongodb://localhost:27017/test', {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

// Define the document model for the 'cats' collection
const Cat = mongoose.model('Cat', {name: String, age: Number})

// Create a new Cat document and save it to the database
const kitty = new Cat({name: 'Callie', age: 6})
kitty.save().then(savedKitty => console.log(`${savedKitty.name} says 'meow'`))

// Directly insert an array of Cat property objects
const newCats = [
  {name: 'Fluffy', age: 2},
  {name: 'Pink Spots', age: 12},
  {name: 'Tabatha', age: 16}
]
Cat.insertMany(newCats).then(docs => console.log(docs))

// List all of the Cats in the collection
Cat.find().then(result => console.log({listAll: result}))

// List only the Cats with 'Spot' in the name
Cat.find({name: /Spot/}).then(result => console.log({regex: result}))

There is a lot more to learn about how to use Mongoose and we will continue next week. In particular we will look at how to integrate Mongoose with our carsRouter.js from last week. In preparation, please review the reading material below.

# EX5 Setup MongoDB

Submit a screenshot of MongoDB Compass showing the connection to localhost:27017/test.cats and the listing of the documents in the collection after our test transactions.

# For next week

Before next week's class, please read these additional online resources.

• Mongoose Docs: Models
• Mongoose Docs: Documents
• Mongoose Docs: Queries
• Regular Expressions (regex) video
• Regex tutorial — A quick cheat sheet by examples
• Intro to Regex video
• RegEx in JS video