在本系列文章中，我们将开发一个静态内容网站原型. It will generate daily-updated, simple static HTML pages for popular GitHub repositories to track their latest releases. Static web page generation frameworks have great features to achieve that—we’ll use Gatsby.js, one of the most popular.

In Gatsby, there are many ways to collect data for a front end without having a back end (serverless), Headless CMS platforms and Gatsby source plugins among them. But we will implement a back end to store basic information about GitHub repositories and their latest releases. 因此，我们将完全控制我们的后端和前端.

Also, I’ll cover a set of tools to trigger a daily update of your application. 您也可以手动触发它，或者在某些特定事件发生时触发它.

我们的前端应用程序将在netflix上运行, 后端应用程序将使用免费计划在Heroku上运行. It will sleep periodically: “When someone accesses the app, the dyno manager will automatically wake up the web dyno to run the web process type.” So, we can wake up it via AWS Lambda and AWS CloudWatch. As of this writing, this is the most cost-effective way to have a prototype online 24/7.

Our Node Static Website Example: What to Expect

To keep these articles focused on one topic, I won’t be covering authentication, validation, scalability, or other general topics. 本文的编码部分将尽可能简单. The structure of the project and usage of the correct set of tools are more important.

In this first part of the series, we will develop and deploy our back-end application. In the second part, we will develop and deploy our front-end application, and trigger daily builds.

The Node.js Back End

The back-end application will be written in Node.js (not mandatory, but for simplicity) and all communications will be over REST APIs. 在这个项目中，我们不会从前端收集数据. (如果你有兴趣的话，可以看看 Gatsby Forms.)

First, we will start by implementing a simple REST API back end that exposes the CRUD operations of the repository collection in our MongoDB. Then we will schedule a cron job that consumes GitHub API v4 (GraphQL) in order to update documents in this collection. 然后我们将所有这些部署到Heroku云. 最后，我们将在cron作业结束时触发前端的重建.

The Gatsby.js Front End

在第二篇文章中，我们将重点介绍 createPages API. We will gather all repositories from the back end and will generate a single home page that contains a list of all repositories, 为返回的每个存储库文档添加一个页面. Then we’ll deploy our front end to Netlify.

From AWS Lambda and AWS CloudWatch

如果您的应用程序不休眠，则此部分不是强制性的. Otherwise, you need to be sure that your back end is up and running at the time of updating repositories. As a solution, you can create a cron schedule on AWS CloudWatch 10 minutes before your daily update and bind it as a trigger to your GET method in AWS Lambda. 访问后端应用程序将唤醒Heroku实例. 更多细节将在第二篇文章的末尾介绍.

下面是我们将要实现的架构:

Assumptions

我假设本文的读者具备以下方面的知识:

• HTML
• CSS
• JavaScript
• REST APIs
• MongoDB
• Git
• Node.js

It’s also good if you know:

• Express.js
• Mongoose
• GitHub API v4 (GraphQL)
• Heroku, AWS, or any other cloud platform
• React

---

让我们深入了解后端实现. We’ll split it into two tasks. The first one is preparing REST API endpoints and bind them to our repository collection. The second is implementing a cron job that consumes GitHub API and updates the collection.

Developing the Node.js静态站点生成器后端，第1步:一个简单的REST API

We will use Express for our web application framework and Mongoose for our MongoDB connection. If you are familiar with Express and Mongoose, you might be able to skip to Step 2.

(On the other hand, if you need more familiarity with Express you can check out the official Express starter guide; if you’re not up on Mongoose, the official Mongoose starter guide should be helpful.)

Project Structure

我们的项目的文件/文件夹层次结构将是简单的:

In more detail:

• env.config.js 环境变量是否为配置文件
• routes.config.js is for mapping rest endpoints
• repository.controller.js 包含在储存库模型上工作的方法
• repository.model.js 包含存储库和CRUD操作的MongoDB模式
• index.js is an initializer class
• package.json contains dependencies and project properties

Implementation

Run npm install (or yarn(如果您安装了Yarn)，然后将这些依赖项添加到 package.json:

{
  // ...
  "dependencies": {
    "body-parser": "1.7.0",
    "express": "^4.8.7",
    "moment": "^2.17.1",
    "moment-timezone": "^0.5.13",
    "mongoose": "^5.1.1",
    "node-uuid": "^1.4.8",
    "sync-request": "^4.0.2"
  }
  // ...
}

Our env.config.js file has only port, environment (dev or prod), and mongoDbUri properties for now:

module.exports = {
  "port": process.env.PORT || 3000,
  "environment": "dev",
  "mongoDbUri": process.env.MONGODB_URI || "mongodb://localhost/github-consumer"
};

routes.config.js contains request mappings and will call the corresponding method of our controller:

const RepositoryController = require('../controller/repository.controller');

exports.routesConfig = function(app) {

  app.post('/repositories', [
    RepositoryController.insert
  ]);

  app.get('/repositories', [
    RepositoryController.list
  ]);

  app.get('/repositories/:id', [
    RepositoryController.findById
  ]);

  app.patch('/repositories/:id', [
    RepositoryController.patchById
  ]);

  app.delete('/repositories/:id', [
    RepositoryController.deleteById
  ]);
};

The repository.controller.js file is our service layer. Its responsibility is to call the corresponding method of our repository model:

const RepositoryModel = require('../model/repository.model');

exports.insert = (req, res) => {
  RepositoryModel.create(req.body)
    .then((result) => {
      res.status(201).send({
        id: result._id
      });
    });
};

exports.findById = (req, res) => {
  RepositoryModel.findById(req.params.id)
    .then((result) => {
      res.status(200).send(result);
    });
};

exports.list = (req, res) => {
  RepositoryModel.list()
    .then((result) => {
      res.status(200).send(result);
    })
};

exports.patchById = (req, res) => {
  RepositoryModel.patchById(req.params.id, req.body)
    .then(() => {
      res.status(204).send({});
    });
};

exports.deleteById = (req, res) => {
  RepositoryModel.deleteById(req.params.id, req.body)
    .then(() => {
      res.status(204).send({});
    });
};

repository.model.js handles the MongoDb connection and the CRUD operations for the repository model. The fields of the model are:

• owner: The repository owner (company or user)
• name: The repository name
• createdAt: The last release creation date
• resourcePath: The last release path
• tagName: The last release tag
• releaseDescription: Release notes
• homepageUrl: The project’s home URL
• repositoryDescription: The repository description
• avatarUrl: The project owner’s avatar URL

const Mongoose = require('mongoose');
const Config = require('../config/env.config');

const MONGODB_URI = Config.mongoDbUri;

Mongoose.connect(MONGODB_URI, {
  useNewUrlParser: true
});

const Schema = Mongoose.Schema;

const repositorySchema = new Schema({
  owner: String,
  name: String,
  createdAt: String,
  resourcePath: String,
  tagName: String,
  releaseDescription: String,
  homepageUrl: String,
  repositoryDescription: String,
  avatarUrl: String
});

repositorySchema.virtual('id').get(function() {
  return this._id.toHexString();
});

// Ensure virtual fields are serialised.
repositorySchema.set('toJSON', {
  virtuals: true
});

repositorySchema.findById = function(cb) {
  return this.model('Repository').find({
    id: this.id
  }, cb);
};

const Repository = Mongoose.model('repository', repositorySchema);

exports.findById = (id) => {
  return Repository.findById(id)
    .then((result) => {
      if (result) {
        result = result.toJSON();
        delete result._id;
        delete result.__v;
        return result;
      }
    });
};

exports.create = (repositoryData) => {
  const repository = new repository (repositoryData);
  return repository.save();
};

exports.list = () => {
  return new Promise((resolve, reject) => {
    Repository.find()
      .exec(function(err, users) {
        if (err) {
          reject(err);
        } else {
          resolve(users);
        }
      })
  });
};

exports.patchById = (id, repositoryData) => {
  return new Promise((resolve, reject) => {
    Repository.findById(id, function(err, repository) {
      if (err) reject(err);
      for (let i in repositoryData) {
        repository[i] = repositoryData[i];
      }
      repository.save(function(err, updatedRepository) {
        if (err) return reject(err);
        resolve(updatedRepository);
      });
    });
  })
};

exports.deleteById = (id) => {
  return new Promise((resolve, reject) => {
    Repository.deleteOne({
      _id: id
    }, (err) => {
      if (err) {
        reject(err);
      } else {
        resolve(err);
      }
    });
  });
};

exports.findByOwnerAndName = (owner, name) => {
  return Repository.find({
    owner: owner,
    name: name
  });
};

This is what we have after our first commit: A MongoDB connection and our REST operations.

我们可以用下面的命令运行我们的应用程序:

node index.js

Testing

For testing, send requests to localhost:3000 (using e.g. Postman or cURL):

Insert a Repository (Only Required Fields)

Post: http://localhost:3000/repositories

Body:

{
  "owner" : "facebook",
  "name" :  "react"
}

Get Repositories

Get: http://localhost:3000/repositories

Get by ID

Get: http://localhost:3000/repositories/:id

Patch by ID

Patch: http://localhost:3000/repositories/:id

Body:

{
  "owner" : "facebook",
  "name" :  "facebook-android-sdk"
}

有了这些工作，是时候自动化更新了.

Developing the Node.js Static Site Generator Back End, Step 2: A Cron Job to Update Repository Releases

In this part, we will configure a simple cron job (which will start at midnight UTC) to update the GitHub repositories that we inserted to our database. We added only the owner and name parameters only in our example above, but these two fields are enough for us to access general information about a given repository.

为了更新我们的数据，我们必须使用GitHub API. For this part, it’s best to be familiar with GraphQL and v4 of the GitHub API.

We also need to create a GitHub access token. The minimum required scopes for that are:

这将生成一个令牌，我们可以用它向GitHub发送请求.

Now let’s go back to our code.

We have two new dependencies in package.json:

• "axios": "^0.18.0" 是一个HTTP客户端，所以我们可以请求GitHub API
• "cron": "^1.7.0" is a cron job scheduler

As usual, run npm install or yarn after adding dependencies.

We’ll need two new properties in config.js, too:

• "githubEndpoint": "http://api.github.com/graphql"
• "githubAccessToken": process.env.GITHUB_ACCESS_TOKEN (you’ll need to set the GITHUB_ACCESS_TOKEN 使用您自己的个人访问令牌的环境变量)

Create a new file under the controller folder with the name cron.controller.js. It will simply call the updateResositories method of repository.controller.js at scheduled times:

const RepositoryController = require('../controller/repository.controller');
const CronJob = require('cron').CronJob;

function updateDaily() {
  RepositoryController.updateRepositories();
}

exports.startCronJobs = function () {
  new CronJob('0 0 * * *'， function () {updateddaily ()}， null, true， 'UTC');
};

The final changes for this part will be in repository.controller.js. 为简洁起见，我们将其设计为一次更新所有存储库. 但是如果您有大量的存储库，您可能会超过 resource limitations of GitHub’s API. If that’s the case, you’ll need to modify this to run in limited batches, spread out over time.

The all-at-once implementation of the update functionality will look like this:

async function asyncUpdate() {

  await RepositoryModel.list().then((array) => {
    const promises = array.map(getLatestRelease);

    return Promise.all(promises);
  });
}

exports.updateRepositories = async function update() {
  console.log('GitHub Repositories Update Started');

  await asyncUpdate().then(() => {
    console.log('GitHub Repositories Update Finished');
  });
};

最后，我们将调用端点并更新存储库模型.

The getLatestRelease 函数将生成一个GraphQL查询并调用GitHub API. 来自该请求的响应随后将在 updateDatabase function.

updateddatabase (responseData, owner, name) {

  let createdAt = '';
  let resourcePath = '';
  let tagName = '';
  let releaseDescription = '';
  let homepageUrl = '';
  let repositoryDescription = '';
  let avatarUrl = '';

  if (responseData.repository.releases) {

    createdAt = responseData.repository.releases.nodes[0].createdAt;
    resourcePath = responseData.repository.releases.nodes[0].resourcePath;
    tagName = responseData.repository.releases.nodes[0].tagName;
    releaseDescription = responseData.repository.releases.nodes[0].description;
    homepageUrl = responseData.repository.homepageUrl;
    repositoryDescription = responseData.repository.description;

    if (responseData.organization && responseData.organization.avatarUrl) {
      avatarUrl = responseData.organization.avatarUrl;
    } else if (responseData.user && responseData.user.avatarUrl) {
      avatarUrl = responseData.user.avatarUrl;
    }

    const repositoryData = {
      owner: owner,
      name: name,
      createdAt: createdAt,
      resourcePath: resourcePath,
      tagName: tagName,
      releaseDescription: releaseDescription,
      homepageUrl: homepageUrl,
      repositoryDescription: repositoryDescription,
      avatarUrl: avatarUrl
    };

    await RepositoryModel.findByOwnerAndName(owner, name)
      .then((oldGitHubRelease) => {
        if (!oldGitHubRelease[0]) {
          RepositoryModel.create(repositoryData);
        } else {
          RepositoryModel.patchById(oldGitHubRelease[0].id, repositoryData);
        }
        console.log(`Updated latest release: http://github.com${repositoryData.resourcePath}`);
      });
  }
}

async function getLatestRelease(repository) {

  const owner = repository.owner;
  const name = repository.name;

  console.log(`Getting latest release for: http://github.com/${owner}/${name}`);

  const query = `
         query {
           organization(login: "${owner}") {
               avatarUrl
           }
           user(login: "${owner}") {
               avatarUrl
           }
           存储库(所有者:“${owner}”，名称:“${name}”){
               homepageUrl
               description
               release (first: 1, order: {field: CREATED_AT, direction: DESC}) {
                   nodes {
                       createdAt
                       resourcePath
                       tagName
                       description
                   }
               }
           }
         }`;

  const jsonQuery = JSON.stringify({
    query
  });

  const headers = {
    'User-Agent': 'Release Tracker',
    “授权”:“承载器${GITHUB_ACCESS_TOKEN}”
  };

  await Axios.post(GITHUB_API_URL, jsonQuery, {
    headers: headers
  }).then((response) => {
    return updateDatabase(response.data.data, owner, name);
  });
}

在第二次提交之后，我们将实现 一个cron调度程序，从我们的GitHub存储库获取每日更新.

We are nearly done with the back end. 但是最后一步应该在实现前端之后完成, so we’ll cover it in the next article.

将Node静态站点生成器后端部署到Heroku

在这一步中，我们将把应用程序部署到Heroku，因此 you’ll need to set up an account with them if you don’t have one already. If we bind our Heroku account to GitHub, it will be much easier for us to have continuous deployment. To that end, I’m hosting my project on GitHub.

登录到你的Heroku账户后，从仪表板中添加一个新应用:

Give it some unique name:

You will be redirected to a deployment section. Select GitHub as the deployment method, search for your repository, then click the “Connect” button:

为简单起见，您可以启用自动部署. 它会在你向GitHub仓库推送提交时部署:

Now we have to add MongoDB as a resource. 转到Resources选项卡并单击“查找更多附加组件”.” (I personally use mLab mongoDB.)

Install it and enter the name of your app in the “App to provision to” input box:

Finally, we have to create a file named Procfile at the root level of our project, which specifies the commands that are executed by the app when Heroku starts it up.

Our Procfile is as simple as this:

web: node index.js

Create the file and commit it. Once you push the commit, Heroku将自动部署您的应用程序, which will be accessible as http://[YOUR_UNIQUE_APP_NAME].herokuapp.com/.

为了检查它是否工作，我们可以发送与我们发送到的相同的请求 localhost.

Node.js、Express、MongoDB、Cron和Heroku:我们已经成功了一半!

After our third commit, this is what our repo will look like.

So far, we’ve implemented the Node.js/Express-based REST API on our back end, the updater that consumes GitHub’s API, and a cron job to activate it. 然后，我们部署了我们的后端，它稍后将为我们的 static web content generator 使用Heroku和钩子进行持续集成. Now you’re ready for the second part，在这里我们实现前端并完成应用程序!