You are about to create a download manager, which queues, starts and repeats over a list of downloadable content.

The example used is a download manager for albums of images.

The Rule

Album downloads are queued. They are not simultaneous, and this save us from server overload.

The Downloader object

Downloader is a singleton class, instantiated by the view component to listen for events.

import EventEmitter from 'events'
class DownloadEmitter extends EventEmitter {}
export default class Downloader {
  constructor() {
    this.downloadPath = 'PATH'

    if(!fs.existsSync(this.downloadPath)) {
      fs.mkdirSync(this.downloadPath)
    }

    this.downloadEmitter = new DownloadEmitter()
  }

  static getInstance() {
    if(!instance) {
      instance = new Downloader()
    }

    return instance
  }
  ...
}

In the constructor, we create the folder for downloads, initialize EventEmitter, etc.

But what is EventEmitter? Why do we emit events? View components are waiting for something to happend. Either it is a user action or background process result. Thanks to EventEmitter, Donwloader will process those variables to our views. Users will be happy to see the UI state changing because they only trust what they see.

The Queue vs The data

The queue will hold albums users are going to download. Once donwloaded, an album is removed from the queue.

Data can hold everything. But for simplicity sake, we represent it this way.

{ albums: [ { id: 1, images: [ { url: ‘http://i.imgur.com/aWaZqVs.jpg' }, … ] } … ] }

let queue = {}

getAlbum(index) {
  return queue[Object.keys(queue)[index]]
}

addToQueue(album, cb) {
  const self = this

  // XXX Save each album in a different location?
  let dir = path.join(self.downloadPath, (album.id).toString())
  album.dir = dir

  // Add properties to image donwload progress on a given album
  album.downloadStarted = false
  album.downloadProgress = 0
  album.downloadedImages = []

  queue[album.id] = album

  self.downloadQueuedAction(album.id)

  cb()
}

shiftQueue() {
  let album = this.getAlbum()
  delete queue[album.id]
}

Callback (cb) is a function parameter to execute when we are done with queueing. Callback can perform for example, UI update. Say, ‘Waiting…’

We will see more about ‘downloadQueuedAction’ later.

Download: the main part

Request is an awesome HTTP client for node. Its use is not limited to query an api, but to download data from the Internet, and much more.

In this section, we query the server for images, write them on disk, and move on.

Let’s start with download behaviour.

downloadRequestWithRetry(options, retry, started, progressed, finished) {
  let self = this
  var filename = (options.id).toString()
  var writeStream = fs.createWriteStream(path.join(options.dir, filename))

  let hasErrors = false
  request(options.file, {timeout: 15000})
  .on('response', function(data) {
    // Only init download started callback first time
    if(retry === 0) {
      started(parseInt(data.headers[ 'content-length' ]))
    }
  })
  .on('error', function(err) {
    // When an error occurs, re initiate download with retry incremented.
    // This supposes we are online.
    hasErrors = true
     if(retry < 5) {
      retry ++
      self.downloadRequestWithRetry(options, retry, started, progressed, finished)
    }
  })
  .on('data', function(chunk) {
    progressed(chunk.length)
  })
  .on('end', function() {
    // We don't call finished callback if an image is not successfully downloaded.
    if(!hasErrors) {
      finished()
    }
  })
  .pipe(writeStream)
}

I know it is a long method. But is it not simple enough? First thing that catch your attention is the name of the method. ‘Retry’. Well, if an image download fails due to bad networking, we re execute the same part of the function.

‘error’ and ‘end’ are both events, both called when request origins errors. ‘end’ event is he result of either successfull/failed download. That’s why we add a boolean to check if the download has been finished successfully. To retrieve the size of the image, we listens for ‘response’ event. Finally, ‘data’ event is triggered as the download progresses.

We need a starting point for our downloads. It looks this way:

downloadNext() {
  let self = this

  // Pick the first album on the queue
  let album = self.getAlbum(0)

  self.downloadEmitter.emit('album-download-started')

  let images = album.images

  // Variables to track download progress, totalSize equals to sum of
  // album images size. Received increments itself with every new chunk progress.
  let totalSize = 0
  let received = 0

  images.forEach(function(image) {
    let options = {
      id: (image.id).toString(),
      url: image.url,
      dir: album.dir
    }

    self.downloadRequestWithRetry(options, 0,
      ((size) => {
        totalSize += size
      }),
      ((data) => {
        received += data
        let progress = Math.floor(received / totalSize * 100)

        self.downloadEmitter.emit('album-download-progress', album.id, progress)
      }),
      (() => {
        album.downloadedImages.push(image.id)

        if(album.downloadedImages.length == images.length) {
          self.downloadEmitter.emit('playlist-download-finish', album.id)
          self.downloadFinishedAction()
        }
      })
    )
  })
}

We carefully notice the emitter part. We emit an events when we start downloading an album, to switch from waiting state to downloading. Then we emit the progress event to print progression percentage.

Starting point ‘downloadNext’ is called when first album is queued, or to process download for pending requests.

Methods ‘downloadQueuedAction’ and ‘downloadFinishedAction’ are there to initiate main download action over the queue object. You will notice that we also used underscore helpers to check if our queue object is empty. So don’t forget to import it.

donwloadQueuedAction() {
  const self = this
  if(isDownloading) {

  }else {
    self.downloadNext()
    isDownloading = true
  }
}

downloadFinishedAction() {
  const self = this
  self.shiftQueue()
  if(_.isEmpty(queue)) {
    isDownloading = false
  } else {
    self.downloadNext()
  }
}

Conclusion

We talked about components and views a bit. We could have used React or Vue.js, two awesome libraries for constructing views. Maybe we will add the UI part in the future.

Doing that way with downloads, we considerably avoided error responses from the server. So it is much safer to process download of one album, and queue others.