Node cooperates with WebSocket to download multiple files and send back progress

cause

Why do you make this thing? I suddenly heard from a back-end colleague Annie This thing, found that this thing download video very convenient, will automatically crawl the video in the web page, and then organize into a list. After the command is executed, it looks like the following:

I thought about it in my mind. Let's play the whole interface. Then it's like this.

list

Download list

Address warehouse: https://github.com/Rynxiao/yh-tools , if you like, welcome star

Technology involved

  • Express backend services
  • Webpack modular compilation tool
  • Nginx mainly compresses the file gzip (only discard nginx if there is a problem in adding gzip to Express)
  • Ant Design front-end UI Library
  • React + React Router
  • WebSocket progress callback service

There is also a little episode. At the beginning, I started a nginx service with docker, but I found that there was always a problem with internal forwarding, and there was also a problem with obtaining host IP, and then I gave up for a long time. (docker research is not deep, please understand)

Download some details

First, the browser will connect to the WebSocket server. At the same time, there is a Map of all clients on the WebSocket server. The browser generates a uuid as the browser client id, and then stores the link as a value in the Map.

Client:

// list.jsx
await WebSocketClient.connect((event) => {
  const data = JSON.parse(event.data);
  if (data.event === 'close') {
    this.updateCloseStatusOfProgressBar(list, data);
  } else {
    this.generateProgressBarList(list, data);
  }
});

// src/utils/websocket.client.js
async connect(onmessage, onerror) {
  const socket = this.getSocket();
  return new Promise((resolve) => {
    // ...
  });
}

getSocket() {
  if (!this.socket) {
    this.socket = new WebSocket(
      `ws://localhost:${CONFIG.PORT}?from=client&id=${clientId}`,
      'echo-protocol',
    );
  }
  return this.socket;
}

Server:

// public/javascript/websocket/websocket.server.js
connectToServer(httpServer) {
  initWsServer(httpServer);
  wsServer.on('request', (request) => {
    // uri: ws://localhost:8888?from=client&id=xxxx-xxxx-xxxx-xxxx
    logger.info('[ws server] request');
    const connection = request.accept('echo-protocol', request.origin);
    const queryStrings = querystring.parse(request.resource.replace(/(^\/|\?)/g, ''));
    
    // Every time a connection is connected to the websocket server, the current connection is saved to the map
    setConnectionToMap(connection, queryStrings);
    connection.on('message', onMessage);
    connection.on('close', (reasonCode, description) => {
      logger.info(`[ws server] connection closed ${reasonCode} ${description}`);
    });
  });

  wsServer.on('close', (connection, reason, description) => {
    logger.info('[ws server] some connection disconnect.');
    logger.info(reason, description);
  });
}

Then when clicking download on the browser side, two main fields resourceid (composed of parentId and childId in the code) and bClientId generated by the client will be passed. What's the use of these two IDS?

  • Each time you click download, a WebSocket client will be generated in the Web server, so the resourceid is the key value of the WebSocket server generated in the server.
  • bClientId is mainly used to distinguish the browser client. Considering that there may be multiple browser accesses at the same time, when messages are generated in WebSocket server, this id can be used to distinguish which browser client should be sent

Client:

// list.jsx
http.get(
  'download',
  {
    code,
    filename,
    parent_id: row.id,
    child_id: childId,
    download_url: url,
    client_id: clientId,
  },
);

// routes/api.js
router.get('/download', async (req, res) => {
  const { code, filename } = req.query;
  const url = req.query.download_url;
  const clientId = req.query.client_id;
  const parentId = req.query.parent_id;
  const childId = req.query.child_id;
  const connectionId = `${parentId}-${childId}`;

  const params = {
    code,
    url,
    filename,
    parent_id: parentId,
    child_id: childId,
    client_id: clientId,
  };

  const flag = await AnnieDownloader.download(connectionId, params);
  if (flag) {
    await res.json({ code: 200 });
  } else {
    await res.json({ code: 500, msg: 'download error' });
  }
});

// public/javascript/annie.js
async download(connectionId, params) {
    //...
  // When annie downloads, it will monitor the data. Here, it will use throttling to prevent the progress from returning too fast and the websocket server cannot respond
  downloadProcess.stdout.on('data', throttle((chunk) => {
    try {
      if (!chunk) {
        isDownloading = false;
      }
      // The main task here is to parse the data, and then send the progress and speed information to the websocket server
      getDownloadInfo(chunk, ws, params);
    } catch (e) {
      downloadSuccess = false;
      WsClient.close(params.client_id, connectionId, 'download error');
      this.stop(connectionId);
      logger.error(`[server annie download] error: ${e}`);
    }
  }, 500, 300));
}

After receiving the progress and speed messages, the server sends them back to the client. If the progress reaches 100%, it will delete the websocket client from the server in the server, and send a notice that the client is closed, informing the browser that the download is complete.

// public/javascript/websocket/websocket.server.js
function onMessage(message) {
  const data = JSON.parse(message.utf8Data);
  const id = data.client_id;

  if (data.event === 'close') {
    logger.info('[ws server] close event');
    closeConnection(id, data);
  } else {
    getConnectionAndSendProgressToClient(data, id);
  }
}

function getConnectionAndSendProgressToClient(data, clientId) {
  const browserClient = clientsMap.get(clientId);
  // logger.info(`[ws server] send ${JSON.stringify(data)} to client ${clientId}`);

  if (browserClient) {
    const serverClientId = `${data.parent_id}-${data.child_id}`;
    const serverClient = clientsMap.get(serverClientId);

    // Send progress and speed from web server to browser
    browserClient.send(JSON.stringify(data));
    // If the progress has reached 100%
    if (data.progress >= 100) {
      logger.info(`[ws server] file has been download successfully, progress is ${data.progress}`);
      logger.info(`[ws server] server client ${serverClientId} ready to disconnect`);
      // Remove the current websocket client created by the web server from the clientsMap
      // Then close the current connection
      // Send the download completed message to the browser at the same time
      clientsMap.delete(serverClientId);
      serverClient.send(JSON.stringify({ connectionId: serverClientId, event: 'complete' }));
      serverClient.close('download completed');
    }
  }
}

As a whole, there are so many. It needs to be pointed out that annie may not be very stable in message processing when parsing, which leads to some problems in my data parsing. However, there will be no problems when I use mock data and mock progress bar to send back.

Final conclusion

Read more books, read more newspapers, eat less snacks, sleep more

Tags: Javascript JSON socket Nginx

Posted on Wed, 06 Nov 2019 21:41:16 -0800 by tourer