Common loader source analysis and a markdown-loader implementation

This article will take you to a simple understanding of the loader of webpack, a loader that converts md into an abstract grammar tree and then into an html string.By the way, take a brief look at the source code and workflow of several style-loader s, vue-loader s, and babel-loader s.

md2html-loader source address

Introduction to loader

webpack allows us to use loader, a node module exported to function, to process files.The matched file can be converted once and the loader can be chained.The loader file processor is a CommonJs-style function that accepts a String/Buffer type input and returns a String/Buffer type return value.

Two forms of loader configuration

Option 1:


// webpack.config.js
module.exports = {
  ...
  module: {
    rules: [{
      test: /.vue$/,
      loader: 'vue-loader'
    }, {
      test: /.scss$/,
      // sass-loader passes first, then css-loader passes the result, and then style-loader.
      use: [
        'style-loader',//Create style node from JS string
        'css-loader',// Translate CSS to CommonJS
        {
          loader: 'sass-loader',
          options: {
            data: '$color: red;'// Compile Sass to CSS
          }
        }
      ]
    }]
  }
  ...
}
Copy Code

Method 2 (called right-to-left)

// module
import Styles from 'style-loader!css-loader?modules!./styles.css';
Copy Code

When chaining multiple loaders, remember that they execute in reverse order.Depending on the array writing format, execute from right to left or from bottom to top.Like pipelining, each loader is processed one by one, the result of the previous loader is passed to the next loader, and the last Loader returns the processed result as a String or Buffer to the compiler.

loader-utils allows you to compile the loader's configuration and validate it through schema-utils

import { getOptions } from 'loader-utils'; 
import { validateOptions } from 'schema-utils';  
const schema = {
  // ...
}
export default function(content) {
  // Get options
  const options = getOptions(this);
  // Verify that loader's options are legal
  validateOptions(schema, options, 'Demo Loader');

  // Write the logic to convert the loader here
  // ...
   return content;   
};
Copy Code
  • content:Represents a source file string or buffer
  • map:Represents a sourcemap object
  • meta: Represents metadata, auxiliary objects

Synchronization loader

Synchronize loader, we can return the output by returning and this.callback

module.exports = function(content, map, meta) {
  //Some synchronization operations
  outputContent=someSyncOperation(content)
  return outputContent;
}
Copy Code

If there is only one return result, you can also use return directly to return the result.However, if you need to return something else, such as a sourceMap or AST grammar tree, you can use the api this.callback provided by the webpack at this time

module.exports = function(content, map, meta) {
  this.callback(
    err: Error | null,
    content: string | Buffer,
    sourceMap?: SourceMap,
    meta?: any
  );
  return;
}
Copy Code

The first parameter must be Error or null The second parameter must be a string or Buffer.Optional: The third parameter must be a source map that can be parsed by this module.Optional: The fourth option, which is ignored by the webpack, can be anything [you can use an abstract syntax tree - AST (e.g. ESTree) as the fourth parameter (meta), which helps speed up compilation if you want to share a common AST among multiple loader s.].

Asynchronous loader

Asynchronous loader, use this.async to get the callback function.

// Cache Loader
module.exports = function(source) {
    var callback = this.async();
    // Do Asynchronous
    doSomeAsyncOperation(content, function(err, result) {
        if(err) return callback(err);
        callback(null, result);
    });
};
Copy Code

Please refer to for details Official API

Develop a simple md-loader

const marked = require("marked");

const loaderUtils = require("loader-utils");
module.exports = function (content) {
   this.cacheable && this.cacheable();
   const options = loaderUtils.getOptions(this);
   try {
       marked.setOptions(options);
       return marked(content)
   } catch (err) {
       this.emitError(err);
       return null
   }
    
};
Copy Code

The above example converts the content in the markdown file to an html string through an existing plugin, but what if there is no plugin?In this case, we can consider another solution, using the AST grammar tree, to help us operate the conversion more easily.

Source Code Conversion Using AST

markdown-ast is an abstract grammar tree node that converts the content s of a markdown file into an array. It is much simpler and more convenient to operate an AST grammar tree than a string:

//Regular processing of strings into intuitive AST syntax trees
const md = require('markdown-ast');
module.exports = function(content) {
    this.cacheable && this.cacheable();
    const options = loaderUtils.getOptions(this);
    try {
      console.log(md(content))
      const parser = new MdParser(content);
      return parser.data
    } catch (err) {
      console.log(err)
      return null
    }
};
Copy Code
const md = require('markdown-ast');
const hljs = require('highlight.js');//Code Highlighting Plugin
// Source Code Conversion Using AST
class MdParser {
	constructor(content) {
    this.data = md(content);
    console.log(this.data)
		this.parse()
	}
	parse() {
		this.data = this.traverse(this.data);
	}
	traverse(ast) {
    console.log("md Convert to abstract grammar tree operation",ast)
     let body = '';
    ast.map(item => {
      switch (item.type) {
        case "bold":
        case "break":
        case "codeBlock":
          const highlightedCode = hljs.highlight(item.syntax, item.code).value
          body += highlightedCode
          break;
        case "codeSpan":
        case "image":
        case "italic":
        case "link":
        case "list":
          item.type = (item.bullet === '-') ? 'ul' : 'ol'
          if (item.type !== '-') {
            item.startatt = (` start=${item.indent.length}`)
          } else {
            item.startatt = ''
          }
          body += '<' + item.type + item.startatt + '>\n' + this.traverse(item.block) + '</' + item.type + '>\n'
          break;
        case "quote":
          let quoteString = this.traverse(item.block)
          body += '<blockquote>\n' + quoteString + '</blockquote>\n';
          break;
        case "strike":
        case "text":
        case "title":
          body += `<h${item.rank}>${item.text}</h${item.rank}>`
          break;
        default:
          throw Error("error", `No corresponding treatment when item.type equal${item.type}`);
      }
    })
    return body
	}
}
Copy Code

The complete code is referenced here

md into abstract language tree

ast abstract syntax number to html string

Some development tips for loader

  1. Try to make sure one loader does one thing, then you can combine different scenario requirements with different loaders
  2. You should not leave the state in the loader when developing.The loader must be a pure function without any side effects. The loader supports asynchronous and therefore I/O operations in the loader.
  3. Modularization: Ensure that the loader is modular.Loader generation modules need to follow the same design principles as normal modules.
  4. Reasonable use of a properly cached cache can reduce the cost of repetitive compilation.The loader executes with the cache turned on by default, so that when the webpack executes during compilation to determine whether the loader instance needs to be recompiled, it skips the rebuild step directly, saving the cost of unnecessary rebuilding.However, if and only if your loader has other unstable external dependencies, such as I/O interface dependencies, you can turn off the cache:
this.cacheable&&this.cacheable(false);
Copy Code
  1. Loader-runner is a very useful tool for developing and debugging loaders, which allows you to run loader npm install loader-runner --save-dev independently of the webpack
// Create run-loader.js
const fs = require("fs");
const path = require("path");
const { runLoaders } = require("loader-runner");

runLoaders(
  {
    resource: "./readme.md",
    loaders: [path.resolve(__dirname, "./loaders/md-loader")],
    readResource: fs.readFile.bind(fs),
  },
  (err, result) => 
    (err ? console.error(err) : console.log(result))
);
Copy Code

Execute node run-loader

Know more loader s

style-loader Source Analysis

What it does: Insert a style into the DOM by inserting a style tag into the head er and writing the style into the innerHTML of the tag to see the source code.

Remove the option processing code first, so it's clearer

Return a piece of js code, require to get css content, and add Style to insert css into the dom to achieve a simple and simple implementation style-loader.js

module.exports.pitch = function (request) {
  const {stringifyRequest}=loaderUtils
  var result = [
    //1. Get the CSS content.2. //Call addStyle to insert CSS content into the DOM (locals are true, CSS is exported by default)
    'var content=require(' + stringifyRequest(this, '!!' + request) + ')', 
    'require(' + stringifyRequest(this, '!' + path.join(__dirname, "addstyle.js")) + ')(content)', 
    'if(content.locals) module.exports = content.locals' 
  ]
  return result.join(';')
}
Copy Code

It should be noted that normally we all use the default method, which uses the pitch method here.The pitch method has an official explanation here, pitching loader.A simple explanation is that the default loaders are right-to-left, and pitching loaders are left-to-right.

{
  test: /\.css$/,
  use: [
    { loader: "style-loader" },
    { loader: "css-loader" }
  ]
}
Copy Code

Why do we execute style-loader first, because we want to output what css-loader gets to code that can be used in CSS styles instead of strings.

addstyle.js

module.exports = function (content) {
  let style = document.createElement("style")
  style.innerHTML = content
  document.head.appendChild(style)
}
Copy Code
Source analysis of babel-loader

First, look at the configuration processing that skips the loader and the babel-loader output

Above we can see the output code and map for transpile(source, options) What does the transpile method do Babel-loader compiles code through babel.transform, so we can implement a simple babel-loader in just a few lines of code

const babel = require("babel-core")
module.exports = function (source) {
  const babelOptions = {
    presets: ['env']
  }
  return babel.transform(source, babelOptions).code
}
Copy Code
Analysis of vue-loader source code

vue single file component (sfc for short)

<template>
  <div class="text">
    {{a}}
  </div>
</template>
<script>
export default {
  data () {
    return {
      a: "vue demo"
    };
  }
};
</script>
<style lang="scss" scope>
.text {
  color: red;
}
</style>
Copy Code

webpack configuration

const VueloaderPlugin = require('vue-loader/lib/plugin')
module.exports = {
  ...
  module: {
    rules: [
      ...
      {
        test: /\.vue$/,
        loader: 'vue-loader'
      }
    ]
  }

  plugins: [
    new VueloaderPlugin()
  ]
  ...
}
Copy Code

VueLoaderPlugin effect: Copies other rules defined by webpack.config and applies them to blocks in the corresponding language in the.vue file.plugin-webpack4.js

 const vueLoaderUse = vueUse[vueLoaderUseIndex]
    vueLoaderUse.ident = 'vue-loader-options'
    vueLoaderUse.options = vueLoaderUse.options || {}
    // cloneRule modifies the resource and resourceQuery configuration of the original rule.
    // The file path with the special query will be applied to the corresponding rule
    const clonedRules = rules
      .filter(r => r !== vueRule)
      .map(cloneRule)

    // global pitcher (responsible for injecting template compiler loader & CSS
    // post loader)
    const pitcher = {
      loader: require.resolve('./loaders/pitcher'),
      resourceQuery: query => {
        const parsed = qs.parse(query.slice(1))
        return parsed.vue != null
      },
      options: {
        cacheDirectory: vueLoaderUse.options.cacheDirectory,
        cacheIdentifier: vueLoaderUse.options.cacheIdentifier
      }
    }

    // Update the rules configuration of the webpack so that clonedRules-related configurations can be applied to each label in the vue single file
    compiler.options.module.rules = [
      pitcher,
      ...clonedRules,
      ...rules
    ]
Copy Code

Get the rules item for webpack.config.js, then copy the rules to make the file with the? Vue&lang=xx...query parameter dependent on configuring the XX suffix file The same loader configures a common loader:pitcher for the Vue file [pitchLoder,... ClonedRules,... Rules] as the new rules for webapck.

Take another look at the output of the vue-loader result

When a Vue file is introduced, vue-loader parse s the Vue single-file components, gets the relevant content of each block, and converts the Vue SFC of different types of block components into a js module string.

// vue-loader uses `@vue/component-compiler-utils` to parse the SFC source code into an SFC descriptor and extracts different types of block s from the SFC based on the type of different module path s (the type field on the query parameter).
const { parse } = require('@vue/component-compiler-utils')
// Parses the contents of a single *.vue file into a descriptor object, also known as a SFC (Single-File Components) object
// descriptor contains attributes and contents of template, script, style tags, etc., which make it easy to process each tag accordingly
const descriptor = parse({
  source,
  compiler: options.compiler || loadTemplateCompiler(loaderContext),
  filename,
  sourceRoot,
  needMap: sourceMap
})

// Generate a unique hash id for a single file component
const id = hash(
  isProduction
  ? (shortFilePath + '\n' + source)
  : shortFilePath
)
// If a style tag contains scopeds, CSS Scoped processing is required
const hasScoped = descriptor.styles.some(s => s.scoped)
Copy Code

Then the next step is to add the newly generated js module to the compilation of the webpack, which is to parse the js module with AST and collect its dependencies.

To see how the source works with different type types (template/script/style), the selectBlock method essentially obtains the content of the corresponding type on the descriptor and passes it to the next loader for processing, depending on the type.

These three pieces of code parse different type s into an import string

import { render, staticRenderFns } from "./App.vue?vue&type=template&id=7ba5bd90&"
import script from "./App.vue?vue&type=script&lang=js&"
export * from "./App.vue?vue&type=script&lang=js&"
import style0 from "./App.vue?vue&type=style&index=0&lang=scss&scope=true&"
Copy Code

Summarize the workflow of vue-loader

  1. Registering VueLoaderPlugin in the plug-in copies the rules entry in the current project's webpack configuration, inserts a public loader when the resource path contains query.lang matches the same rules through resourceQuery and executes the corresponding loader, and inserts a corresponding custom loader based on query.type during pitch phase
  2. When loading *.vue, the vue-loader.vue file is parsed into a descriptor object, which contains attributes such as template, script, styles corresponding to each tag. For each tag, src? Vue&query reference code is stitched according to the tag attributes, where SRC is a single-page component path, query is a parameter of some features, lang, type and scoped are more important.If the lang attribute is included, the same rules as the suffix are matched and the corresponding loaders are applied to execute the corresponding custom loader according to the type. The template executes the template Loader, the style executes the stylePostLoader
  3. In template Loader, templates are converted to render functions through vue-template-compiler, in which the incoming scopeId is appended to each tag's segments, and finally passed to the createElemenet method as a configuration property of the vnode, which renders the page as the original property when the render function calls and renders it
  4. In stylePostLoader, parse style tag content through PostCSS

Reference

  1. webpack loader api
  2. Hand-on instructions for writing webpack yaml-loader
  3. Yanchuan-webpack Source Parsing Series
  4. Analysis of CSS Scoped Implementation from vue-loader Source

Tags: Vue Webpack sass npm

Posted on Sun, 19 Apr 2020 22:05:59 -0700 by RockyShark