Buffers and streams are important concepts every Nodejs developer should know, while it is not often used in the everyday development of backend applications, understanding how they work allows you to process and manipulate large data in bits without exhausting the system memory.
Before discussing the concept of buffers and streams in Nodejs, let us define Buffers in general computer science. Buffer in computer science is a space in the computer’s physical memory used to store temporary data. The purpose of the buffer is to store data right before it is used; With this definition in mind let us now define NodeJs Buffer.
Nodejs Buffer is a fixed size area of memory allocated outside of the V8 Javascript engine. They store sequences of integers similar to Javascript arrays, however, the difference is that once the buffer size has been allocated it cannot be changed, unlike javascript arrays. The Buffer class in Nodejs is designed to handle raw binary data, which leads to the next questions; what are binary data and why binary data.
Binary data is a type of data represented in the binary numeral system (Base 2 number system). The circuits in computers are made up of billions of transistors. A transistor is a tiny switch that is activated by the electronic signals it receives. The digits 1 and 0 used in binary data reflect the on and off states of a transistor. Computers don’t understand high-level languages (such as our everyday English), hence, before a computer can process a set of instructions it is encoded into binary data.
The Buffer Class in Node.Js
As mentioned earlier The buffer class in Node.Js is designed to handle raw binary data. It is a global class, so there is no need in using the required() method to import it.
How to create a Buffer
There are few ways to create a new Buffer which includes Buffer.from(), Buffer.alloc(size), Buffer.allocUnsafe(size).
Buffer.from()
const buf1 = Buffer.from("Hello, welcome to Node.Js")
console.log(buf1) //<Buffer 48 65 6c 6c 6f 2c 20 77 65 6c 63 6f 6d 65 20 74 6f 20 4e 6f 64 65 2e 4a 73>
console.log(buf1[0]) //72
console.log(buf1[1]) //101
console.log(buf1[2]) //108
console.log(buf1[3]) //108
const buf2 = Buffer.from([23,78,12,9,89])
console.log(buf2) //<Buffer 17 4e 0c 09 59>
console.log(buf2[0]) //23
console.log(buf2[1]) //78
console.log(buf2[2]) //12
console.log(buf2[3]) //9
When creating a new Buffer from a string or array of integers, the corresponding response will be encoded in UTF-8. For example, H in Hello welcome to Node.Js has been converted to its Unicode equivalent of 48
The numbers in each index of the Buffer indicate the position of the character in the Buffer. For example in Buffer1, the position of H in Hello welcome to Node.Js is 72
To view the content of a string in Buffer, use buf.toString() as seen below
const buf = Buffer.from("Hello, welcome to Node.Js")
console.log(buf.toString()) //Hello, welcome to Node.Js
Buffer.alloc(size, fill)
Buffer.alloc(size, fill) allocates a new Buffer of size bytes. If fill is undefined it will fill the arrays with 0
const buf1 = Buffer.alloc(8) console.log(buf1) //<Buffer 00 00 00 00 00 00 00 00> const buf2 = Buffer.alloc(6, 2) console.log(buf2) //<Buffer 02 02 02 02 02 02> const buf3 = Buffer.alloc(9, 3) console.log(buf3) //<Buffer 03 03 03 03 03 03 03 03 03>
From the above, we created three buffers, buf1 can only contain 8 bytes pre-filled with 0s, buf2 can only contain 6 bytes pre-filled with 2s as specified and buf3 can only contain 9 bytes pre-filled 3s as specified. Let us try to write into buf1, you will notice that if you try to write into buf1 more than its allotted size, it will only fill in the available space and discard the others. See example below
const buf1 = Buffer.alloc(8)
buf1.write("Hello, welcome to Node.Js")
console.log(buf1.toString())//Hello, w
Buffer.allocUnsafe(size, fill)
Buffer.allocUnsafe is similar in operation with Buffer.alloc; they are both used to create a new buffer and allocate default byte size and values, but the difference is that in terms of performance Buffer.allocUnsafe is faster in creating the buffers. However, the trade-off is that the allocated segment of memory might contain old data that is potentially sensitive. A more detailed answer can be found here.
Other fun parts of Buffer
Buffer.isBuffer
This is used to check if a variable is a buffer similar to the Array.isArray (for checking if a variable is an array) as seen below
const buf = Buffer.allocUnsafe(8, 1)
buf.write("Hello, welcome to Node.Js")
console.log(Buffer.isBuffer(buf))//true
console.log(Buffer.isBuffer("buf"))//false
buffer.length
This is used to check the length of a buffer
const buf = Buffer.from("Hello, welcome to Node.Js")
console.log(buf.length)//25
buffer.copy
buffer.copy(target, targetStart=0, sourceStart=0, sourceEnd=buffer.length)
It allows you to copy the content from one buffer to another. You can only specify the starting and the end position of the contents that are to be copied in the bufferCopy. Some examples will shed more light on the features of buffer.copy
const buf = Buffer.from("Hello, welcome to Node.Js")
const bufCopy = Buffer.alloc(26)
buf.copy(bufCopy)
console.log(bufCopy.toString()) //Hello, welcome to Node.Js
const buf = Buffer.from("Hello, welcome to Node.Js")
const bufCopy = Buffer.alloc(26)
buf.copy(bufCopy, 0, 12, 26)
console.log(bufCopy.toString()) //me to Node.Js
buffer.slice
buffer.slice allows the ability to extract a section of the contents in the buffer
const buf = Buffer.from("Hello, welcome to Node.Js")
const newBuf = buf.slice(0, 5)
console.log(newBuf.toString()) //Hello
Now that we have understood the concept of Buffers in Node.Js. Let’s discuss streams
Streams in NodeJs
Stream is the method of transferring large amounts of data in an efficient way. They are used to read or write input into output sequentially. Furthermore, they are used to handling reading/writing files, network communications, or any kind of end-to-end information exchange in an efficient manner.
A popular example to explain streaming is YouTube, whenever you play a video on YouTube. You will observe that the video is not available all at once, the player downloads the video in bits while you watch, peradventure your bandwidth is low, or you are experiencing a bad network, you will notice that you will get to a point where you have watched all the available downloaded frames and the YouTube player shows a loading animation trying to download the next available frames for you to watch.
Streaming is like the concept of Pay as You Go, you only pay for what you use now. Imagine YouTube did not implement the Streams concept, what this signifies is that for you to watch a 1-hour video on YouTube, you will need to wait for the player to download the video from the YouTube servers before it can become available for you to watch. Furthermore, if you lose interest in the video like 10 minutes into an hour video, the data used in downloading the other parts of the video is wasted.
Advantages of streams
- It is time-efficient – Because data is available in bits, it takes less time to start processing the data than waiting for the whole data to be available.
- It is memory efficient – It requires less memory to process the data because you don’t need all the data to be available in the memory before processing it
Types of NodeJs Streams
- Readable stream – A readable stream is an abstraction for a source from which data can be read, in other words, it lets you read data from a source
- Writeable stream – A writable stream is an abstraction for a destination to which data can be written
- Duplex stream – You can both read and write into it, in other words, it is a combination of both readable and writeable streams. Example – net.Socket
- Transform stream – It is similar to a duplex stream, but it can modify the data as it is being written and read. Example – zlib.createGzip
Reading from a stream
This is a stream you can read from but can not send data into it. When data is pushed into a readable stream, it is buffered until a consumer starts reading from the stream.
const fs = require("fs");
const readerStream = fs.createReadStream("./veryLargeFile.txt");
readerStream.on("data", function (chunk) {
console.log(chunk.toString());
});
readerStream.on("end", function () {
console.log("Stream Ended");
});
readerStream.on("error", function (err) {
console.log(err.stack);
});
On line 5 of the example above, each chunk is a buffer, as mentioned in the explanation of buffer above, for it to become readable, you have to call the toString property method on it as seen
Writing to a stream
This is a stream used for writing data in chunks, instead of all at once
const fs = require("fs");
const writer = fs.createWriteStream("./newFile.txt");
for (let i = 0; i <= 2500; i++) {
writer.write("Displaying index " + i + "\n");
}
Piping Streams
Piping is a mechanism in NodeJs where the output of a stream is fed as an input into another stream. With that mentioned, let us see an example of piping
const fs = require("fs");
const reader = fs.createReadStream("./sample.txt");
const writer = fs.createWriteStream("./newFile.txt");
reader.pipe(writer);
console.log("Piping ended");
Conclusion
Congratulations you have added another set of NodeJs concepts to your wealth of knowledge. See you next time cheers !!!
