1. Let's make a reactive framework Pt-1
I tried authoring my own javascript library to build a reactive frontend for the web. Sounds like React and every other Reactive Javascript library out there? That's because it is. Now in this post I will go through, how I tried to implement my own version of react. Now those of you who know how React works will immediately realise just how stupid I am. But stick with me here..... for the duration of time that it will take you to read this article, I want you you to forget everything you know about React's Virtual DOM, diffing algorithm and hooks. I want to take you through the journey I went through to do this as the complete noob I was was with 0 knowledge of how react work 3 months ago.
It actually started because I wanted to build a devtool for myself(it's a secret) and performace and customizability was something I wanted as part of the whatever frontend framework I used. But I read about how other projects similar to the one I had in mind refrained from using any framework because they didn't want to be tied down by the restrictions of the framework and didn't want to hunt down performance issues caused by it. Now I could just write everything I wanted in vanilla javascript and propogate changes in the UI with individual DOM Manipulation calls. But reactive systems like React had spoilt me. I couldn't go back to writing lines and lines of DOM manipulation just to change the look of a few UI components. So I decided to write my own reactive framework. Atleast this way if there was any performance issue I could just look at my own code and fix it instead having to go through the mammoth codebase that is react to figure out how to change something. Idk how big react is, I haven't checked.
Ok so off to the races, we basically need to write our own version of react. Setup a typescript project, write some TSX(Typescript JSX) files, the typescript compiler will let you know you don't have anything in place to process JSX yet. So then you do a bit of research and find out that you can add a function that typscript will substitute the JSX tags with and basics of it can be figured out just through linter and build errors. After a couple of days of trying, I figured out how React worked, atleast I think this is how react works. Some bits I had to cheat to understand by console logging some of the functions React provides and other bits I sort of had a vague understanding of which helped when I really thought about it. This is sort of how it went.
We'll use the following snippet as our test code:
function InternalComponent(name){
console.log("Inside InternalComponent");
return (<span>Hello {name}</span>);
}
function App(){
console.log("Inside App");
return (<div><InternalComponent name="George" /></div>);
}
How we render each element
React converts each JSX element into a function call.
<div id="id">Name</div> becomes React.createElement("div", {"id": "id"}, "Name")
<App>Name: {name}</App> becomes React.createElement(App, {}, "Name: ", name)
So we see that it's converting each JSX element into a function call with the first argument as the element, second argument as the props/attributes of the element and arguments 3 onward will be the child JSX elements. So if the element has 4 children then arguments 3, 4, 5, and 6 will be the 4 corresponding children and since the children are also JSX those will also get converted into JSX calls.
We see here that the properties, attributes and children of these elements we create are all javascript objects and expressions. But that's not how browsers work, browsers expect HTML to be a document that describes your UI and you can change that UI using Javascript. Now the issue with it is that whenever you want to link the data or actions of one HTML component with another. Let's say you want to change the value of a certain <h> tag with the value of an <input> tag as you type. This would mean attaching an event handler for onchange to the input tag and then on each chaange you update the text inside the h tag by calling the specific DOM function. And the more of these links you make between elements or components the more you have to carry out these manipulation calls. There isn't a way to make it so that if you change one value then it will automatically just update all the elements that use that value. We want our UI to automatically react to the changes in our data. This behaviour we want is called reactivity. This is what React enables. React... Reactivity. This is what the code sort of achieves. Because the props, values and children are all just javascript objects, you can change the javascript objects and that will change the attributes and children and so on.
So I write my JSX rendering function that will take the element, either a string for an HTML element or an identifier for a Component element, props for the element and it's children. If it is an HTML element, then it will create HTML element with document.createElement, set it's properties and append the children passed to it, and return the element. If it's a component then run the component function by passing the properties and children to the function and return the element returned by that component. This will happen recursively and I tried it out. It all works great. UI is rendering exactly How I want it to.
A very basic implementation of createElement is right below. This obviously isn't the full function. The full function actually has a whole diff algorithm that checks if the element has change from it's previous version and if it hasn't then has it's props changed and what all DOM manipulationg it needs to do depending on those changes. And the old element is stored in a Virtual DOM ouitside of the function that the function cheks and updates as it goes. But i've skipped that for the sake of simplicity. This will show what happens when a new element is met with that wasn't there before without any context of the external Virtual DOM.
function createElement(element, props, ...children){
if(typeof element === 'string'){
// This means this is an HTML Element
let e = document.createElement(element);\
// Set attribute and event listeners according to the props
// e.setAttribute("", "")
// e.onEvent(eventHandler)
// Append children to element
for(let child of children){
e.appendChild(child)
}
return e;
}else{
// This means a function component was passed
return element({...props, children});
}
}
Compiler output from our test snippet is this:
function InternalComponent(){
console.log("Inside InternalComponent");
return createElement('span', {}, 'Hello', name);
}
function App(){
console.log("Inside App");
return createElement('div', {}, createElement(InternalComponent, {"name": "George"}));
}
And when I run it console prints:
Inside App
Inside InternalComponent
How we maintain state and re-render each element
Now you will notice that each component is a function so any values inside the function are discarded once the function is executed. So if we want to have some value that carries over theough multiple life cycles of the component, say a number that keeps incrementing everytime we click a button, that number cannot be stored inside this function because it will be lost eventually.
How react does this is with a function called useState. useState returns the value of the state and a setState function to the set that state. Now this seems simple, all you need is a function that when you pass a value to, stores this value somewhere, returns that value, together a function that updates the stored value and re runs the function that we'll refer to as the setState function. When you call setState, it reruns that same component function but this time the call to useState will return the updated value and again the setState function
Looks sort of like this:
function App(){
console.log("Inside App");
const [name, setName] = useState("George");
return (<div onClick={()=>{setName("Thomas")}}>{name}</div>)
}
At first the text will be George but then when you click on the div it will change to Thomas.
But the thing to realise here is that useState is a global function. The value that's being stored or updated depends on the function that called useState which is essentially a local function but useState in itself is a global function. When you call the setState function, how does React know which components useState this call corresponds to.
There's nothing you do in your component that tells useState "Hey this function right here. The reference to this function that's being run right now. This is what you need to rerender when I call the setState you give me." But there's no way to do this. There's no way for useState to implicitly get a reference to the function that's calling it. But there is another function that does know.
The createElement() function is what actually calls the component function. So everytime createElement runs we can let the global useState function know that the function that createElement received is what needs to be rendered when setState gets called. And everytime createElement is called with a new element is called, useState changes to return a value and setState specific to that component because the createElement call for that component set that up.
We can actually just update the global useState function to do this directly. It's much simpler imo. The updated createElement will look like this:
const useState: (value: T)=>[value, (value)=>void];
function createElement(element, props, ...children){
if(typeof element === 'string'){
// This means this is an HTML Element
let e = document.createElement(element);\
// Set attribute and event listeners according to the props
// e.setAttribute("", "")
// e.onEvent(eventHandler)
// Append children to element
for(let child of children){
e.appendChild(child)
}
return e;
}else{
// This means a function component was passed
let componentStates = [];
let stateIndex = 0;
useState = (initialValue)=>{
if(componentStates.length==stateIndex){
// This is the first time useState
// has been called for this state value
// So we store add it's initial value to the list of states
componentStates.push(initialValue)
}
let currIndex = stateIndex;
stateIndex++;
return [
componentStates[currIndex],
(newValue)=>{
componentStates[currIndex] = newValue;
createElement(element, props, ...children)
// There's an extra bit here to update child back
// to the Virtual DOM and thereafter the actual page
}
];
}
return element({...props, children});
}
}
createElement will maintain a list of state values of each component and useState will use and update that list of states to figure out whether to initialize state or to return existing state. stateIndex helps useState keep track of which state value is being initialized or updated. So when you update a state value, it calls the function again and this time when it calls useState it has the updated list of states. I stored this state list in the virtual DOM I mentioned earlier on in the post so that it would be tucked away somewhere for me to fetch again. That's been ommited for the sake of simplicity.
This is not actually how I implemented this functionality, the re-render wasn't called quite so simple either, especially with the complication we will be discussing next. I honestly don't remember how I did it and there's not way to know since I scrapped this and moved to a different approach which we will discuss to later.
Now it looks like state is implemented, we call createElement, that sets up state vairables and the it calls the function component that calls useState and it's all hunky dory. Sadly no, if we go back to the original compiler output and examine the two compiled JSX lines.
// Inside App
return createElement('div', {}, createElement(InternalComponent, {"name": "George"}));
When the createElement inside App is called, it will first evaluate the createElement function calls that are passed as children to it. That's just the order of evaluation that needs to happen. It cannot call the first createElement without first knowing everything that needs to be passed which means evaluating the second createElement. What this means is that in a given block of JSX, the createElement for children are evaluated before the parents and lastly the root element of that Block.
So each time the a function component is rendered. This is what happens:
- createElement for a the function component is first called.
- Then the function component is called.
- Then the return expression ie. the JSX block must be evaluated. This a series of createElement calls.
- The way it is structured, the createElements of the leaf nodes in the block of JSX needs to be evaluated first before it's parents within that JSX block can be executed.
- createElements of the JSX block are called in a bottom up fashion.
- Whenever a node within the JSX block is another function component, this cycle continues again for that function component before continuing with the original function components cycle.
What this results in is a weird call system where the component functions are called in a sort of top-down manner but the createElements within a JSX block is called in a bottom-up manner. This was causing a huge problem when I implemented the useState function. I can't now recall what that huge problem is because that was back in November 2022 and now it's February 2023. Basically what happened was because of this fuck-all order of evaluation I wasn't able to call the right function component to re-render when setState was called and the global value of useState had already changed by the time setState is called by an event handler and something and all. But looking at what I've written now I can't seem to figure out why that was the case.
Now this next bit seems kind of useless because I can't remember what it actually solves....... but I guess we carry on anyway
How we actually re-renders each element
Ok we need it to only be top down, we can't have the JSX be evaluting itself in the bottom up manner. I don't know how I realsied this but I did, I realised that the evaluation of the child jsx elements needs to be run inside the evaluation of the parent jsx elements and not outide and before it. So internal createElements need to be deferred. But this isn't possible, the order of evaluation just won't allow for that, createElement calls of child JSX nodes will run before parent JSX nodes.
But if we really think about it, it's not that we want child createElements to be run by the parent. We want what createElement currently does for child elements to be called by parent elements. So what we need to do is move the component creation out of createElement. So what will createElement? Well in essence nothing. We wangt it to not run the code but instead return a function that does that, let's call this the InternalFunction. So now createElement no longer returns a child. Instead it returns an InternalFunction that creates a child. This also means that the children passed to a createElement call are no longer the actual children elements, they are InternalFunctions. So instead of just appending the child in the createElement like we do in the original createElement function, we must first run the children InternalFunctions to get the element and then append it to the parent.
This means that the node creation, setting of attributes and event listeners within a JSX block is now carried out in a top down manner because the InternalFunction themselves are created and returned in the bottom up order of the JSX evaluation. The runnning of the InternalFunction of children is actually done by the parent that it's passed to. So the actual work we want done is now executed top-down.
const useState: (value: T)=>[value, (value)=>void];
function createElement(element, props, ...children){
return () => {
if(typeof element === 'string'){
// This means this is an HTML Element
let e = document.createElement(element);\
// Set attribute and event listeners according to the props
// e.setAttribute("", "")
// e.onEvent(eventHandler)
// Evaluate children and append to element
for(let child of children){
e.appendChild(child())
}
return e;
}else{
// This means a function component was passed
let componentStates = [];
let stateIndex = 0;
useState = (initialValue)=>{
if(componentStates.length==stateIndex){
// This means this is the first time useState has been called for this state value
// So we store the initial value of state
componentStates.push(initialValue)
}
let currIndex = stateIndex;
stateIndex++;
return [
componentStates[currIndex],
(newValue)=>{
componentStates[currIndex] = newValue;
(createElement(element, props, ...children))();
// There's an extra bit here to update child back
// to the Virtual DOM and thereafter the actual page
}
];
}
return element({...props, children});
}
};
}
React actually doesn't do it like this. It actually separates the the createElement and internal function into two separate non-nested functions. createElement just returns a plain object of the element type, it's props and it's children, which are all in this same object format so it returns a recursive top down tree of the JSX block. This object is passed to a function ReactDOM.render, which carries out the function of the internal function we wrote and instead of calling the children InternalFunctions, since they are not functions any more, now they are these tree objects, it calls ReactDOM.render on these children objects as well to get the corresponding HTML element to append to the child. We've all seen this ReactDOM.render element in the index.js file of our react apps.
Fascinating stuff. I'm sure I made very little sense in this post but if you want a explaination feel free to hit me up on twitter.
We're not done yet though. I realised that since I have to call createElement for every single JSX element in my app and run the diff algorithm on every single one of them(comparing old props and new props to know which attributes, styles and event handlers to change) and all this runs at runtime. The thing is that 90% of these elements never change in any of these aspects. Sure components might get added and removed depending on the state but when you take any single component about 90% of it is not state dependent. But the dynamic nature of the diff algorithm assumes every aspect of a component can change and carries out these tree comparisons on ever single node. The Virtual DOM diffing is still several degrees faster than HTMl DOM diffing. But when you consider how much of the diff actually results in DOM manipulations and how much of it leaves an HTML element as it is, you will realise just how much unnecessary work is being done in diffing static components.
So I gave up and decided to go back to Vanilla Javascript. Until I thought of something else which we will talk about in Part-2.....