Different Blog

Month: June 2023

If You Are a New Programmer With These Bad Habits, Be Careful

June 22, 2023 / / 0 Comments

I’ve been doing pair programming for a long time to support people who are not very good at programming. I will summarize the bad habits I noticed and the good habits for growth.

Power of habit

I’ve always wondered what determines the power of programming. Some people are good at learning, some people are bad. It’s easy to call these things “sense” and discard them, but I thought there might be factors that determine these things apart from whether one’s brain is good or bad.

A few years ago, I had an epiphany when I looked at my colleague’s screen next to me or the various engineers in my office. For some reason, I felt that the lower the output speed, the more likely the “stack trace screen output by the web application framework” would be displayed.

I’m not keeping statistics on this, so I can’t say for sure, but my colleague’s development cycle next to him went something like this:

  • Write code
  • Save code
  • Switch the screen to the browser
  • To reload
  • An error screen is an output
  • Immediately return to the code screen

I thought this was very inefficient. He wrote the implementation of the part close to the model layer (although the processing was written in the controller as a matter of course).

In no time, I taught them how to “check syntax” and “run modules on screen.” I also taught “how to write a small test code” using it. When I asked him why, he was at a loss for an answer. He said, “Since the screen will be displayed in the end, it will be closer to completion if you look at the screen.’’

It was a matter of habit. At that time, there was no CI in our system. Many engineers were not in the habit of writing tests.

After that, CI was introduced, and writing test code became a necessity, and after confirming that it worked on the browser, he was writing tests without much trouble.

The confirmation cycle was long, and the problem was only compounded by the fact that it had been merged, and the error message was output. I didn’t read much, so maybe I didn’t care.

The degree of cohesion of the code was high, and the difficulty of the test increased, probably because I wrote the test without being conscious of the unit test. Even systems designed to support productivity ended up undermining productivity through bad habits.

Bad habit

Recently, I’ve been doing pair programming with a few young engineers. It is touched for a relatively long time on a scale of several hours a day. I will introduce the bad habits I have pointed out or discovered.

Low code reading ratio

In programming, the percentage of code reading tends to be high, depending on the type of project. Some things are 80% reading and 20% writing. Code reading is necessary to understand the framework, peripheral code, and essential parts.

However, in the case of engineers whose output is stagnant, the ratio of the act of reading the source is meager. For example, I’m trying to use some framework features, but the result is not working correctly. The framework side’s error message told me it was an invalid input. However, he did not read the corresponding part of the framework.

So what are you doing?

  • Look at what you have written and vaguely search for mistakes
  • Vaguely see the copy-and-paste source code that seems to work correctly

That’s what I was doing. And when he didn’t understand, his style was to ask himself what was wrong with this.

How to improve

I taught how to find the corresponding code of the framework and made it possible to find it. So I asked them to read their actions to see why my code wasn’t working.

Psychological problems

When I asked why he didn’t look into the framework code, he replied, “Because it seems difficult. I want to finish it quickly.” Frameworks are becoming more complicated, and there are places where it’s difficult to grasp the whole.

However, as you read it, you will understand the language functions and libraries, and your knowledge will expand. Getting rid of this psychological resistance and reading the code to solve the problem quickly is essential.

Therefore, it is also essential for mentors not to give instructions such as “You don’t have to understand this’’ or “Just move it without reading it’’. I suspect that the depth of digging into the code is a fundamental parameter of the limits of programmer growth.

Do not check for small cycles

As with the very first example of the power of habit, it’s important to have the fastest confirmation cycle possible. In other words, it is also automated in the end.

It wasn’t until he did a pair project that he realized he was “looking at the code” after writing it. He wasn’t sure what he was doing, but when asked, he said, “I’m checking.” He goes through the code he wrote line by line to see if something is wrong.

How about a syntax check? I answered, “What is a syntax check?” After I taught her how to do it, she asked, “Are you okay?”

I told him to check it out, but he didn’t understand what I discussed.

Only then did I realize that confirmation other than operation confirmation combined with tests was not a habit? Unit tests must be designed to have consistency, so they require somewhat complicated preparations, and if you are not a test-first practitioner, the start phase will be delayed.

How to improve

Register the processing to check the syntax in the editor and execute it as appropriate. Then, if necessary, make it work automatically on save. Since the editor was Vim, I put in a plugin like Quickrun or Syntastic and let it run accordingly.

Anywhere is fine. Create an environment where the class/function can be executed alone, and check the operation. After confirming some correct behavior, move it to a test file.

Make a habit of that flow. Instruct them to complete those phases before asking themselves, “Is this OK?”

Once you get used to it, creating a small CI environment using a debugger or watch would be a good idea, and having it run every time you save.

Psychological problems

You may think these environmental problems are a matter of knowledge and unrelated to psychological problems. Still, if you look closely, they are binary, such as “correct code” and “errors you don’t understand why.” The part that is doing how to catch the problem is glimpsed.

There is no only correct code. However, this seems to be a common way of thinking for people who use copy and paste a lot, and the programming cycle of searching for something that does not work even if you copy and paste is doing something wrong and not being able to copy and paste accurately. I felt that there was a background in the fact that it has become a habit.

Therefore, once the code is written, it looks like a list of symbols, and you have to look for any mistakes with your eyes. Since my writing also starts with copying and pasting, it is a list of symbols I don’t understand.

Checking while understanding line by line seems less efficient than copying and pasting.

Don’t read error message logs.

Error messages in programming languages ​​and error messages output by libraries are information humans write for humans to identify problem areas. But slow-growing programmers don’t read it. It is binarized and taken as information that there was a problem.

If it is an IDE, you can immediately jump to the relevant part of the module. If this is raw Vim or a log message output via a web browser, it becomes your job to understand and jump.

It doesn’t read. It doesn’t jump to the relevant part, so the time to display the error screen is extremely short. I thought he was testing my kinetic vision because I was doing a pair project, and the error screen disappeared so quickly, but he didn’t read it.

As a result, I am staring at what I wrote earlier. I’m going back and forth between method names to see if there’s a typo.

However, the message output was not the message from hitting a nonexistent method. If you don’t read the error message, the possibilities for where the bug might occur are endless. However, he said he thought something he had written was “wrong” and needed to be fixed.

How to improve

Read the error message together and check the meaning one by one.

And let them understand the correspondence between error messages and problems. It also creates a minimal snippet that causes it and makes assumptions about various cases.

Psychological problems

He says, “I can only see symbols.” Language and framework errors are challenging to make sense of without understanding the terminology and structure of the language and framework.

It’s not because the language is unfriendly or the framework is unfriendly. The error message doesn’t represent the “implementation problem itself.” So knowledge of the framework, the language, and tracking the implementation only then can you connect the problem to the error.

Unless you make it a habit, the error will remain a symbol until you can do it yourself.

I don’t know how to find the problem.

For example, suppose that when an error occurs, the problem is discovered for the first time in the corresponding part written in the error message.

If that happens, it also means that the code worked, at least that far. Also, if an error has occurred, what you have entered has likely had an effect, and that is what is happening.

For example, with printf debugging, you can get an overview of the problem by looking at what went into the module, how it changed, and what caused the error, and then walk up the call stack to Able to identify problem areas.

However, suppose you don’t know how to search. In that case, you start checking from where it is expected to work usually or blindly output information without knowing the order relationship of the call stack.

Sometimes, they “get stuck” without knowing where the problem is. Without a strategy to identify the problem, it’s straightforward to fall into the phenomenon of “getting stuck.”

How to improve

After a problem has arisen, ask them to share their strategy for identifying it with an “Okay, what should I do?” before taking action. Ask questions such as how to collect the necessary information and to what extent is expected, and make a strategy.

Also, what does this mean if you get one piece of information? Will strategy change? Making them conscious of such things will lead them to search with a firm will instead of blindly searching.

Psychological problems

The fact that a program error occurs should be one of the pieces of evidence that it is “close to completion.” This is because one piece of information that has not been conscious of until now becomes clear, the part where the wrong hypothesis was made is rejected, and a following explicit action to correct it is obtained.

More or less, however, programmers who aren’t good at searching for problems tend to “panic” over self-generated error messages. “I’m sure it should work,” but it doesn’t. I think it’s a terrible “wrong.”

One example is the phenomenon of “I don’t want to run a test that should fail.” The most significant value of a test is that it fails and provides information. It says “This will cause an error.”

Aside from errors in production, no one will be bothered if an error occurs in the development environment. Rather beneficial.

It would be beneficial to have them experience and understand this area.

Good habits

You can get good habits by overturning bad habits.

  • Take time to read and understand code often
  • always use wisdom to simplify the problem at hand.
  • Don’t be afraid to make mistakes; make a minimal case to understand important information
  • Develop a straightforward course of action for tracking issues

Here are a few other things I think are good habits:

  • Improving and automating tools
  • boy scout policy
  • learn multiple languages
  • learn systematic knowledge
  • Is that what you mean?

Improving and automating tools

Good habits are easier to establish with good tools. For example, whenever I write a new language, I look for code for matters that produce standard output and incorporate them.

If it is an IDE, it will naturally compensate for the necessary space. I don’t want to spend precious keystrokes adjusting my coding style.

Boy scout policy

Boy Scout policy is to pick up any trash you pass by. Based on this, the policy is to correct garbage and errors in the peripheral code. I can’t fix the code unless I read it carefully, but since it’s a necessary step anyway, I’ll fix warnings, etc., at that time.

Doing this will naturally lead to a better understanding and cleaner overall code.

Learn multiple languages

Even if you learn only one language, you will be able to understand the points that are difficult to understand by learning multiple languages. New concepts are often just borrowed from other languages, so they are quickly mastered.

Acquire systematic knowledge

I want newcomer programmers to know The history of humankind’s acquisition of object-oriented programming. The quality of programming changes by knowing the background, history, and occasional issues. come.

Suppose you try to pick and choose what is good about knowledge or do not organize your understanding of grammar and paradigms in your way. In that case, it will become tips instead of knowledge, and you will end up endlessly pursuing tips without applying them.

Lastly,

Since I believe competence is integral to habits, improving habits will improve competence. And there are psychological barriers to them, and as a result, it is often the case that habits do not improve, and even if habits improve, it is impossible to become a super engineer in a leap. It is not a sense of helplessness but whether or not you enjoy learning new things that most influence your results.

10 things should be avoided as a javascript developer

June 20, 2023 / / 0 Comments

As a JavaScript developer, it is important to keep in mind the best practices and avoid certain common pitfalls in order to write efficient and maintainable code. Here are ten things that you should avoid as a JavaScript developer

1. Overuse of global variables:

Using too many global variables can make it difficult to keep track of the state of your application and can lead to name collisions. Instead, try to use modules and closure to keep your variables within a well-defined scope

// Avoid this
let userName = "John Doe";
let userAge = 30;
let userAddress = "123 Main St.";

// Instead, use modules
const userModule = (() => {
  let name = "John Doe";
  let age = 30;
  let address = "123 Main St.";

  return {
    getUser: () => ({ name, age, address })
  };
})();

// Usage
console.log(userModule.getUser());

In this example, instead of having three global variables, they are all scoped within the userModule closure. This makes the state of the application easier to manage and reduces the risk of name collisions.

2. Blocking the UI thread:

JavaScript is single-threaded, which means that long-running tasks can freeze the user interface. To avoid this, try to use Web Workers or asynchronous functions to run intensive tasks in the background.

// Avoid this
for (let i = 0; i < 1000000000; i++) {
  // Do something intensive
}

// Instead, use Web Workers
const worker = new Worker("./worker.js");
worker.postMessage({ data: "Hello, Worker!" });

3. Neglecting cross-browser compatibility:

Not all browsers implement the same features or behave in the same way. Be sure to test your code on multiple browsers and use feature detection or polyfills to ensure compatibility

// Avoid this
const result = new Map().values();

// Instead, use feature detection or polyfills
if (typeof Map.prototype.values === "function") {
  const result = new Map().values();
} else {
  // Implement polyfill
}

4. Writing complex code:

Simple code is often easier to read, understand, and maintain. Try to write small, focused functions that do one thing well, and use descriptive functions and variable names to make your code self-explanatory

// Avoid this
const complexFunction = (input) => {
  let output = input;
  for (let i = 0; i < input.length; i++) {
    output = output.slice(0, i) + output.slice(i + 1);
  }
  return output;
};

// Instead, write simple code
const reverseString = (input) => input.split("").reverse().join("");

5. Not using strict mode:

Strict mode is a mode in JavaScript that makes it easier to write secure and reliable code. Enable strict mode by adding “use strict”; at the top of your script or function.

// Avoid this
function doSomething() {
  // Not in strict mode
}

// Instead, use strict mode
function doSomething() {
  "use strict";
  // In strict mode
}

6. Ignoring performance considerations:

performance is important, especially on mobile devices and low-end hardware. Be mindful of the performance implications of your code, and use tools like the browser’s performance profiler to identify and optimize slow code.

// Avoid this
const slowFunction = (input) => {
  let result = 0;
  for (let i = 0; i < input.length; i++) {
    result += input[i];
  }
  return result;
};

// Instead, consider performance
const fastFunction = (input) => input.reduce((a, b) => a + b, 0);

7. Not handling errors:

Failing to handle errors can lead to unexpected behavior and make it difficult to diagnose and fix problems. Use try-catch blocks and throw meaningful error messages to handle errors and make debugging easier.

// Avoid this
const unreliableFunction = (input) => {
  return input.split("/");
};

// Instead, handle errors
const reliableFunction = (input) => {
  try {
    return input.split("/");
  } catch (error) {
    throw new Error(`Cannot split string: ${error.message}`);
  }
};

8. Not keeping up with best practices and new features:

Not keeping up with best practices and new features: The JavaScript language and its ecosystem are constantly evolving. Stay up-to-date with the latest best practices and new features by reading blogs and following industry leaders.

// Avoid this
const oldWay = (input) => {
  let result = 0;
  for (let i = 0; i < input.length; i++) {
    result += input[i];
  }
  return result;
};

// Instead, use new features
const newWay = (input) => input.reduce((a, b) => a + b, 0);

9. Overusing this keyword:

The value of this in JavaScript can be confusing, especially in complex code. Try to use arrow functions, bind, call, and apply to explicitly set the value of this, or use functional programming techniques to avoid it altogether.

// Avoid this
const user = {
  name: "John Doe",
  age: 30,
  address: "123 Main St.",
  getUserInfo: function() {
    console.log(`Name: ${this.name}`);
    console.log(`Age: ${this.age}`);
    console.log(`Address: ${this.address}`);
  }
};

// Instead, use destructuring
const user = {
  name: "John Doe",
  age: 30,
  address: "123 Main St."
};

const getUserInfo = ({ name, age, address }) => {
  console.log(`Name: ${name}`);
  console.log(`Age: ${age}`);
  console.log(`Address: ${address}`);
};

// Usage
getUserInfo(user);

10. Writing spaghetti code:

Spaghetti code is code that is hard to follow, understand, and maintain. Write clean and well-organized code, and use comments, indentation, and whitespace to make your code readable and understandable.

// Avoid this
function fetchData() {
  let data = null;
  makeApiCall().then(res => {
    data = res;
    if (data.status === 200) {
      // do something with the data
    } else {
      // handle error
    }
  });
}

// Instead, use a clean and modular approach
async function fetchData() {
  try {
    const res = await makeApiCall();
    if (res.status !== 200) {
      throw new Error("Failed to fetch data");
    }
    // do something with the data
  } catch (error) {
    // handle error
  }
}

Advanced Techniques for Detecting Memory Leaks in Node.js Applications

June 16, 2023 / / 0 Comments

Memory leaks can be a serious issue in any application, and Node.js is no exception. In this post, we’ll explore some common causes of memory leaks in Node.js applications and how to detect and fix them.

What is a Memory Leak?

A memory leak occurs when an application allocates memory but fails to release it when it’s no longer needed. This can cause the application to consume more and more memory over time, eventually leading to performance issues or even crashes.

In the diagram above, we can see how a memory leak can cause an application to consume more and more memory over time. As the application continues to allocate memory without releasing it, the amount of memory used by the application grows until it eventually causes problems.

Common Causes of Memory Leaks in Node.js

There are several common causes of memory leaks in Node.js applications. Some of the most common include:

1. Global Variables

Global variables can easily cause memory leaks if they’re not managed properly. If you’re using global variables, make sure to clean them up when they’re no longer needed.

Here’s an example of how a global variable can cause a memory leak:

// This global variable will never be cleaned up
let myGlobalVariable = [];

function myFunction() {
  // This data will be added to the global variable
  // every time the function is called
  myGlobalVariable.push(someData);
}

// This will cause the global variable to grow
// every time the function is called
myFunction();
myFunction();
myFunction();

In this example, we have a global variable myGlobalVariable that is never cleaned up. Every time we call myFunction, we add more data to this global variable. Over time, this can cause the global variable to grow and consume more and more memory.

2. Event Listeners

Event listeners can also cause memory leaks if they’re not removed when they’re no longer needed. Make sure to remove event listeners when they’re no longer needed.

Here’s an example of how an event listener can cause a memory leak:

// This event listener will never be removed
document.addEventListener('click', () => {
  // This code will be executed every time
  // the document is clicked
  doSomething();
});

In this example, we have an event listener that is never removed. Every time the document is clicked, the doSomething function is called. Over time, this can cause the event listener to consume more and more memory.

3. Closures

Closures can also cause memory leaks if they’re not managed properly. Make sure to only keep references to the data that’s actually needed by the closure.

Here’s an example of how a closure can cause a memory leak:

function createClosure() {
  let largeData = [];

  // This closure will keep a reference to largeData
  return () => {
    // This code will be executed every time
    // the closure is called
    doSomethingWith(largeData);
  };
}

// This will create a closure that keeps
// a reference to largeData
let myClosure = createClosure();

// This will call the closure and execute
// the code inside it
myClosure();

In this example, we have a closure that keeps a reference to largeData. Every time we call myClosure, we execute the code inside the closure that uses largeData. Over time, this can cause the closure to consume more and more memory.

Detecting Memory Leaks

There are several tools and techniques you can use to detect memory leaks in your Node.js applications. Some of the most common include:

1. Heap Snapshots

You can use heap snapshots to see how much memory your application is using and where that memory is being allocated. This can help you identify potential memory leaks.

Here’s an example of how you might use heap snapshots to detect a memory leak in a Node.js application:

const heapdump = require('heapdump');
const fs = require('fs');

// Take a heap snapshot
heapdump.writeSnapshot((err, filename) => {
  if (err) {
    console.error(err);
  } else {
    console.log(`Heap snapshot written to ${filename}`);
  }
});

// Load the heap snapshot
const snapshot = heapdump.readFileSync(filename);

// Analyze the heap snapshot
const topConsumers = snapshot.getTopConsumers();

// Log the top consumers
console.log(topConsumers);

In this example, we’re using the heapdump module to take a heap snapshot of our application. We then load the snapshot and use it to analyze our application’s memory usage. Finally, we log the top consumers of memory in our application.

2. Profiling

Profiling tools can help you identify which parts of your code are using the most memory. This can help you pinpoint potential memory leaks.

Here’s an example of how you might use profiling to detect a memory leak in a Node.js application:

const inspector = require('inspector');

// Start the profiler
const session = new inspector.Session();
session.connect();
session.post('Profiler.enable', () => {
  session.post('Profiler.start', () => {
    // Run your code here
    doSomething();

    // Stop the profiler
    session.post('Profiler.stop', (err, { profile }) => {
      // Log the profile data
      console.log(profile);
    });
  });
});

In this example, we’re using the inspector module to start a profiling session. We then run our code and stop the profiler when we’re done. Finally, we log the profile data to see which parts of our code are using the most memory.

3. Logging

You can also use logging to track memory usage over time. This can help you identify trends and patterns that may indicate a memory leak.

Here’s an example of how you might use logging to detect a memory leak in a Node.js application:

// Log the memory usage every second
setInterval(() => {
  const memoryUsage = process.memoryUsage();
  console.log(memoryUsage);
}, 1000);

In this example, we’re using setInterval to log our application’s memory usage every second. Over time, we can analyze these logs to see if there are any trends or patterns that may indicate a memory leak.

Fixing Memory Leaks

Once you’ve identified a potential memory leak, there are several steps you can take to fix it. Some common techniques include:

1. Refactoring Your Code

In many cases, refactoring your code to avoid common causes of memory leaks (such as global variables and event listeners) can help fix the issue.

Here’s an example of how you might refactor your code to avoid a memory leak caused by a global variable:

// Instead of using a global variable,
// pass the data as an argument
function myFunction(someData) {
  // Use the data passed as an argument
  doSomethingWith(someData);
}

// Pass the data as an argument
myFunction(someData);

In this example, we’ve refactored our code to avoid using a global variable. Instead of storing our data in a global variable, we pass it as an argument to myFunction. This allows us to avoid the memory leak caused by the global variable.

2. Using a Garbage Collector

Node.js has a built-in garbage collector that can help clean up unused memory. Make sure your code is structured in a way that allows the garbage collector to do its job effectively.

Here’s an example of how you might structure your code to allow the garbage collector to clean up unused memory:

function myFunction() {
  // Allocate some memory
  let someData = [];

  // Use the data
  doSomethingWith(someData);

  // Allow the garbage collector to clean up
  // the unused memory
  someData = null;
}

myFunction();

In this example, we’ve structured our code in a way that allows the garbage collector to clean up unused memory. After we’re done using someData, we set it to null. This allows the garbage collector to reclaim the memory used by someData.

3. Monitoring Your Application

Finally, make sure to monitor your application for signs of memory leaks. This will help you catch any issues early on and fix them before they become serious problems.

Here’s an example of how you might monitor your application for signs of memory leaks:

// Monitor the application for signs of memory leaks
setInterval(() => {
  const memoryUsage = process.memoryUsage();

  // Check if the heapUsed value is growing over time
  if (memoryUsage.heapUsed > someThreshold) {
    // There may be a memory leak
    console.warn('Potential memory leak detected!');
  }
}, 1000);

In this example, we’re using setInterval to monitor our application for signs of memory leaks. Every second, we check if the heapUsed value is growing over time. If it is, we log a warning message indicating that there may be a potential memory leak.

In this post, we’ve explored some common causes of memory leaks in Node.js applications and how to detect and fix them. By understanding these causes and using the tools and techniques we’ve discussed, you can help prevent memory leaks in your own Node.js applications.

Thank you for reading! I hope this post has been helpful and informative. If you have any questions or comments, please feel free to leave them below.

Common Problems in Message Queues [With Solutions]

June 15, 2023 / / 0 Comments

Queue : A message queue is a service that allows applications to send and receive messages/events. They provide a way for applications to communicate asynchronously and decouple the sending and receiving of messages, allowing for more flexible and scalable systems.

Problems:

Queue-based systems are widely used for asynchronous communication and message processing, but they can encounter a variety of issues that can affect their performance and reliability. Some common issues that can occur in queue-based systems include:

  • Message loss: Messages can be lost due to network failures, system crashes, or other unexpected events.
  • Message ordering: In a distributed system, messages may not be received in the same order they were sent, which can cause issues when processing the messages.
  • Poison messages: Poison messages are messages that cannot be processed by the consuming application due to errors such as format errors, missing dependencies, or application errors. These messages can block the processing of other messages in the queue, causing delays and other problems.
  • Overload: Queue-based systems can become overwhelmed if the rate of incoming messages exceeds the system’s ability to process them. This can lead to high memory usage, slow performance, and even system crashes.
  • High latency: Queue-based systems can experience high latency due to long processing times, large message sizes, or network congestion.

Solution to Common Problems:

Solving Queue Overloading :

  1. Implement backpressure: By implementing backpressure, the message queue can slow down or stop accepting messages when it becomes overloaded, preventing the queue from becoming overwhelmed.
  2. Use multiple queues: By using multiple queues, different types of messages can be processed in parallel, reducing the likelihood of a single queue becoming overloaded.
  3. Use a dead-letter queue: A dead-letter queue is a queue that holds messages that cannot be processed by a consumer, it can be used to hold messages that are not critical and can be processed later, or that are blocked because of a problem.
  4. Implement retry mechanism: Implementing a retry mechanism for failed messages can help reduce the number of messages in the queue and improve the overall performance of the system.

Others: Monitor the queue for actual problems, Scale the queue

Solving Poison Messages:

https://www.enterpriseintegrationpatterns.com/patterns/messaging/RequestReply.html

Use DLQ : A dead letter queue (DLQ) is a queue used to hold messages that cannot be successfully processed by the consuming application. These messages may be undeliverable due to a variety of reasons, such as format errors, missing dependencies, or application errors.

The purpose of a DLQ is to provide a mechanism for isolating and handling problematic messages, so that they do not block the processing of other messages in the main queue. Once a message is placed in a DLQ, it can be examined and potentially re-processed, or discarded if it is determined to be unimportant or unprocessable. This process of handling a message in DLQ is also known as poison message handling.

DLQs are commonly used in message-oriented middleware systems, such as RabbitMQ and Kafka etc. DLQs can be implemented in different ways, but typically, they are associated with a primary queue, and a message will be moved to the DLQ if it cannot be successfully processed by the consuming application after a certain number of retries.

Solving Message Loss:

  1. Implement acknowledgement: Acknowledgement is a mechanism that allows the consumer to confirm that it has received and processed a message. This can be used to ensure that messages are not lost if the consumer crashes or is disconnected.
  2. Use persistence: Persistence is a mechanism that allows messages to be stored on disk, rather than in memory. This ensures that messages are not lost if the message queue is restarted or if there is a power outage.
  3. Use replication: Replication is a mechanism that allows messages to be replicated across multiple servers. This ensures that messages are not lost if one of the servers goes down.
  4. Use a transaction: using a transaction to handle message enqueue and dequeue can help ensure that a message is not lost if the dequeue process is failed.

Solving Large Data Volume:

  1. Use a distributed message queue: A distributed message queue allows messages to be stored and processed across multiple servers, allowing for greater scalability and the ability to handle large volumes of data.
  2. Use compression: Compressing the data before adding it to the queue can significantly reduce the amount of data that needs to be stored, allowing the message queue to handle larger volumes of data.
  3. Partition the data: By partitioning the data, it can be divided into smaller chunks, which can be processed separately, reducing the load on the message queue.
  4. Consider using a database/data lake: If the data volume is really large, using a database specifically designed to handle large data volume such as NoSQL database like Cassandra, MongoDB could be more suitable than using a message queue.

Others: Scaling, Monitoring..

Solving High Latency :

  1. Optimize network communication: By optimizing the network communication, such as using faster protocols or reducing the amount of data that needs to be sent over the network, the time it takes for messages to be delivered can be reduced, reducing latency.
  2. Cache: See if we can use a cache or in memory message queue like redis

Others: Scaling, Monitoring, distributed queue ..

Solving message ordering :

  1. Using a timestamp: By adding a timestamp to each message, messages can be sorted by their timestamp and processed in the order they were sent.
  2. Using a unique ID / sequence number : By assigning a unique ID to each message, messages can be sorted by their ID and processed in the order they were sent.
  3. Using a distributed log system like Apache Kafka, it uses a log-based storage system and each message is assigned an offset in the log. This allows for the messages to be read in the order they were written to the log.

Definition : Distributed message queue

A distributed message queue is typically built on top of a distributed architecture, which allows for the message queue to be spread across multiple servers, each running the message queue software. Each server can handle a portion of the message queue, and can communicate with the other servers to ensure that all messages are processed.

Some popular distributed message queue systems include Apache Kafka, RabbitMQ, Apache ActiveMQ and Amazon SQS

Junior developers: Your job is more than just coding!

June 13, 2023 / / 0 Comments

This is a misunderstanding I see all the time.

Junior developers think they just need to learn more technical skills in order to advance in their career.

But often, they have enough coding ability to reach the next level.

Here’s what they’re missing…

Proactive & pragmatic

Junior engineers struggle to understand the other parts of our job.

They’re missing the situational awareness of a senior engineer. They don’t operate with enough autonomy. In difficult situations, they struggle uselessly while trying to resolve a problem.

Great senior engineers have a few characteristics in common that take them to the next level:

  1. They’re proactive. If something is happening that they don’t understand, they’ll research it. They find ways (code, Git history, documentation, monitoring tools, etc) to trace down the underlying logic. When they discover something new, they communicate about it with the team.
  2. They’re pragmatic. They know that the simplest solution is usually the best. There’s no need to over-complicate a technical problem. But the same goes for people problems. Reduce complexity, friction, and back and forth.

Asking good questions

I see junior engineers fall into this trap all the time.

They ask:

“It’s not working! How do I make it work?”

The problem with that question is it doesn’t show any effort or context. You’re literally presenting a problem to the other engineers on your team without any attempt at a solution.

Instead, ask a question like this:

“I’m trying to do x. But it’s failing because of y. I’ve tried working around it with z, but I haven’t had success. Any recommendations?”

The tone of that question is entirely different.

It shows you put effort into trying to find a solution. Moreover, it’s abundantly clear the context and exact issue you’re facing.

Most importantly, it’s the way senior engineers talk to one another on the same level. The wording of the question assumes that you’re peers with your colleagues, working together to find the solution.

Communication is the job

Asking questions in a good manner is one form of communication.

But as you grow as a developer, you’ll realize that our entire job is communication. The technical implementation of a feature may be difficult, but it’s not the hardest part of the job.

Collaborating with others, setting expectations, and clearly communicating the plan is the hardest part.

That’s why I encourage all developers to practice communication skills.

Specifically, I think writing skills are the most important to take your communication to the next level.

A developer who can write well sets themselves apart. It’s an amazing skill to add you your toolbelt. It’ll do more to take you to senior levels than any new technical skill.

More resources

There’s a ton more to our job than just coding. This article only scratches the surface. Luckily, I’ve written about this a bunch of other times. Check out the articles below for more!

5 Daily Practices of Top Developers

June 13, 2023 / / 0 Comments

Consistency is king in this career. Our industry rewards developers who can ship & quickly solve problems.

Let’s talk about what the best developers do every day…

(Reminder: “The Habits of Top Developers” is available with exclusive content through the end of the day. I’ve also extended the 20% off link through tonight. If you want the deep dive on daily practices, that book is for you!)

1. Get clarity

Standup is (too often) a waste of time.

But a great standup has so much potential as a forcing function. The real reason for your standup update is to bring the most important thing to the surface.

Great engineers know this, and they plan for it.

Before standup, top engineers have reviewed the list of possible things they could work on today. They’ve picked the most important problem to resolve first.

In standup, they say what that problem is and whether there are any blockers.

A great standup update is quick, because it has only the most important thing (plus whatever is standing in your way).

2. Review code

Your team is depending on you.

When you take a long time to review code, it slows everything down. Conversely, when you review code on a quick, regular cadence; everything speeds up.

Great engineers know that shorter feedback loops lead to better code.

Waiting for code review is a feedback loop. And that loop is often VERY long.

Do something about it. Keep reviews top of mind, or batch them for bulk reviewing sessions at least once per day.

3. Test, test, test

How do you know your code works? How do you develop confidence in your application?

Every day you should be testing code in some way.

Most often, that’s programmatically with unit tests. If there’s a day where you didn’t write a unit test, then that’s a red flag.

But great developers don’t stop there.

They layer their tests to include functional tests & end-to-end tests. Let the computers test themselves at every step and level of the code.

Finally, you should be manually smoke testing your code.

I know a lot of developers who believe manual testing is beneath them. That’s a mistake. There’s no better way to connect with your product than to run through the user flows yourself.

Manual testing builds a user-centric muscle common to top developers.

4. Lots of writing

I write a daily post, but that’s meager in comparison to the amount of writing I do for work.

During a normal day, most of that writing is on Slack. The communication that happens there is incredibly valuable. It’s how you collaborate and unblock your teammates.

Moreover, you should be writing some form of documentation — code comments, READMEs, API docs, internal wikis — most days that you work as a developer.

There’s such thing as too much documentation. And old documentation is hard to maintain.

But writing is something that top developers do all day most days.

5. Digging through logs/metrics

It’s surprising. You think that great developers spend a lot of time in the code.

But actually, as you advance in your career, you spend less time in the code. Usually the code changes you make are smaller, focused.

Instead, you spend a ton of time figuring out the problems and the best ways to solve them. For that, you need to spend a lot of time looking at monitoring and observability software.

Great developers look at logs, APM, CI pipelines, etc for much of the day to diagnose and monitor what’s going on with the application. Learning to use these tools is a key part of working at an elite level as a developer.

Fall in love with the logs & they’ll help you immensely.

5 important Mistakes made by Nodejs Developers

June 12, 2023 / / 0 Comments
  1. Lack of Knowledge About Asynchronous Programming : Developers who do not use asynchronous functions correctly can cause performance problems. Developers should be able to decide well what should work synchronously and asynchronously.
async function fetchData() {
  try {
    const response = await fetch('https://api.example.com/data');
    const data = await response.json();
    return data;
  } catch (error) {
    console.log('Error fetching data:', error);
  }
}

fetchData().then((data) => {
  console.log('Data fetched:', data);
}).catch((error) => {
  console.log('Error:', error);
});

In this example, the fetchData function is defined with the async keyword, which allows us to use await to pause the execution of the function until the API request is completed. Once the data is retrieved, the await keyword is used again to extract the JSON data from the response.

The function returns the data, which is then logged to the console in the then block of a promise. If there is an error during the API request, the catch block will log the error to the console.

2. Memory Leaks : Node.js uses garbage collector to minimize memory leaks. However, developers’ code can have memory leaks, which can lead to performance issues.

The following example is we memory leak example:

function exampleData() {
  const data = [];
  // ...
  data.push(newData);
  // ...
  return data;
}

setInterval(() => {
  const result = exampleData();
  console.log(result);
}, 1000);

This example causes the data array created by the exampleData function to be spooled in memory before being used in a function that repeats every second. This leads to a memory leak and negatively affects the performance of the program.

To fix this code, the created objects must be deleted from memory after use. Here is an updated version of the exampleData function from the previous example:

function exampleData() {
  return new Promise((resolve, reject) => {
    const data = [];
    // ...
    data.push(newData);
    // ...
    resolve(data);
  });
}

setInterval(() => {
  exampleData().then((result) => {
    console.log(result);
  }).catch((error) => {
    console.log(error);
  });
}, 1000);

The function now returns a Promise corresponding to the array, instead of returning an array.

The fetchData function is called on each call, the result is returned as a Promise object, and the result is automatically cleared by the garbage collector after it is used. This prevents memory leaks and improves the performance of the program.

3. Incorrect use of Callback function : The following example uses a callback function to read data from a file. However, if used incorrectly, this function can cause errors and become hard-to-read code:

const fs = require('fs');

function readDataFromFile(callback) {
  const filePath = './data.txt';

  fs.readFile(filePath, 'utf8', (err, data) => {
    if (err) {
      callback(err);
    } else {
      callback(null, data);
    }
  });
}

readDataFromFile((err, data) => {
  if (err) {
    console.error('Error:', err);
  } else {
    console.log('Data:', data);
  }
});

In this example, the **readDataFromFile** function takes a **callback** function and reads data from a file via the **fs** module. However, this example is used incorrectly. The callback function is always called, even when an error occurs. This requires you to write more code to **catch** and handle errors.

Here is a corrected version of the above code:

const fs = require('fs');

function readDataFromFile(filePath, callback) {
  fs.readFile(filePath, 'utf8', callback);
}

readDataFromFile('./data.txt', (err, data) => {
  if (err) {
    console.error('Error:', err);
  } else {
    console.log('Data:', data);
  }
});

In this example, the readDataFromFile function is using it properly. The callback function is passed directly from fs.readFile and is handled on failure or when called successfully. This allows you to write less code and make the code more readable.

4. Not using async waterfall : Async waterfall is a function provided by “**async**“, a widely used asynchronous control flow library in Node.js. ****This function allows us to perform a series of actions sequentially, and after each action is completed it allows us to move on to the next action.

Using an async waterfall is especially useful when there are dependencies between processes. For example, there is a dependency between operations such as downloading a file, opening the file, and processing its contents, and we need to perform these operations sequentially.

Using the async waterfall allows us to perform operations sequentially and accurately and improves the readability of the code.

Here is an example:

const async = require('async');
const fs = require('fs');

async.waterfall([
  function downloadFile(callback) {
    // Download a file
    // ...
    callback(null, 'file.txt');
  },
  function processFile(filename, callback) {
    // Read and process the file contents
    fs.readFile(filename, 'utf8', (err, data) => {
      if (err) {
        callback(err);
        return;
      }
      // Process the file contents
      const processedData = data.toUpperCase();
      callback(null, processedData);
    });
  },
  function uploadFile(processedData, callback) {
    // Upload the processed data to a server
    // ...
    callback(null, 'success');
  }
], (err, result) => {
  if (err) {
    console.error('Error:', err);
    return;
  }
  console.log('Result:', result);
});

In the example above, the async.waterfall function sequentially executes three operations, downloadFile, processFile, and uploadFile, which run depending on each other’s results.

The first process downloads a file and passes the filename to the second process. The second process reads and processes the contents of the file, then passes the result to the third process. The third operation uploads the processed data to the server and returns the result.

Each action depends on the result of the previous action and will only run after the previous action is complete. This allows us to perform operations sequentially and accurately.

5. Not Doing Debugging Operations: Below is an example of code written without error catching:

const fs = require('fs');

fs.readFile('myfile.txt', (data) => {
  console.log('File contents:', data);
});

console.log('Program ended');

In this example, if an error occurs, the program will simply crash and we won’t know what went wrong. Below is a more accurate example:

const fs = require('fs');

fs.readFile('myfile.txt', (err, data) => {
  if (err) {
    console.error('Error reading file:', err);
    return;
  }
  console.log('File contents:', data);
});

console.log('Program ended');

© 2026 Different Blog

Theme by Anders Noren — Up ↑