Recent by Martijn de Jong 

Building custom add-ons for your Domino container image – Martijn's Blog  

By Martijn de Jong | 10/14/24 3:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This is a post that I thought I had already written, but I realised today that I hadn’t. It’s about a feature that Daniel Nashed added to the Domino container community project in the past year and that I showed in my presentations on the Domino container project at Engage and OpenNTF. But apparently, apart from that, Daniel and I never documented it. So here it is. The documentation on how to create your own custom add-on packages for your Domino container image.

Installing Domino REST API in an existing Domino container server – Martijn's Blog  

By Martijn de Jong | 10/3/24 1:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Domino REST API, a.k.a. DRAPI, is a requirement for running HCL Volt MX Go. On a native Domino server, it’s an add-on that you can install. The installation will install files in both a special install directory, the Domino program directory and the Domino data directory. On a Domino server using Domino container images, you need a Domino image with the REST API included. After all, the Domino program directory is not persistent, which means that any addition to this directory that was added in the container and not in the image, is lost when the Domino container is stopped and restarted. Something that happens whenever you reboot the host machine. Luckily, the Domino container community image build tool includes the Domino REST API in the build menu, so it’s easy to add.

End of Life for CentOS 7 AND CentOS 8 Stream  

By Martijn de Jong | 7/2/24 6:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

CentOS 7 was released on the 7th of July 2014. For many years, it has been the operating system for millions of servers. Last Sunday (30th of June) was the day when, after almost 10 years, CentOS 7 became end-of-life. This means that no (security) updates for CentOS 7 will be released any more, and that servers running CentOS 7 are at risk. I personally know of quite a few servers that are still running on CentOS 7. Even though the EOL date of CentOS 7 has been known for a very long time, many companies waited till the very last moment to phase out these systems and then missed their target. This is a bad situation to be in. I expect that it won’t be long before vulnerabilities in these systems become public, which then can no longer be patched. Migrating these systems to a new operating system should be top priority for these companies!

Installing Huddo Boards on WebSphere-only Connections   

By Martijn de Jong | 6/24/24 8:21 AM | Infrastructure - Connections | Added by Roberto Boccadoro

Let me clarify what this article is about. Huddo (also known as ISW) is selling 2 versions of their Boards product: Kudos Boards – The Java application that can be installed on WebSphere. Active development has stopped a few years ago Huddo Boards – The product consisting of several containers, which can be installed on Kubernetes or a container engine like Docker. A version with limited functionality is part of the HCL Connections component pack and known as Activities Plus. This product is actively being developed and also available for other platforms. Although the Connections Component pack was introduced in version 6 already, many companies refrained from installing it due to the serious amount of extra infrastructure needed to run these components, which includes a Kubernetes cluster. We call these the WebSphere-only Connections installations, as they only include the WebSphere stack part of Connections. So what if you would like to use the full power of Huddo Boards without having to install a Kubernetes cluster? That’s what this article is about.

My thoughts on Engage 2024  

By Martijn de Jong | 5/3/24 6:42 PM | Business - Events / People | Added by Oliver Busse

Last week was the HCL user group event Engage in Antwerp, Belgium. It’s one of my favourite events. It’s meticulously organized by Theo Heselmans, who this year organized Engage and its predecessor, BLUG, for the 15th time. As he announced last year in Amsterdam, it was also the last time that he would organize this event, and it’s therefore logical that this fact played an important role during the event.

The conf-file in the Domino Container build script  

By Martijn de Jong | 5/2/24 9:19 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In my previous post, I showed that the Domino-container build script now has a menu. When Daniel announced his plans to include a menu, I asked him to add the option to output the result of a menu build in the form build.sh domino 12.0.2 FP3 -verse -nomad etc. Why? So you could use this in a script to build the same container image with an updated Linux OS base layer unattended. Daniel listened, but implemented it in a different way.

Building your Domino Container Image in 2024  

By Martijn de Jong | 4/18/24 1:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

When you have a child which you see everyday, you don’t really notice how much he/she grew until you compare their current height with the line on the doorpost of the year before. It’s like that with the Domino container community project for me. My last major post on the Domino container project was in July 2022. Daniel Nashed, the main contributor to this project, has been steadily working on and there are many additions to the project. I use Domino containers on a daily basis, so I’ve seen the progress step by step. Only when reading my post from 2022, I realised how far the project has progressed in the past 21 months. Time for an update! The project also got a new status as since Domino 12.0.2, HCL’s official container images, which you can download from FlexNet, are now also based on the community container scripts! There are 2 new additions which make creating a Domino container image much easier: The use of the domdownload script The build menu Next to that there are a couple of very interesting new options. In this article, I’ll mainly focus on these two items. In another article, I’ll focus on the new options.

Domino Containers – The Next Step  

By Martijn de Jong | 4/11/24 5:12 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

With the Engage conference less than two weeks away, I’m working hard on my presentation. My topic will be “Domino Containers – The Next Step”. It’s a sequel to the presentation that I gave at Engage 2022 (and that same year at CollabSphere and OpenNTF) about the Domino container community project. Two years ago, I showed that Domino containers were ready to be used in production. On HCL’s FlexNet you had been able to download Domino docker images for quite a while already, but HCL never formally announced that those were for production use as well. During my session, I showed that the community images had quite a few benefits over HCL’s image and that Domino containers, based on these images, were a sensible replacement for your native Domino installations. So this time, we go a step further. Daniel Nashed has been working hard on the build-scripts for the community image and it has become easier than ever to build your own Domino image. I will show this live during my session.

Security bulletin: Passwords of Domino Internet users are vulnerable  

By Martijn de Jong | 2/22/24 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The official title of the security bulletin is: “HCL Domino is susceptible to a weak cryptography vulnerability (CVE-2023-37495).” The problem is with person documents that were created using the “Add Person” button in the Domino Directory. For people less savvy in Domino: that’s not the usual way to add users to Domino. In Domino, we register users using a certifier file. The only time we add persons to the Domino Directory using the “Add person” button, is when we know that these users will only ever access a Domino application through a web browser. The problem with these “internet users” is that the hash in the Domino Directory for the HTTP password uses a cryptographically weak hash algorithm. If an attacker has access to these hashes, he could determine the user’s password through a brute force attack. You can’t see these hashes from a browser, so the attacker needs to have access to the Domino Directory through a Notes or Nomad client. That limits the potential attackers to all users who are registered as Notes users inside the company.

Installing wireguard on CentOS Stream 9   

By Martijn de Jong | 1/15/24 3:37 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As I do a lot of my research on new Domino versions, Connections versions and HCL DX on my own server at home and as I’m often not at home, I figured I needed a VPN tunnel to my server, so I can work as if I am home. Wireguard has become kind of the de facto standard for these kind of situations, so I looked into installing it on my CentOS Stream 9 host.

Nginx as reverse proxy and SNI  

By Martijn de Jong | 11/10/23 4:43 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

I had some difficulty to find a good title for this article that would really cover the contents. Therefore, let me start with describing the problem I faced which led to this article. I have a lot of sites running on my home server (this blog being one of them) using different technologies. As I have a single IPv4 address, all these sites are behind a reverse proxy, for which I use Nginx. A couple of those sites are Domino sites and last week I realised there was something wrong in that area. I have several internet site documents on Domino for different urls. However, last week I realised that all my urls that were forwarded to Domino, were being serviced based on the same internet site document. In other words, Domino did not recognise for which internet site a request was meant.

Certificate Store: Submit vs Save  

By Martijn de Jong | 3/30/23 2:39 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

I regularly receive question about the Certificate Store and CertMgr, which made me realise that there’s a lot of confusion around the Submit Request and the Save & Close buttons in the store and when to use what. Time for an article to hopefully solve some of that confusion.

On Domino thread IDs and Linux/Windows process IDs  

By Martijn de Jong | 3/1/23 9:53 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

A short tip on something which many people are probably not aware of, but which can be a huge time saver when you’re troubleshooting a Domino problem. As an example, see this error message from a Domino log: [062372:000014-00007F8001776700] 28/02/2023 13:16:20 CertStore: Error opening CertStore database [CN=PROD02/OU=SRV/O=ACME!!certstore.nsf] : The server is not responding. The server may be down or you may be experiencing network or VPN problems. Contact your system administrator if this problem persists. [062372:000014-00007F8001776700] 28/02/2023 13:16:20 CertStore: Error opening CertStore on [CN=PROD02/OU=SRV/O=ACME] : The server is not responding. The server may be down or you may be experiencing network or VPN problems. Contact your system administrator if this problem persists. Your first hunch might be that this is an error that’s caused by the CertMgr process. It’s related to the Certificate Store after all. But is this really the case?

Huddo Boards & Minio problems – Read before you restart!  

By Martijn de Jong | 11/21/22 2:06 AM | Infrastructure - Connections | Added by Roberto Boccadoro

Last week I got the unpleasant surprise of a no longer working Huddo Boards for Component pack installation at a customer after I had rebooted my Kubernetes environment. I had to reboot this environment after I updated the Kubernetes certificates. Of course, after a change you immediately think that your problem is related to the change you just made, but in this case the only relation was with the restart, which means that this can happen to everyone running Huddo/Kudos Boards for Component pack or Huddo Boards Docker.

Protecting your Domino container with fail2ban  

By Martijn de Jong | 11/7/22 4:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

If your Domino server is connected to the Internet, you’ll find that bots (hacked systems running a script) will throw a brute force attack on your Domino server. For me, especially, my SMTP server was under heavy attack. The reason why it’s interesting for hackers to find a valid login on an SMTP server, is that this will probably allow them to send spam through your mail server. Most mail servers allow sending mail through their servers for other domains for authenticated users only. The chances of them guessing any of the users in my Domino directory right and then also guessing the password correctly are basically zero, but the pollution of my log file is reason enough to stop them. Fail2ban is a very elegant program for Linux to do just that. You can configure it to scan log files for certain patterns (it uses RegEx to recognise them) and add hosts that match those patterns too often within a defined period of time, to the block list of iptables.

HCL Traveler and error 500  

By Martijn de Jong | 7/21/22 1:31 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

HCL Traveler is one of those addons for Domino that just works. If you have a properly configured HTTPS stack, you install it, start it and you’re basically done. From now on, you can connect your mobile devices to your Domino server to read your mail and calendar. At least, that has always been my experience until very recently. The other day I was sent to a customer to fix their problem with Traveler. They had upgraded their Domino server and Traveler installation from 8.5.3 FP5 to 12.0.1 FP1. Everything worked (Kudos for Domino!) except Traveler. Though on further discussion with the client it became clear that Traveler actually already broke earlier and hadn’t been working for the past 6 years or so.

Domino containers revisited   

By Martijn de Jong | 7/20/22 1:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As I wrote in my last post about the Engage conference, a lot has happened in the Domino container space since I wrote my articles, as Daniel Nashed did some serious refactoring on all scripts, removing an insane amount of old code lines and adding some new functionality. This article will show the changes to the project compared to the time that I wrote the original 6-part series.

Working with standard Certificate Authorities in Domino 12  

By Martijn de Jong | 3/28/22 1:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In the past weeks, I helped some colleagues with importing certificates in the Certificate Store of Domino 12 and while doing so, I noticed something peculiar. For many years, we haven’t had a proper way of creating certificates in Domino. The pre-12 database to create keys was completely outdated and didn’t allow for creating strong keys. As a result, most administrators got used to creating keys outside Domino, usually through on openssl command in Linux. This way of working found its way into procedures and many admins, instead of using the Certificate Store database, still follow these old procedures and create their keys outside Domino. I therefore decided to create a short article on how to create certificates with Domino 12 which are signed by a certificate authority which doesn’t support the ACME protocol.

Installing Tivoli/Security Directory Integrator on RHEL 8ì  

By Martijn de Jong | 1/14/22 11:21 AM | Infrastructure - Connections | Added by Roberto Boccadoro

On a new SDI 7.2 installation (with Java 8 and the latest Connections TDISOL directory for Java 8), I ran into a weird error: CTGDKG023E Error while starting main class.java.lang.reflect.InvocationTargetException .. Caused by: java.lang.UnsatisfiedLinkError: i4clntjni (Not found in java.library.path) Luckily, Google could help me on this one. This technote shows that if there are missing libraries, SDI doesn’t properly install and you will have to uninstall SDI, install the missing libraries, and reinstall SDI.

Expired certificate on your Kubernetes environment  

By Martijn de Jong | 12/7/21 2:36 AM | Infrastructure - Connections | Added by Roberto Boccadoro

Normally a Kubernetes environment is well maintained and regularly updated with the most recent versions of Kubernetes. However, with a Kubernetes environment that is just used a an HCL Connections Component pack installation, this might not be on your radar and it’s easy to let it just run attended. If you do that for too long though, like longer than one year, you’ll get into trouble

Domino-docker explained – Part 5 : Adding add-ons on top of your Domino image  

By Martijn de Jong | 11/2/21 2:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In the previous parts, I explained how to create a Domino image and deploy it. But what if you want to add fix packs to your Domino image? Or Traveler, Volt or Verse? The scripts of the domino-docker project make this super simple. In this part, I’ll show you how to do this.

Domino-docker explained – Part 4 : The domino_container script  

By Martijn de Jong | 10/22/21 7:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In the previous part, I showed how you can simply start and stop and open the Domino console with the domino_container command. This piece of script is responsible for interacting with the Domino container in a way where the average administrator doesn’t even have to realise that Domino is running inside a container. There are many more functions in this script that will help you manage your Domino server and in this part I will discuss them.

Domino-docker explained – Part 3 : Running your first Domino server in a container  

By Martijn de Jong | 9/30/21 10:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In part 2 we created a Domino container image. Now we want to start the image. Of course, we could just use docker run <options> <imagename>, but with the scripts from the Domino Docker project, there’s a much easier option. In this part, I’ll show you what to do to make running, restarting and stopping images super easy.

Domino-docker explained – Part 2 : Creating your first Domino image  

By Martijn de Jong | 9/28/21 1:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

n the previous part, I looked at reasons why you might want to run your Domino server inside a container. In this part, I’m going to show how to create your first Domino image. We have to take one step back though, as since a couple of years, HCL provides their own docker image for Domino. So why would you want to create your own image? My experience is that it leads to a better image and it gives options to add your own tooling to the image. Nevertheless, using HCL’s image is an option and the script also provides an option to build on top of the standard HCl image. My advice: create your own.

Domino-docker explained – Part 1: Why run Domino inside a container?  

By Martijn de Jong | 9/28/21 1:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In November 2018, Thomas Hampel (at that time still working for IBM) created the domino-docker github repository as an open source initiative to create scripts that would make it easier to run Domino inside a container. Even though the repository was started by IBM, the work was done by the community with most of the work done by one man in particular: Daniel Nashed. He contributed his Linux start/stop scripts to the project, but also wrote scripts to completely automate the build of the images. While working with the scripts, I realised two things: Daniel has built fantastic scripts to both build and run Domino containers With so much functionality added, the project didn’t manage to document this new functionality in detail With help from Daniel, I managed to build my own customised container and I experienced in the past months all the benefits from running Domino as a container, combined with the scripts from the Domino Docker project. However, if this project wants to get the attention it deserves, the documentation needs to be fixed and this is exactly what I’ll try to do in a series of 6 articles:

Domino 12 and Borg backup  

By Martijn de Jong | 4/20/21 4:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

I must admit that I’m quite excited about Domino 12. I was thinking this morning why actually. The new features in Domino 12 aren’t necessarily groundbreaking. They’re more about fixing things which should have been in the platform already, but were neglected by IBM in the years in which it would have been logical to implement them.

Domino 12 – SSL Performance  

By Martijn de Jong | 3/22/21 2:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

A few weeks ago I wrote about the new Certificate Manager in Domino 12, which enabled Domino 12 to request and automatically update LetsEncrypt certificates and implemented a better way of Server Name Indication (previously introduced in Domino 11.0.1), so you can use different SSL certificates for different websites without needing multiple IP addresses. The Certificate Manager also allows you to use the most recent (ECDSA) ciphers. The lack of this functionality in previous versions of Domino was an important reason why, in many Domino installations, an Nginx, Apache or IHS server is placed in front of the Domino HTTP task as a reverse proxy. There was however another reason: Domino used a lot of cpu power for and was rather slow to decrypt and encrypt SSL traffic. Letting Nginx/Apache/IHS offload the SSL de-/encryption task, reduced total load on the server and sped up performance. I therefore wondered if HCL also managed to solve this problem.

Domino V12 – The Certificate Manager  

By Martijn de Jong | 2/28/21 4:49 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

HCL Domino V12 is in beta, and we currently have beta 2 to work with. One of the interesting new features of Domino V12 is the Certificate Manager task (certmgr). I’ve been playing around with this task and in this post I’ll tell about my experiences.

Installing the HCL Connections Component Pack 6.5 CR1 – Part 6: Configuring the applications  

By Martijn de Jong | 10/2/20 8:23 AM | Infrastructure - Connections | Added by Roberto Boccadoro

In part 5 I discussed the installation of all components. Now it’s time to configure them. My goal is not to duplicate the HCL documentation on this point, but to highlight where this documentation is ambiguous or incomplete. So by all means, also read that documentation.

Installing the HCL Connections Component Pack 6.5 CR1 – Part 4: Prepare the application environment  

By Martijn de Jong | 6/22/20 4:25 AM | Infrastructure - Connections | Added by John Oldenburger

If you followed the steps from the previous parts, you have a working Kubernetes cluster and a Docker registry containing all images necessary for installing the Component Pack. In this part I’ll discuss installing all helm charts up to, but not including, the actual components like OrientMe, Elasticsearch etc.