Why I prefer Go for Elasticsearch
I’ve used C#, Python, Go, and dabbled in a few other languages over the years with Python being my preferred language currently for scripts and Go for any apps or programs I need to write. I learned C# originally to make games in Unity but do not enjoy writing in it, but sadly it is the main app language my company uses.
My experience working with Elastic is from the admin side of it working in DevOps to help developers use it, create ways to abstract parts of it away from developers, to make it easier for us to scale our usage of it, or to create my owns tools for managing and monitoring it.
If you are unaware of Elasticsearch, it is a non relational, document based database where most everything is JSON which means that there must decoding and encoding when reading from it and writing to it. The above languages all handle this very differently on top of the pros and cons of the languages themselves and the pros and cons of the first party Elastic client for each of them.
So why Golang for Elasticsearch? It’s a simple answer. Go allows me to be as strict as I want to be while getting at least decent if not very good performance.
First Python. Easy to write, easy to run. Zero strictness. The Elasticsearch Python Library maps directly to a Dictionary. Straight forward and simple. The library is also well written and fully documented. Now you could create classes if you want to add some strictness but the library does not care.
However when scaling beyond scripts the great experience of the library is undercut by the language itself. Difficulty of multi threading, performance pitfalls, memory leaks, debugging and so on. Is part of that my own skill issues with Python? Absolutely. However it is also partly the inherent issue I have with Python and C which is that you can do pretty much anything any way but you better know the one or two ways to do it correctly. For me that means that my development time on Python for anything more than a script is spent debugging and bug fixing, or trying to solve serious performance problems.
Second we have .NET and C#. My experience with it, is that it is a bloated but rather performant object oriented language. When I have had to write it to either help developer teams debug issues between it and Elasticsearch or create examples for them to follow I have absolutely hated working with it and Elasticsearch. The reason for that is strictness, and lack of documentation of the Library, especially after Elastic re-wrote it for Elastic v8 and the documentation still has not caught up.
The Elasticsearch client for .NET is absolutely strict and requires everything to be an Class which means serialization from JSON to the object. This means that even writing a simple API call or getting a single document requires more code than the Python version and presents a few other disadvantages. Elasticsearch supports multi index searches, get document, bulk get document, and so on with each being structured differently in Elasticsearch’s JSON response which means you need a class to handle each one with the proper types. All of that has left working between .NET and Elasticsearch extremely slow and painful for me. Serialization also adds a resource usage impact.
Then you have the Library’s problems. Lack of documentation can mean that searching for third party code can be the main way to figure out how to do something, especially for the more advanced or less used features. Another issue is how was written to be more object oriented. The example I have for this is working with a PIT search which requires you to open a PIT search through one API call, search with the search API, and then best practice is to close the PIT search. Opening a PIT search gives you a id which is just a string but in the library it has a class, which you can pass directly into the search request. To close the search though? There is a second class for sending the PIT id which means you cannot just pass in the object from when you opened it or just pass it in as a straight string. By itself that is minor issue but I ran into that all across the library which significantly increases the complexity of working with it.
Overall between the language itself and the way the library was made to work with the language while not having first class support from Elastic makes working with it a slow, painful, and complex process.
Golang. The language itself is not object oriented, however it does have structs, and being a compiled language types, which means it sits in the middle of strictness between .NET and Python. However due to the way the Elasticsearch Library is written the level of strictness is left up to me.
The Library has two types of clients, a strongly typed client and a regular client, further you can either unmarhsal to a struct or to a map[string]interface{}. What this leaves you with is some flexibility to deal with the lack of strictness within JSON with the benefits of the language over a language like Python.
Want to have structs for some cases but not all? You can. The benefit here is let’s say you had a multi index search with around 100 fields, but you only want to edit 10 before passing down the rest. You could create a struct defining the response and document, or you could just unmarhsal to a map, work with just the fields you care about, and if it goes back out as JSON marshal it and send it. You do have to write the code to handle all the pieces you do not to care about unless you want to. Another benefit is that if the upstream added a new field to an index your code would not break or lose the new field.
Overall my experience now writing multiple apps to work with Elasticsearch is that I can write the code nearly as fast as I would in Python, have near or better performance than I would with .NET and spend little or no time debugging, or troubleshooting performance problems. The Library isn’t as well officially documented on Elastic’s doc pages however with the library itself most things are documented.
My experience with each language biases me, but my goal at work is to get the task complete and my experience is that I can get a well functioning Golang app that uses Elasticsearch out much faster than Python or .NET that is easier to maintain and uses less CPU and RAM than either.