by Bozho | Jun 21, 2020 | Aggregated, Developer tips, Opinions
If we have to integrate two (or more) systems nowadays, we know – we either use an API or, more rarely, some message queue. Unfortunately, many systems in the world do not support API integration. And many more a being created as we speak, that don’t have APIs. So when you inevitably have to integrate with them, you are left with imperfect choices to make. Below are seven patterns to integrate with legacy systems (or not-so-legacy systems that are built in legacy ways). Initially I wanted to use “bad integration patterns”. But if you don’t have other options, they are not bad – they are inevitable. What’s bad is that fact that so many systems continue to be built without integration in mind. I’ve seen all of these, on more than one occasion. And I’ve heard many more stories about them. Unfortunately, they are not the exception (fortunately, they are also not the rule, at least not anymore). Files on FTP – one application uploads files (XML, CSV, other) to an FTP (or other shared resources) and the other ones reads them via a scheduled job, parses them and optionally spits a response – either in the same FTP, or via email. Sharing files like that is certainly not ideal in terms of integration – you don’t get real-time status of your request, and other aspects are trickier to get right – versioning, high availability, authentication, security, traceability (audit trail). Shared database – two applications sharing the same database may sound like a recipe for disaster, but it’s not uncommon to see it in the wild. If you are...
by Bozho | May 19, 2020 | Aggregated, low-code, Opinions, rad, rapit application development
Recently, many low-code/no-code solutions have gained speed in the enterprise, giving non-technical people the option to create simple applications. Analysts predict that the low-code industry will grow by 20+% each year. But what is low-code, why is it getting so popular and what are the issues with it? Low-code is something that we’ve occasionally seen in the past decades – a drag-and-drop UI designer that allows you to create simple applications without coding skills. Products have matured and practically all offer similar features – the ability to design entity relationships in drag-and-drop entity-relationship diagrams, the ability to design UI via WYSIWYG, to design simple processes via BPMN-like notations, to call external services via importing web service definitions and to fetch and store data in external databases and spreadsheets. There are many tools in this domain – MS PowerApps, OutSystems, Appian, Mendix, Google’s recently acquired AppSheets, Ninox, WaveMaker, and many more. And they may differ slightly in their approach and in their feature set, but the whole point is to be able to easily create applications (web-based, mobile-based, hybrid) that solve some immediate pain that these users have, where going through a full-blown IT project with the associated development cost is an overkill. And that sounds great – you don’t have to rely on the overly busy and not often too responsive IT department in your non-IT company, you just build something yourself to optimize your immediate problem and to digitize your paper processes. And it can do that. To be honest, I like the idea of such tools existing, because a large portion of digital transformation is not handled...
by Bozho | Apr 7, 2020 | Aggregated, expertise, Opinions, software engineering
The other day I had a conversation with a scientist friend who said something alone the lines of “yes, I work in that general field, but I’m not an expert in your question in particular”. IT is not science, of course, but I asked myself whether I am a real expert in the things that I do. And while it’s nearly impossible to hit exactly the fine line between impostor syndrome and boasting, this post is neither and has a point, so bear with me. I’ve been doing a lot of things in the general IT field – from general purpose software engineering, IT architecture, information security, applications of cryptography, blockchain, e-government, algorithmic music composition, data analysis. And I’ve seen myself as having relatively expert knowledge. I even occasionally give TV and radio interviews, where I’m labelled as “Expert in X”. But… Am I a real expert in software engineering and software architecture? I’ve been doing that for 15+ years, and I follow and somethings define or clarify best practices, I’m familiar with different methodologies and have been part of teams that implemented some of them correctly and efficiently. I have taken part in the decision making process of building large systems with their architectural implications. But I’ve never been formally assigned as an “architect” (not that I insist), my UML skills are rather basic and I’ve never had to integrate dozens of legacy systems. I’ve never used formal methods for assessing software, I’ve made mistakes in selecting technologies, I’ve never done proper TDD and I have only a basic understanding of networking. Maybe just the sheer amount of...
by Bozho | Mar 13, 2020 | Aggregated, aws, Developer tips, elasticsearch, Opinions, post-mortem
So it happened that we had a production issue on the SaaS version of LogSentinel – our Elasticsearch stopped indexing new data. There was no data loss, as elasticsearch is just a secondary storage, but it caused some issues for our customers (they could not see the real-time data on their dashboards). Below is a post-mortem analysis – what happened, why it happened, how we handled it and how we can prevent it. Let me start with a background of how the system operates – we accept audit trail entries (logs) through a RESTful API (or syslog), and push them to a Kafka topic. Then the Kafka topic is consumed to store the data in the primary storage (Cassandra) and index it for better visualization and analysis in Elasticsearch. The managed AWS Elasticsearch service was chosen because it saves you all the overhead of cluster management, and as a startup we want to minimize our infrastructure management efforts. That’s a blessing and a curse, as we’ll see below. We have alerting enabled on many elements, including the Elasticsearch storage space and the number of application errors in the log files. This allows us to respond quickly to issues. So the “high number of application errors” alarm triggered. Indexing was blocked due to FORBIDDEN/8/index write. We have a system call that enables it, so I tried to run it, but after less than a minute it was blocked again. This meant that our Kafka consumers failed to process the messages, which is fine, as we have a sufficient message retention period in Kafka, so no data can be lost. I...
by Bozho | Feb 20, 2020 | Aggregated, Developer tips, Opinions, programming
In enterprise software the top one question you have to answer as a developer almost every day is “Where is this coming from?”. When trying to fix bugs, when developing new features, when refactoring. You have to be able to trace the code flow and to figure out where a certain value is coming from. And the bigger the codebase is, the more complicated it is to figure out where something (some value or combination of values) is coming from. In theory it’s either from the user interface or from the database, but we all know it’s always more complicated. I learned that very early in my career when I had to navigate a huge telecom codebase in order to implement features and fix bugs that were in total a few dozens line of code. Answering the question means navigating the code easily, debugging and tracing changes to the values passed around. And while that seems obvious, it isn’t so obvious in the grand scheme of things. Frameworks, architectures, languages, coding styles and IDEs that obscure the answer to the question “where is this coming from?” make things much worse – for the individual developer and for the project in general. Let me give a few examples. Scala, for which I have mixed feelings, gives you a lot of cool features. And some awful ones, like implicits. An implicit is something like a global variable, except there are nested implicit scopes. When you need some of those global variables, so just add the “implicit” keyword and you get the value from the inner-most scope available that matches the type...
by Bozho | Jan 31, 2020 | Aggregated, Developer tips, Opinions
Last month I published an article about getting email settings right. I had recently configured DKIM on our company domain and was happy to share the experience. However, the next day I started receiving error DMARC reports saying that our Office365-backed emails are failing DKIM which can mean they are going in spam. That was quite surprising as I have followed every step in the Microsoft guide. Let me first share a bit about that as well, as it will contribute to my final point. In order to enable DKIM, on a normal email provider, you just have to get the DKIM selector value from some settings screen and copy them in your DNS admin dashboard (in a new ._domainkey. record). Microsoft, instead, make you do the following: Connect to Exchange Powershell Oh, you have 2FA enabled? Sorry, that doesn’t work – follow this tutorial instead Oh, that fails when downloading their custom application? An obscure stackoverflow answer tells you that you should not download it with Firefox or Chrome – it fails to run then. You should download it with Internet Explorer (or Edge) and then it works. Just Microsoft. Now that you are connected, follow the DKIM guide In addition to powershell, you should also go in the Exchange admin in your Office365 admin, and enable DKIM Yes, you can imagine you can waste some time here. But it finally works, and you set the CNAME records and assume that what they wrote – namely, that the TXT records to which the CNAME records point, are generated on Microsoft’s end (for onmicrosoft.com), and so anyone trying to...
Recent Comments